Create a Pulsar Cluster
Introduction
KubeBlocks can quickly integrate new engines through good abstraction. The functions tested in KubeBlocks include Pulsar cluster creation and deletion, vertical and horizontal scaling of Pulsar cluster components, storage expansion, restart, and configuration changes.
KubeBlocks supports Pulsar's daily operations, including basic lifecycle operations such as cluster creation, deletion, and restart, as well as advanced operations such as horizontal and vertical scaling, storage expansion, and configuration changes.
Environment Recommendation
Refer to the Pulsar official document for the configuration, such as memory, cpu, and storage, of each component.
Components | Replicas |
---|---|
zookeeper | 1 for test environment or 3 for production environment |
bookies | at least 3 for test environment, at lease 4 for production environment |
broker | at least 1, for production environment, 3 replicas recommended |
recovery (Optional) | 1; if autoRecovery is not enabled for bookie, at least 3 replicas needed |
proxy (Optional) | 1; and for production environment, 3 replicas needed |
Before you start
-
Install kbcli if you want to manage the StarRocks cluster with
kbcli
. -
Check whether the Pulsar Addon is enabled. If this Addon is disabled, enable it first.
-
View all the database types and versions available for creating a cluster.
- kubectl
- kbcli
kubectl get clusterdefinition pulsar
>
NAME TOPOLOGIES SERVICEREFS STATUS AGE
pulsar pulsar-basic-cluster,pulsar-enhanced-cluster Available 16mView all available versions for creating a cluster.
kubectl get clusterversions -l clusterdefinition.kubeblocks.io/name=pulsar
>
NAME CLUSTER-DEFINITION STATUS AGE
pulsar-2.11.2 pulsar Available 16m
pulsar-3.0.2 pulsar Available 16mkbcli clusterdefinition list
kbcli clusterversion list -
To keep things isolated, create a separate namespace called
demo
.kubectl create namespace demo
>
namespace/demo created
Create Pulsar cluster
-
Create a Pulsar cluster in basic mode. For other cluster modes, check out the examples provided in the GitHub repository. If you only have one node for deploying a Pulsar Cluster, configure the cluster affinity by setting
spec.schedulingPolicy
orspec.componentSpecs.schedulingPolicy
. For details, you can refer to the API docs. But for a production environment, it is not recommended to deploy all replicas on one node, which may decrease the cluster availability.cat <<EOF | kubectl apply -f -
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
name: mycluster
namespace: demo
spec:
terminationPolicy: Delete
clusterDef: pulsar
topology: pulsar-basic-cluster
services:
- name: broker-bootstrap
serviceName: broker-bootstrap
componentSelector: broker
spec:
type: ClusterIP
ports:
- name: pulsar
port: 6650
targetPort: 6650
- name: http
port: 80
targetPort: 8080
- name: kafka-client
port: 9092
targetPort: 9092
- name: zookeeper
serviceName: zookeeper
componentSelector: zookeeper
spec:
type: ClusterIP
ports:
- name: client
port: 2181
targetPort: 2181
componentSpecs:
- name: broker
serviceVersion: 3.0.2
replicas: 1
env:
- name: KB_PULSAR_BROKER_NODEPORT
value: "false"
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "200m"
memory: "512Mi"
- name: bookies
serviceVersion: 3.0.2
replicas: 4
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "200m"
memory: "512Mi"
volumeClaimTemplates:
- name: ledgers
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
- name: journal
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
- name: zookeeper
serviceVersion: 3.0.2
replicas: 1
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: "100m"
memory: "512Mi"
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
EOFField Definition spec.terminationPolicy
It is the policy of cluster termination. Valid values are DoNotTerminate
,Delete
,WipeOut
. For the detailed definition, you can refer to Termination Policy.spec.clusterDef
It specifies the name of the ClusterDefinition to use when creating a Cluster. Note: DO NOT UPDATE THIS FIELD. The value must be pulsar
to create a Pulsar Cluster.spec.topology
It specifies the name of the ClusterTopology to be used when creating the Cluster. spec.services
It defines a list of additional Services that are exposed by a Cluster. spec.componentSpecs
It is the list of ClusterComponentSpec objects that define the individual Components that make up a Cluster. This field allows customized configuration of each component within a cluster. spec.componentSpecs.serviceVersion
It specifies the version of the Service expected to be provisioned by this Component. Valid options are [2.11.2,3.0.2]. spec.componentSpecs.disableExporter
It determines whether metrics exporter information is annotated on the Component's headless Service. Valid options are [true, false]. spec.componentSpecs.replicas
It specifies the amount of replicas of the component. spec.componentSpecs.resources
It specifies the resources required by the Component. spec.componentSpecs.volumeClaimTemplates
It specifies a list of PersistentVolumeClaim templates that define the storage requirements for the Component. spec.componentSpecs.volumeClaimTemplates.name
It refers to the name of a volumeMount defined in componentDefinition.spec.runtime.containers[*].volumeMounts
.spec.componentSpecs.volumeClaimTemplates.spec.storageClassName
It is the name of the StorageClass required by the claim. If not specified, the StorageClass annotated with storageclass.kubernetes.io/is-default-class=true
will be used by default.spec.componentSpecs.volumeClaimTemplates.spec.resources.storage
You can set the storage size as needed. For more API fields and descriptions, refer to the API Reference.
-
Verify the cluster created.
kubectl get cluster mycluster -n demo
When the status is Running, the cluster is created successfully.