Skip to main content
Version: Preview

Create a Pulsar Cluster

Introduction

KubeBlocks can quickly integrate new engines through good abstraction. The functions tested in KubeBlocks include Pulsar cluster creation and deletion, vertical and horizontal scaling of Pulsar cluster components, storage expansion, restart, and configuration changes.

KubeBlocks supports Pulsar's daily operations, including basic lifecycle operations such as cluster creation, deletion, and restart, as well as advanced operations such as horizontal and vertical scaling, storage expansion, and configuration changes.

Environment Recommendation

Refer to the Pulsar official document for the configuration, such as memory, cpu, and storage, of each component.

ComponentsReplicas
zookeeper1 for test environment or 3 for production environment
bookiesat least 3 for test environment, at lease 4 for production environment
brokerat least 1, for production environment, 3 replicas recommended
recovery (Optional)1; if autoRecovery is not enabled for bookie, at least 3 replicas needed
proxy (Optional)1; and for production environment, 3 replicas needed

Before you start

  • Install kbcli if you want to manage the StarRocks cluster with kbcli.

  • Install KubeBlocks.

  • Check whether the Pulsar Addon is enabled. If this Addon is disabled, enable it first.

  • View all the database types and versions available for creating a cluster.

    kubectl get clusterdefinition pulsar
    >
    NAME TOPOLOGIES SERVICEREFS STATUS AGE
    pulsar pulsar-basic-cluster,pulsar-enhanced-cluster Available 16m

    View all available versions for creating a cluster.

    kubectl get clusterversions -l clusterdefinition.kubeblocks.io/name=pulsar
    >
    NAME CLUSTER-DEFINITION STATUS AGE
    pulsar-2.11.2 pulsar Available 16m
    pulsar-3.0.2 pulsar Available 16m
  • To keep things isolated, create a separate namespace called demo.

    kubectl create namespace demo
    >
    namespace/demo created

Create Pulsar cluster

  1. Create a Pulsar cluster in basic mode. For other cluster modes, check out the examples provided in the GitHub repository. If you only have one node for deploying a Pulsar Cluster, configure the cluster affinity by setting spec.schedulingPolicy or spec.componentSpecs.schedulingPolicy. For details, you can refer to the API docs. But for a production environment, it is not recommended to deploy all replicas on one node, which may decrease the cluster availability.

    cat <<EOF | kubectl apply -f -
    apiVersion: apps.kubeblocks.io/v1
    kind: Cluster
    metadata:
    name: mycluster
    namespace: demo
    spec:
    terminationPolicy: Delete
    clusterDef: pulsar
    topology: pulsar-basic-cluster
    services:
    - name: broker-bootstrap
    serviceName: broker-bootstrap
    componentSelector: broker
    spec:
    type: ClusterIP
    ports:
    - name: pulsar
    port: 6650
    targetPort: 6650
    - name: http
    port: 80
    targetPort: 8080
    - name: kafka-client
    port: 9092
    targetPort: 9092
    - name: zookeeper
    serviceName: zookeeper
    componentSelector: zookeeper
    spec:
    type: ClusterIP
    ports:
    - name: client
    port: 2181
    targetPort: 2181
    componentSpecs:
    - name: broker
    serviceVersion: 3.0.2
    replicas: 1
    env:
    - name: KB_PULSAR_BROKER_NODEPORT
    value: "false"
    resources:
    limits:
    cpu: "1"
    memory: "512Mi"
    requests:
    cpu: "200m"
    memory: "512Mi"
    - name: bookies
    serviceVersion: 3.0.2
    replicas: 4
    resources:
    limits:
    cpu: "1"
    memory: "512Mi"
    requests:
    cpu: "200m"
    memory: "512Mi"
    volumeClaimTemplates:
    - name: ledgers
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 8Gi
    - name: journal
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 8Gi
    - name: zookeeper
    serviceVersion: 3.0.2
    replicas: 1
    resources:
    limits:
    cpu: "1"
    memory: "512Mi"
    requests:
    cpu: "100m"
    memory: "512Mi"
    volumeClaimTemplates:
    - name: data
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 8Gi
    EOF
    FieldDefinition
    spec.terminationPolicyIt is the policy of cluster termination. Valid values are DoNotTerminate, Delete, WipeOut. For the detailed definition, you can refer to Termination Policy.
    spec.clusterDefIt specifies the name of the ClusterDefinition to use when creating a Cluster. Note: DO NOT UPDATE THIS FIELD. The value must be pulsar to create a Pulsar Cluster.
    spec.topologyIt specifies the name of the ClusterTopology to be used when creating the Cluster.
    spec.servicesIt defines a list of additional Services that are exposed by a Cluster.
    spec.componentSpecsIt is the list of ClusterComponentSpec objects that define the individual Components that make up a Cluster. This field allows customized configuration of each component within a cluster.
    spec.componentSpecs.serviceVersionIt specifies the version of the Service expected to be provisioned by this Component. Valid options are [2.11.2,3.0.2].
    spec.componentSpecs.disableExporterIt determines whether metrics exporter information is annotated on the Component's headless Service. Valid options are [true, false].
    spec.componentSpecs.replicasIt specifies the amount of replicas of the component.
    spec.componentSpecs.resourcesIt specifies the resources required by the Component.
    spec.componentSpecs.volumeClaimTemplatesIt specifies a list of PersistentVolumeClaim templates that define the storage requirements for the Component.
    spec.componentSpecs.volumeClaimTemplates.nameIt refers to the name of a volumeMount defined in componentDefinition.spec.runtime.containers[*].volumeMounts.
    spec.componentSpecs.volumeClaimTemplates.spec.storageClassNameIt is the name of the StorageClass required by the claim. If not specified, the StorageClass annotated with storageclass.kubernetes.io/is-default-class=true will be used by default.
    spec.componentSpecs.volumeClaimTemplates.spec.resources.storageYou can set the storage size as needed.

    For more API fields and descriptions, refer to the API Reference.

  2. Verify the cluster created.

    kubectl get cluster mycluster -n demo

    When the status is Running, the cluster is created successfully.