Skip to main content
Version: Preview

Create a Kafka cluster

This document shows how to create and connect to a Kafka cluster.

Before you start

  • Install kbcli.

  • Install KubeBlocks: You can install KubeBlocks by kbcli or by Helm.

  • Make sure kafka addon is enabled with kbcli addon list.

    kbcli addon list
    >
    NAME TYPE STATUS EXTRAS AUTO-INSTALL INSTALLABLE-SELECTOR
    ...
    kafka Helm Enabled true
    ...
note
  • KubeBlocks integrates Kafka v3.3.2, running it in KRaft mode.
  • You are not recommended to use kraft cluster in combined mode in production environment.
  • The controller number suggested ranges from 3 to 5, out of complexity and availability.

Create a Kafka cluster

The cluster creation command is simply kbcli cluster create. Further, you can customize your cluster resources as demanded by using the --set flag.

kbcli cluster create kafka

See the table below for detailed descriptions of customizable parameters, setting the --termination-policy is necessary, and you are strongly recommended to turn on the monitor and enable all logs.

📎 Table 1. kbcli cluster create flags description

OptionDescription
--mode='combined'Mode for Kafka kraft cluster, 'combined' is combined Kafka controller and broker,'separated' is broker and controller running independently. Legal values [combined, separated]
--replicas=1The number of Kafka broker replicas for combined mode. In combined mode, this number also refers to the number of the kraft controller. Valid value range[1,3,5]
--broker-replicas=1The number of Kafka broker replicas for separated mode.
--controller-replicas=1The number of Kafka controller replicas for separated mode. In separated mode, this number refers to the number of kraft controller. Valid value range [1,3,5]
--termination-policy='Delete'The termination policy of cluster. Legal values [DoNotTerminate, Halt, Delete, WipeOut].
- DoNotTerminate: DoNotTerminate blocks the delete operation.
- Halt: Halt deletes workload resources such as statefulset, deployment workloads but keeps PVCs.
- Delete: Delete is based on Halt and deletes PVCs. - WipeOut: WipeOut is based on Delete and wipes out all volume snapshots and snapshot data from backup storage location.
--storage-enable=falseEnable storage for Kafka.
--host-network-accessible=falseSpecify whether the cluster can be accessed from within the VPC.
--publicly-accessible=falseSpecify whether the cluster can be accessed from the public internet.
--broker-heap='-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64'Kafka broker's jvm heap setting.
--controller-heap='-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64'Kafka controller's jvm heap setting for separated mode. The setting takes effect only when mode='separated'.
--cpu=1CPU cores.
--memory=1Memory, the unit is Gi.
--storage=20Data Storage size, the unit is Gi.
--storage-class=''The StorageClass for Kafka Data Storage.
--meta-storage=5Metadata Storage size, the unit is Gi.
--meta-storage-class=''The StorageClass for Kafka Metadata Storage.
--monitor-enable=falseEnable monitor for Kafka.
--monitor-replicas=1The number of Kafka monitor replicas.
--sasl-enable=falseEnable authentication using SASL/PLAIN for Kafka.
-server: admin/kubeblocks
-client: client/kubeblocks
built-in jaas file stores on /tools/client-ssl.properties
  • Create a Kafka cluster in combined mode.

    # create kafka in combined mode 
    kubectl apply -f - <<EOF
    apiVersion: apps.kubeblocks.io/v1alpha1
    kind: Cluster
    metadata:
    name: kafka-combined
    namespace: default
    spec:
    affinity:
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys:
    - kubernetes.io/hostname
    clusterDefinitionRef: kafka
    clusterVersionRef: kafka-3.3.2
    componentSpecs:
    - componentDefRef: kafka-server
    monitor: false
    name: broker
    noCreatePDB: false
    replicas: 1
    resources:
    limits:
    cpu: "0.5"
    memory: 0.5Gi
    requests:
    cpu: "0.5"
    memory: 0.5Gi
    serviceAccountName: kb-kafka-sa
    terminationPolicy: Delete
    EOF
  • Create a Kafka cluster in separated mode.

    # Create kafka cluster in separated mode
    kubectl apply -f - <<EOF
    apiVersion: apps.kubeblocks.io/v1alpha1
    kind: Cluster
    metadata:
    name: kafka-separated
    namespace: default
    spec:
    affinity:
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys:
    - kubernetes.io/hostname
    clusterDefinitionRef: kafka
    clusterVersionRef: kafka-3.3.2
    componentSpecs:
    - componentDefRef: controller
    monitor: false
    name: controller
    noCreatePDB: false
    replicas: 1
    resources:
    limits:
    cpu: "0.5"
    memory: 0.5Gi
    requests:
    cpu: "0.5"
    memory: 0.5Gi
    serviceAccountName: kb-kafka-sa
    tls: false
    - componentDefRef: kafka-broker
    monitor: false
    name: broker
    noCreatePDB: false
    replicas: 1
    resources:
    limits:
    cpu: "0.5"
    memory: 0.5Gi
    requests:
    cpu: "0.5"
    memory: 0.5Gi
    serviceAccountName: kb-kafka-sa
    tls: false
    terminationPolicy: Delete
    EOF