Kafka Architecture in KubeBlocks

KubeBlocks supports three Kafka deployment topologies:

The *_monitor variants add a standalone kafka-exporter Component that scrapes Kafka-specific metrics (consumer group lag, partition offsets, topic throughput) and exposes them on port 9308 for Prometheus.

Combined Architecture (combined / combined_monitor)

In the combined topology every pod simultaneously acts as both a broker (stores and serves partition data) and a controller (participates in the KRaft metadata quorum). There is a single KubeBlocks Component (kafka-combine) for all combined nodes, and the entire set of pods forms both the KRaft controller quorum and the broker cluster.

Resource Hierarchy

Cluster  →  Component (kafka-combine)   →  InstanceSet  →  Pod × N
         →  Component (kafka-exporter)  →  InstanceSet  →  Pod × 1   [combined_monitor only]

KubeBlocks provisions kafka-combine first; kafka-exporter (if present) starts after.

Containers Inside Each Pod

Each combined pod runs two containers. A kafkatool init container runs before the main containers start, copying the /sasl directory to /shared-tools/sasl for SASL authentication support:

Each pod mounts a single PVC at /bitnami/kafka. Kafka stores the partition log data under /bitnami/kafka/data and the KRaft metadata log under /bitnami/kafka/metadata, both on the same PVC.

KRaft Consensus

Since Kafka 3.3 (KIP-833), the KRaft metadata quorum fully replaces ZooKeeper. In the combined topology every pod is a controller-eligible node, so the whole pod set forms the controller quorum:

A combined node that fails loses both its broker and controller roles simultaneously. The remaining quorum elects a new active controller; partition leaders for the affected partitions are re-elected from the ISR.

Traffic Routing

Kafka clients do not use a single ClusterIP service for all brokers. Instead, KubeBlocks creates one per-pod ClusterIP service (via podService: true) for each combined pod so that every broker can advertise a unique, stable address:

Clients bootstrap by connecting to the seed list of per-pod ClusterIP addresses on port 9092. After bootstrap, the client fetches the full cluster metadata and connects directly to the partition leader for each partition using the advertised per-pod address.

# Bootstrap seed list (all per-pod ClusterIP addresses)
{cluster}-kafka-combine-advertised-listener-0.{namespace}.svc.cluster.local:9092,
{cluster}-kafka-combine-advertised-listener-1.{namespace}.svc.cluster.local:9092,
{cluster}-kafka-combine-advertised-listener-2.{namespace}.svc.cluster.local:9092

Combined ± Monitor

The combined topology provisions only the kafka-combine Component. The combined_monitor topology additionally provisions the kafka-exporter Component, which connects to the Kafka cluster and exposes consumer group lag, topic/partition offsets, and throughput metrics on port 9308. If you use combined_monitor, configure your Prometheus scrape target to point at the kafka-exporter pod on port 9308.

Separated Architecture (separated / separated_monitor)

In the separated topology the controller and broker roles are placed in independent KubeBlocks Components, each backed by its own InstanceSet. This allows you to scale and update controllers and brokers independently and provides clear operational isolation between the metadata plane (controllers) and the data plane (brokers).

Resource Hierarchy

Cluster  →  Component (kafka-controller)  →  InstanceSet  →  Pod × N
         →  Component (kafka-broker)      →  InstanceSet  →  Pod × N
         →  Component (kafka-exporter)    →  InstanceSet  →  Pod × 1  [separated_monitor only]

KubeBlocks provisions the components in order: kafka-controller first (the quorum must elect an active controller before brokers can register), then kafka-broker, then kafka-exporter (if present). On termination, the order reverses.

Containers Inside Each Pod

Controller pods (no init containers):

Each controller pod mounts a PVC at /bitnami/kafka for the KRaft metadata log (/bitnami/kafka/metadata).

Broker pods (init container: kafkatool copies /sasl to /shared-tools/sasl before startup):

Each broker pod mounts a PVC at /bitnami/kafka/data for partition log storage.

Exporter pod (present in separated_monitor only; no init containers):

KRaft Consensus

The controller Component forms a dedicated KRaft quorum. Brokers do not participate in the quorum — they only consume the __cluster_metadata log and register themselves with the active controller:

Traffic Routing

Controllers have no client-facing service. Only broker pods get per-pod ClusterIP services:

Clients connect to broker pods only. The controller headless service is for internal Kafka use:

# Bootstrap seed list (all broker per-pod ClusterIP addresses)
{cluster}-kafka-broker-advertised-listener-0.{namespace}.svc.cluster.local:9092,
{cluster}-kafka-broker-advertised-listener-1.{namespace}.svc.cluster.local:9092,
...

Separated ± Monitor

The separated topology provisions only kafka-controller and kafka-broker. The separated_monitor topology additionally provisions the kafka-exporter Component on port 9308 for consumer group and partition metrics.

Kafka 2.x Architecture (kafka2-external-zk)

The kafka2-external-zk topology deploys Kafka 2.7 in the traditional ZooKeeper-based mode. Instead of the KRaft metadata quorum, an external ZooKeeper ensemble (deployed as a separate KubeBlocks cluster or external service) provides cluster coordination, controller election, and topic metadata storage.

Resource Hierarchy

Cluster  →  Component (kafka-broker)    →  InstanceSet  →  Pod × N
         →  Component (kafka-exporter)  →  InstanceSet  →  Pod × 1

The kafka-broker ComponentDefinition for Kafka 2.x declares a serviceRefDeclaration named kafkaZookeeper (required; matches ZooKeeper 3.5–3.9). The Cluster CR must provide a serviceRef pointing to an available ZooKeeper ensemble before the broker component can start:

spec:
  topology: kafka2-external-zk
  serviceRefs:
    - name: kafkaZookeeper
      namespace: <zk-namespace>
      cluster: <zk-cluster-name>

Containers Inside Each Pod

Each broker pod runs two containers. The kafkatool init container copies /sasl to /shared-tools/sasl before startup, as in Kafka 3.x:

Each broker pod mounts a PVC at /bitnami/kafka/data for partition log storage.

ZooKeeper Coordination

In Kafka 2.x, ZooKeeper is responsible for all coordination tasks that KRaft handles in Kafka 3.x:

Traffic Routing

Traffic routing is identical to Kafka 3.x brokers — per-pod ClusterIP services via advertised-listener:

Partition Leader Election

Partition leader election governs data availability and is the core HA mechanism across all Kafka topologies. The active KRaft controller (or ZooKeeper-elected controller in Kafka 2.x) manages all partition leader assignments:

The replication factor for each topic determines how many ISR replicas exist per partition. A replication factor of 3 tolerates 1 broker failure per partition without any data loss or unavailability.

Automatic Failover

When a Kafka node fails, KubeBlocks and Kafka's internal protocols respond as follows:

1. Broker failure — the active controller detects the broker's session timeout and removes it from all ISR lists
2. Partition leader election — for each partition where the failed broker was leader, the controller elects a new leader from the remaining ISR members; clients refreshing metadata see the new leader
3. Producer/consumer continuity — clients encounter a NOT_LEADER_OR_FOLLOWER error, refresh metadata from any available broker, and reconnect to the new partition leaders; brief retries are normal
4. Controller failure (combined topology) — if the active controller's pod fails, the remaining quorum members hold a Raft election and elect a new active controller within seconds
5. Controller failure (separated topology) — controller Component pods independently hold Raft election; broker pods continue serving data traffic without interruption while the quorum recovers
6. KubeBlocks pod recovery — KubeBlocks restarts the failed pod; on startup the broker fetches the current metadata from the active controller, re-joins the cluster, and begins catching up on missed partition data

System Accounts

KubeBlocks automatically creates the following Kafka SASL accounts for the broker and combined components. Credentials are stored in Secrets named {cluster}-{component}-account-{name}.

Both accounts use SCRAM-SHA-256 and SCRAM-SHA-512 for authentication. In Kafka 3.x, credentials are stored in the __cluster_metadata log. In Kafka 2.x (kafka2-external-zk), credentials are provisioned via kafka-configs.sh --zookeeper during the KubeBlocks accountProvision lifecycle action.