Manage Kafka Services Using the Declarative Cluster API in KubeBlocks

This guide provides step-by-step instructions for exposing Kafka services managed by KubeBlocks, both externally and internally. You'll learn to configure external access using cloud provider LoadBalancer services, manage internal services, and properly disable external exposure when no longer needed.

Prerequisites

Before proceeding, ensure the following:

Environment Setup:
- A Kubernetes cluster is up and running.
- The kubectl CLI tool is configured to communicate with your cluster.
- KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Kafka Cluster

KubeBlocks uses a declarative approach for managing Kafka Clusters. Below is an example configuration for deploying a Kafka Cluster with 3 components

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  topology: separated_monitor
  componentSpecs:
    - name: kafka-broker
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "true"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-controller
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-exporter
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "1Gi"
        requests:
          cpu: "0.1"
          memory: "0.2Gi"

NOTE

These three components will be created strictly in controller->broker->exporter order as defined in ClusterDefinition.

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster kafka-separated-cluster -n demo -w

Expected Output:

kubectl get cluster kafka-separated-cluster -n demo
NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka-separated-cluster   kafka                Delete               Creating   13s
kafka-separated-cluster   kafka                Delete               Running    63s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Expected Output:

NAME                                         READY   STATUS    RESTARTS   AGE
kafka-separated-cluster-kafka-broker-0       2/2     Running   0          13m
kafka-separated-cluster-kafka-controller-0   2/2     Running   0          13m
kafka-separated-cluster-kafka-exporter-0     1/1     Running   0          12m

Once the cluster status becomes Running, your Kafka cluster is ready for use.

TIP

If you are creating the cluster for the very first time, it may take some time to pull images before running.

View Network Services

List the Services created for the Kafka cluster:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Example Services:

NAME                                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kafka-separated-cluster-kafka-broker-advertised-listener-0   ClusterIP   10.96.101.247   <none>        9092/TCP   19m

Expose Kafka Service

External service addresses enable public internet access to Kafka, while internal service addresses restrict access to the user's VPC.

Service Types Comparison

Type	Use Case	Cloud Cost	Security
ClusterIP	Internal service communication	Free	Highest
NodePort	Development/testing	Low	Moderate
LoadBalancer	Production external access	High	Managed via security groups

Option 1: Using OpsRequest

To expose the Kafka service externally using a LoadBalancer, create an OpsRequest resource:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: kafka-separated-cluster-expose-enable-ops
  namespace: demo
spec:
  type: Expose
  clusterName: kafka-separated-cluster
  expose:
  - componentName: kafka-broker
    services:
    - name: internet
      # Determines how the Service is exposed. Defaults to 'ClusterIP'.
      # Valid options are 'ClusterIP', 'NodePort', and 'LoadBalancer'.
      serviceType: LoadBalancer
      ports:
        - name: kafka-client
          port: 9092
          targetPort: kafka-client
      # Contains cloud provider related parameters if ServiceType is LoadBalancer.
      # Following is an example for AWS EKS
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # or "true" for an internal VPC IP
    switch: Enable

Wait for the OpsRequest to complete:

kubectl get ops kafka-separated-cluster-expose-enable-ops -n demo

Example Output:

NAME                                        TYPE     CLUSTER                   STATUS    PROGRESS   AGE
kafka-separated-cluster-expose-enable-ops   Expose   kafka-separated-cluster   Succeed   1/1        31s

Option 2: Using Cluster API

Alternatively, update the spec.services section in the Cluster resource to include a LoadBalancer service:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  # expose a external service
  services:
    - annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb  # Use Network Load Balancer
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # or "true" for an internal VPC IP
      componentSelector: kafka-broker
      name: kafka-internet
      serviceName: kafka-internet
      spec:  # defines the behavior of a K8s service.
        ipFamilyPolicy: PreferDualStack
        ports:
        - name: kafka-client
          # port to expose
          port: 9092 # port 15672 for kafka management console
          protocol: TCP
          targetPort: kafka-client
        type: LoadBalancer
  componentSpecs:
  ...

The YAML configuration above adds a new external service under the services section. This LoadBalancer service includes annotations for AWS Network Load Balancer (NLB).

NOTE

Cloud Provider Annotations

When using a LoadBalancer service, you must include the appropriate annotations specific to your cloud provider. Below is a list of commonly used annotations for different cloud providers:

service.beta.kubernetes.io/aws-load-balancer-type: nlb  # Use Network Load Balancer
service.beta.kubernetes.io/aws-load-balancer-internal: "true"  # Use "false" for internet-facing LoadBalancer

Azure

service.beta.kubernetes.io/azure-load-balancer-internal: "true" # Use "false" for internet-facing LoadBalancer

networking.gke.io/load-balancer-type: "Internal"  # Restricts the LoadBalancer to internal VPC access only. Defaults to internet-facing if not specified.
cloud.google.com/l4-rbs: "enabled" # Optimization for internet-facing LoadBalancer

Alibaba Cloud

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"  # Use "intranet" for internal-facing LoadBalancer

NOTE

The service.beta.kubernetes.io/aws-load-balancer-internal annotation controls whether the LoadBalancer is internal or internet-facing. Note that this annotation cannot be modified dynamically after service creation.

  service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Use "true" for internal VPC IPs

If you change this annotation from "false" to "true" after the Service is created, the annotation may update in the Service object, but the LoadBalancer will still retain its public IP.

To properly modify this behavior:

First, delete the existing LoadBalancer service.
Recreate the service with the updated annotation (service.beta.kubernetes.io/aws-load-balancer-internal: "true").
Wait for the new LoadBalancer to be provisioned with the correct internal or external IP.

Wait for the Cluster status to transition to Running using the following command:

kubectl get cluster kafka-separated-cluster -n demo -w

NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-separated-cluster   kafka                Delete               Running   18m

Verify the Exposed Service

Check the service details to confirm the LoadBalancer service is created:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Example Output:

NAME                                                         TYPE            CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kafka-separated-cluster-kafka-broker-advertised-listener-0   ClusterIP       10.96.101.247   <none>           9092/TCP         24m
kafka-separated-cluster-kafka-broker-internet                LoadBalancer    10.96.180.189   <EXTERNAL_IP>    9092:31243/TCP   59s

Disable External Exposure

Option 1: Using OpsRequest

To disable external access, create an OpsRequest:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: kafka-separated-cluster-expose-disable-ops
  namespace: demo
spec:
  clusterName: kafka-separated-cluster
  expose:
  - componentName: kafka-broker
    services:
    - name: internet
      serviceType: LoadBalancer
    switch: Disable
  preConditionDeadlineSeconds: 0
  type: Expose

Wait for the OpsRequest to complete:

kubectl get ops kafka-separated-cluster-expose-disable-ops -n demo

Example Output:

NAME                                         TYPE     CLUSTER                   STATUS    PROGRESS   AGE
kafka-separated-cluster-expose-disable-ops   Expose   kafka-separated-cluster   Succeed   1/1        16s

Option 2: Using Cluster API

Alternatively, remove the spec.services field from the Cluster resource:

kubectl patch cluster kafka-separated-cluster -n demo --type=json -p='[
  {
    "op": "remove",
    "path": "/spec/services"
  }
]'

Monitor the cluster status until it is Running:

kubectl get cluster kafka-separated-cluster -n demo -w

NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-separated-cluster   kafka                Delete               Running   26m

Verify Service Removal

Ensure that the 'kafka-separated-cluster-kafka-internet' Service is removed:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Expected Result: The 'kafka-separated-cluster-kafka-internet' Service should be removed.

Cleanup

To remove all created resources, delete the Kafka cluster along with its namespace:

kubectl delete cluster kafka-separated-cluster -n demo
kubectl delete ns demo

Summary

This guide demonstrated how to:

Expose a Kafka service externally or internally using KubeBlocks.
Configure LoadBalancer services with cloud provider-specific annotations.
Manage external access by enabling or disabling services via OpsRequest or direct updates to the Cluster API.

KubeBlocks provides flexibility and simplicity for managing MySQL services in Kubernetes environments. simplicity for managing Kafka services in Kubernetes environments.

Manage Kafka Services Using the Declarative Cluster API in KubeBlocks

Prerequisites

Before proceeding, ensure the following:

Environment Setup:
- A Kubernetes cluster is up and running.
- The kubectl CLI tool is configured to communicate with your cluster.
- KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Kafka Cluster

KubeBlocks uses a declarative approach for managing Kafka Clusters. Below is an example configuration for deploying a Kafka Cluster with 3 components

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  topology: separated_monitor
  componentSpecs:
    - name: kafka-broker
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "true"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-controller
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-exporter
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "1Gi"
        requests:
          cpu: "0.1"
          memory: "0.2Gi"

NOTE

These three components will be created strictly in controller->broker->exporter order as defined in ClusterDefinition.

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster kafka-separated-cluster -n demo -w

Expected Output:

kubectl get cluster kafka-separated-cluster -n demo
NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka-separated-cluster   kafka                Delete               Creating   13s
kafka-separated-cluster   kafka                Delete               Running    63s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Expected Output:

NAME                                         READY   STATUS    RESTARTS   AGE
kafka-separated-cluster-kafka-broker-0       2/2     Running   0          13m
kafka-separated-cluster-kafka-controller-0   2/2     Running   0          13m
kafka-separated-cluster-kafka-exporter-0     1/1     Running   0          12m

Once the cluster status becomes Running, your Kafka cluster is ready for use.

TIP

If you are creating the cluster for the very first time, it may take some time to pull images before running.

View Network Services

List the Services created for the Kafka cluster:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Example Services:

NAME                                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kafka-separated-cluster-kafka-broker-advertised-listener-0   ClusterIP   10.96.101.247   <none>        9092/TCP   19m

Expose Kafka Service

External service addresses enable public internet access to Kafka, while internal service addresses restrict access to the user's VPC.

Service Types Comparison

Type	Use Case	Cloud Cost	Security
ClusterIP	Internal service communication	Free	Highest
NodePort	Development/testing	Low	Moderate
LoadBalancer	Production external access	High	Managed via security groups

Option 1: Using OpsRequest

To expose the Kafka service externally using a LoadBalancer, create an OpsRequest resource:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: kafka-separated-cluster-expose-enable-ops
  namespace: demo
spec:
  type: Expose
  clusterName: kafka-separated-cluster
  expose:
  - componentName: kafka-broker
    services:
    - name: internet
      # Determines how the Service is exposed. Defaults to 'ClusterIP'.
      # Valid options are 'ClusterIP', 'NodePort', and 'LoadBalancer'.
      serviceType: LoadBalancer
      ports:
        - name: kafka-client
          port: 9092
          targetPort: kafka-client
      # Contains cloud provider related parameters if ServiceType is LoadBalancer.
      # Following is an example for AWS EKS
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # or "true" for an internal VPC IP
    switch: Enable

Wait for the OpsRequest to complete:

kubectl get ops kafka-separated-cluster-expose-enable-ops -n demo

Example Output:

NAME                                        TYPE     CLUSTER                   STATUS    PROGRESS   AGE
kafka-separated-cluster-expose-enable-ops   Expose   kafka-separated-cluster   Succeed   1/1        31s

Option 2: Using Cluster API

Alternatively, update the spec.services section in the Cluster resource to include a LoadBalancer service:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  # expose a external service
  services:
    - annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb  # Use Network Load Balancer
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # or "true" for an internal VPC IP
      componentSelector: kafka-broker
      name: kafka-internet
      serviceName: kafka-internet
      spec:  # defines the behavior of a K8s service.
        ipFamilyPolicy: PreferDualStack
        ports:
        - name: kafka-client
          # port to expose
          port: 9092 # port 15672 for kafka management console
          protocol: TCP
          targetPort: kafka-client
        type: LoadBalancer
  componentSpecs:
  ...

The YAML configuration above adds a new external service under the services section. This LoadBalancer service includes annotations for AWS Network Load Balancer (NLB).

NOTE

Cloud Provider Annotations

When using a LoadBalancer service, you must include the appropriate annotations specific to your cloud provider. Below is a list of commonly used annotations for different cloud providers:

service.beta.kubernetes.io/aws-load-balancer-type: nlb  # Use Network Load Balancer
service.beta.kubernetes.io/aws-load-balancer-internal: "true"  # Use "false" for internet-facing LoadBalancer

Azure

service.beta.kubernetes.io/azure-load-balancer-internal: "true" # Use "false" for internet-facing LoadBalancer

networking.gke.io/load-balancer-type: "Internal"  # Restricts the LoadBalancer to internal VPC access only. Defaults to internet-facing if not specified.
cloud.google.com/l4-rbs: "enabled" # Optimization for internet-facing LoadBalancer

Alibaba Cloud

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "internet"  # Use "intranet" for internal-facing LoadBalancer

NOTE

  service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # Use "true" for internal VPC IPs

If you change this annotation from "false" to "true" after the Service is created, the annotation may update in the Service object, but the LoadBalancer will still retain its public IP.

To properly modify this behavior:

First, delete the existing LoadBalancer service.
Recreate the service with the updated annotation (service.beta.kubernetes.io/aws-load-balancer-internal: "true").
Wait for the new LoadBalancer to be provisioned with the correct internal or external IP.

Wait for the Cluster status to transition to Running using the following command:

kubectl get cluster kafka-separated-cluster -n demo -w

NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-separated-cluster   kafka                Delete               Running   18m

Verify the Exposed Service

Check the service details to confirm the LoadBalancer service is created:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Example Output:

NAME                                                         TYPE            CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
kafka-separated-cluster-kafka-broker-advertised-listener-0   ClusterIP       10.96.101.247   <none>           9092/TCP         24m
kafka-separated-cluster-kafka-broker-internet                LoadBalancer    10.96.180.189   <EXTERNAL_IP>    9092:31243/TCP   59s

Disable External Exposure

Option 1: Using OpsRequest

To disable external access, create an OpsRequest:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: kafka-separated-cluster-expose-disable-ops
  namespace: demo
spec:
  clusterName: kafka-separated-cluster
  expose:
  - componentName: kafka-broker
    services:
    - name: internet
      serviceType: LoadBalancer
    switch: Disable
  preConditionDeadlineSeconds: 0
  type: Expose

Wait for the OpsRequest to complete:

kubectl get ops kafka-separated-cluster-expose-disable-ops -n demo

Example Output:

NAME                                         TYPE     CLUSTER                   STATUS    PROGRESS   AGE
kafka-separated-cluster-expose-disable-ops   Expose   kafka-separated-cluster   Succeed   1/1        16s

Option 2: Using Cluster API

Alternatively, remove the spec.services field from the Cluster resource:

kubectl patch cluster kafka-separated-cluster -n demo --type=json -p='[
  {
    "op": "remove",
    "path": "/spec/services"
  }
]'

Monitor the cluster status until it is Running:

kubectl get cluster kafka-separated-cluster -n demo -w

NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-separated-cluster   kafka                Delete               Running   26m

Verify Service Removal

Ensure that the 'kafka-separated-cluster-kafka-internet' Service is removed:

kubectl get service -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Expected Result: The 'kafka-separated-cluster-kafka-internet' Service should be removed.

Cleanup

To remove all created resources, delete the Kafka cluster along with its namespace:

kubectl delete cluster kafka-separated-cluster -n demo
kubectl delete ns demo

Summary

This guide demonstrated how to:

Expose a Kafka service externally or internally using KubeBlocks.
Configure LoadBalancer services with cloud provider-specific annotations.
Manage external access by enabling or disabling services via OpsRequest or direct updates to the Cluster API.

KubeBlocks provides flexibility and simplicity for managing MySQL services in Kubernetes environments. simplicity for managing Kafka services in Kubernetes environments.