Vertical Scaling for Kafka Clusters with KubeBlocks

This guide demonstrates how to vertically scale a Kafka Cluster managed by KubeBlocks by adjusting compute resources (CPU and memory) while maintaining the same number of replicas.

Vertical scaling modifies compute resources (CPU and memory) for Kafka instances while maintaining replica count. Key characteristics:

• Non-disruptive: When properly configured, maintains availability during scaling
• Granular: Adjust CPU, memory, or both independently
• Reversible: Scale up or down as needed

KubeBlocks ensures minimal impact during scaling operations by following a controlled, role-aware update strategy: Role-Aware Replicas (Primary/Secondary Replicas)

• Secondary replicas update first – Non-leader pods are upgraded to minimize disruption.
• Primary updates last – Only after all secondaries are healthy does the primary pod restart.
• Cluster state progresses from Updating → Running once all replicas are stable.

Role-Unaware Replicas (Ordinal-Based Scaling) If replicas have no defined roles, updates follow Kubernetes pod ordinal order:

• Highest ordinal first (e.g., pod-2 → pod-1 → pod-0) to ensure deterministic rollouts.

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Kafka Cluster

KubeBlocks uses a declarative approach for managing Kafka Clusters. Below is an example configuration for deploying a Kafka Cluster with 3 components

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-separated-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  topology: separated_monitor
  componentSpecs:
    - name: kafka-broker
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "true"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-controller
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-exporter
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "1Gi"
        requests:
          cpu: "0.1"
          memory: "0.2Gi"

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster kafka-separated-cluster -n demo -w

Expected Output:

kubectl get cluster kafka-separated-cluster -n demo
NAME                      CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka-separated-cluster   kafka                Delete               Creating   13s
kafka-separated-cluster   kafka                Delete               Running    63s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=kafka-separated-cluster -n demo

Expected Output:

NAME                                         READY   STATUS    RESTARTS   AGE
kafka-separated-cluster-kafka-broker-0       2/2     Running   0          13m
kafka-separated-cluster-kafka-controller-0   2/2     Running   0          13m
kafka-separated-cluster-kafka-exporter-0     1/1     Running   0          12m

Once the cluster status becomes Running, your Kafka cluster is ready for use.

Vertical Scale

Expected Workflow:

1. Pods are updated in pod ordinal order, from highest to lowest, (e.g., pod-2 → pod-1 → pod-0)
2. Cluster status transitions from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: kafka-separated-cluster-vscale-ops
  namespace: demo
spec:
  clusterName: kafka-separated-cluster
  type: VerticalScaling
  verticalScaling:
  - componentName: kafka-broker
    requests:
      cpu: '1'
      memory: 1Gi
    limits:
      cpu: '1'
      memory: 1Gi

kubectl -n demo get ops kafka-separated-cluster-vscale-ops -w

NAME                                 TYPE              CLUSTER                   STATUS    PROGRESS   AGE
kafka-separated-cluster-vscale-ops   VerticalScaling   kafka-separated-cluster   Running   0/1        12s
kafka-separated-cluster-vscale-ops   VerticalScaling   kafka-separated-cluster   Running   1/1        13s
kafka-separated-cluster-vscale-ops   VerticalScaling   kafka-separated-cluster   Running   1/1        13s
kafka-separated-cluster-vscale-ops   VerticalScaling   kafka-separated-cluster   Succeed   1/1        13s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: kafka-broker
      replicas: 1
      resources:
        requests:
          cpu: "1"       # Update the resources to your need.
          memory: "1Gi"  # Update the resources to your need.
        limits:
          cpu: "1"       # Update the resources to your need.
          memory: "1Gi"  # Update the resources to your need.
  ...

Best Practices & Considerations

Planning:

• Scale during maintenance windows or low-traffic periods
• Verify Kubernetes cluster has sufficient resources
• Check for any ongoing operations before starting

Execution:

• Maintain balanced CPU-to-Memory ratios
• Set identical requests/limits for guaranteed QoS

Post-Scaling:

• Monitor resource utilization and application performance
• Consider adjusting Kafka parameters if needed

Verification

Verify the updated resources by inspecting the cluster configuration or Pod details:

kbcli cluster describe kafka-separated-cluster -n demo

Expected Output:

Resources Allocation:
COMPONENT          INSTANCE-TEMPLATE   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
kafka-broker                           1 / 1                1Gi / 1Gi               data:20Gi      <none>

Key Benefits of Vertical Scaling with KubeBlocks

• Seamless Scaling: Pods are recreated in a specific order to ensure minimal disruption.
• Dynamic Resource Adjustments: Easily scale CPU and memory based on workload requirements.
• Flexibility: Choose between OpsRequest for dynamic scaling or direct API updates for precise control.
• Improved Availability: The cluster remains operational during the scaling process, maintaining high availability.

Cleanup

To remove all created resources, delete the Kafka Cluster along with its namespace:

kubectl delete cluster kafka-separated-cluster -n demo
kubectl delete ns demo

Summary

In this guide, you learned how to:

1. Deploy a Kafka Cluster managed by KubeBlocks.
2. Perform vertical scaling by increasing or decreasing resources for the kafka component.
3. Use both OpsRequest and direct Cluster API updates to adjust resource allocations.

Vertical scaling is a powerful tool for optimizing resource utilization and adapting to changing workload demands, ensuring your Kafka Cluster remains performant and resilient.