Horizontal Scaling for Redis Clusters with KubeBlocks

This guide explains how to perform horizontal scaling (scale-out and scale-in) on a Redis cluster managed by KubeBlocks. You'll learn how to use both OpsRequest and direct Cluster API updates to achieve this.

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Redis Replication Cluster

KubeBlocks uses a declarative approach for managing Redis Replication Clusters. Below is an example configuration for deploying a Redis Replication Cluster with two components, redis and redis sentinel.

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: redis-replication
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: redis
  topology: replication
  componentSpecs:
    - name: redis
      serviceVersion: "7.2.4"
      disableExporter: false
      replicas: 2
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: redis-sentinel
      replicas: 3
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster redis-replication -n demo -w

Expected Output:

NAME                CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
redis-replication   redis                Delete               Running   3m49s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=redis-replication -L  kubeblocks.io/role -n demo

Expected Output:

NAME                                 READY   STATUS    RESTARTS   AGE     ROLE
redis-replication-redis-0            3/3     Running   0          3m38s   primary
redis-replication-redis-1            3/3     Running   0          3m16s   secondary
redis-replication-redis-sentinel-0   2/2     Running   0          4m35s
redis-replication-redis-sentinel-1   2/2     Running   0          4m17s
redis-replication-redis-sentinel-2   2/2     Running   0          3m59s

Once the cluster status becomes Running, your Redis cluster is ready for use.

Scale-out (Add Replicas)

Expected Workflow:

1. New pod is provisioned, and transitions from Pending to Running with secondary role
2. Data synced from primary to new replica
3. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: redis-replication-scale-out-ops
  namespace: demo
spec:
  clusterName: redis-replication
  type: HorizontalScaling
  horizontalScaling:
  - componentName: redis
    # Specifies the replica changes for scaling in components
    scaleOut:
      # Specifies the replica changes for the component.
      # add one more replica to current component
      replicaChanges: 1

kubectl get ops redis-replication-scale-out-ops -n demo -w

NAME                              TYPE                CLUSTER             STATUS    PROGRESS   AGE
redis-replication-scale-out-ops   HorizontalScaling   redis-replication   Running   0/1        9s
redis-replication-scale-out-ops   HorizontalScaling   redis-replication   Running   1/1        20s
redis-replication-scale-out-ops   HorizontalScaling   redis-replication   Succeed   1/1        20s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: redis
      serviceVersion: "7.2.4"
      disableExporter: false
      replicas: 3 # increase replicas to scale-out
...

kubectl patch cluster redis-replication -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 3}]'

Verify Scale-Out

After applying the operation, you will see a new pod created and the Redis cluster status goes from Updating to Running, and the newly created pod has a new role secondary.

kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication

Example Output (3 Pods):

NAME                                 READY   STATUS    RESTARTS   AGE
redis-replication-redis-0            3/3     Running   0          9m47s
redis-replication-redis-1            3/3     Running   0          10m
redis-replication-redis-2            3/3     Running   0          4m48s
redis-replication-redis-sentinel-0   2/2     Running   0          16m
redis-replication-redis-sentinel-1   2/2     Running   0          16m
redis-replication-redis-sentinel-2   2/2     Running   0          17m

New replicas automatically join as secondary nodes.

kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication -L kubeblocks.io/role

Example Output:

NAME                                 READY   STATUS    RESTARTS   AGE     ROLE
redis-replication-redis-0            3/3     Running   0          10m     secondary
redis-replication-redis-1            3/3     Running   0          11m     primary
redis-replication-redis-2            3/3     Running   0          5m27s   secondary
redis-replication-redis-sentinel-0   2/2     Running   0          17m
redis-replication-redis-sentinel-1   2/2     Running   0          17m
redis-replication-redis-sentinel-2   2/2     Running   0          17m

Scale-in (Remove Replicas)

Expected Workflow:

1. Selected replica (the one with the largest ordinal) is removed
2. If removing a primary replica, automatic switchover occurs first
3. Pod is terminated gracefully
4. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: redis-replication-scale-in-ops
  namespace: demo
spec:
  clusterName: redis-replication
  type: HorizontalScaling
  horizontalScaling:
  - componentName: redis
    # Specifies the replica changes for scaling in components
    scaleIn:
      # Specifies the replica changes for the component.
      # remove one replica from current component
      replicaChanges: 1

kubectl get ops redis-replication-scale-in-ops -n demo -w

NAME                             TYPE                CLUSTER             STATUS    PROGRESS   AGE
redis-replication-scale-in-ops   HorizontalScaling   redis-replication   Running   0/1        8s
redis-replication-scale-in-ops   HorizontalScaling   redis-replication   Running   1/1        24s
redis-replication-scale-in-ops   HorizontalScaling   redis-replication   Succeed   1/1        24s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: redis
      serviceVersion: "7.2.4"
      disableExporter: false
      replicas: 1 # decrease replicas to scale-out

kubectl patch cluster redis-replication -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 1}]'

Verify Scale-In

Example Output (ONE Pod):

kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication,apps.kubeblocks.io/component-name=redis
NAME                        READY   STATUS    RESTARTS   AGE
redis-replication-redis-0   3/3     Running   0          16m

Troubleshooting

If the scale-in operation gets stucked for quite a long time, please check these resources:

# Check agent logs on both current primary and candidate
kubectl logs -n demo <primary-pod> -c kbagent
kubectl logs -n demo <candidate-pod> -c kbagent

# Check cluster events for errors
kubectl get events -n demo --field-selector involvedObject.name=pg-cluster

# Check kubeblocks logs
kubectl -n kb-system logs deploy/kubeblocks

If you get errors like the following from the primary replica:

INFO	Action Executed	{"action": "switchover", "result": "exit code: 1: failed"}
INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 7}

It could be a switchover error, and please check KubeBlocks logs for more details.

Best Practices

When performing horizontal scaling:

• Scale during low-traffic periods when possible
• Monitor cluster health during scaling operations
• Verify sufficient resources exist before scaling out
• Consider storage requirements for new replicas

Cleanup

To remove all created resources, delete the Redis cluster along with its namespace:

kubectl delete cluster redis-replication -n demo
kubectl delete ns demo

Summary

In this guide you learned how to:

• Perform scale-out operations to add replicas to a Redis cluster.
• Perform scale-in operations to remove replicas from a Redis cluster.
• Use both OpsRequest and direct Cluster API updates for horizontal scaling.

KubeBlocks ensures seamless scaling with minimal disruption to your database operations. with minimal disruption to your database operations.