Operations
Backup And Restores
Custom Secret
Monitoring
tpl
Horizontal Scaling for Redis Clusters with KubeBlocks
This guide explains how to perform horizontal scaling (scale-out and scale-in) on a Redis cluster managed by KubeBlocks. You'll learn how to use both OpsRequest and direct Cluster API updates to achieve this.
Prerequisites
Before proceeding, ensure the following:
- Environment Setup:
- A Kubernetes cluster is up and running.
- The kubectl CLI tool is configured to communicate with your cluster.
- KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
- Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:
kubectl create ns demo
namespace/demo created
Deploy a Redis Replication Cluster
KubeBlocks uses a declarative approach for managing Redis Replication Clusters. Below is an example configuration for deploying a Redis Replication Cluster with two components, redis and redis sentinel.
Apply the following YAML configuration to deploy the cluster:
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
name: redis-replication
namespace: demo
spec:
terminationPolicy: Delete
clusterDef: redis
topology: replication
componentSpecs:
- name: redis
serviceVersion: "7.2.4"
disableExporter: false
replicas: 2
resources:
limits:
cpu: '0.5'
memory: 0.5Gi
requests:
cpu: '0.5'
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName: ""
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
- name: redis-sentinel
replicas: 3
resources:
limits:
cpu: '0.5'
memory: 0.5Gi
requests:
cpu: '0.5'
memory: 0.5Gi
volumeClaimTemplates:
- name: data
spec:
storageClassName: ""
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
Verifying the Deployment
Monitor the cluster status until it transitions to the Running state:
kubectl get cluster redis-replication -n demo -w
Expected Output:
NAME CLUSTER-DEFINITION TERMINATION-POLICY STATUS AGE
redis-replication redis Delete Running 3m49s
Check the pod status and roles:
kubectl get pods -l app.kubernetes.io/instance=redis-replication -L kubeblocks.io/role -n demo
Expected Output:
NAME READY STATUS RESTARTS AGE ROLE
redis-replication-redis-0 3/3 Running 0 3m38s primary
redis-replication-redis-1 3/3 Running 0 3m16s secondary
redis-replication-redis-sentinel-0 2/2 Running 0 4m35s
redis-replication-redis-sentinel-1 2/2 Running 0 4m17s
redis-replication-redis-sentinel-2 2/2 Running 0 3m59s
Once the cluster status becomes Running, your Redis cluster is ready for use.
If you are creating the cluster for the very first time, it may take some time to pull images before running.
Scale-out (Add Replicas)
Expected Workflow:
- New pod is provisioned, and transitions from
Pending
toRunning
withsecondary
role - Data synced from primary to new replica
- Cluster status changes from
Updating
toRunning
Option 1: Using Horizontal Scaling OpsRequest
Scale out the Redis cluster by adding 1 replica to redis component:
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
name: redis-replication-scale-out-ops
namespace: demo
spec:
clusterName: redis-replication
type: HorizontalScaling
horizontalScaling:
- componentName: redis
# Specifies the replica changes for scaling in components
scaleOut:
# Specifies the replica changes for the component.
# add one more replica to current component
replicaChanges: 1
Monitor the progress of the scaling operation:
kubectl get ops redis-replication-scale-out-ops -n demo -w
Expected Result:
NAME TYPE CLUSTER STATUS PROGRESS AGE
redis-replication-scale-out-ops HorizontalScaling redis-replication Running 0/1 9s
redis-replication-scale-out-ops HorizontalScaling redis-replication Running 1/1 20s
redis-replication-scale-out-ops HorizontalScaling redis-replication Succeed 1/1 20s
Option 2: Direct Cluster API Update
Alternatively, you can perform a direct update to the replicas
field in the Cluster resource:
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
componentSpecs:
- name: redis
serviceVersion: "7.2.4"
disableExporter: false
replicas: 3 # increase replicas to scale-out
...
Or you can patch the cluster CR with command:
kubectl patch cluster redis-replication -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 3}]'
Verify Scale-Out
After applying the operation, you will see a new pod created and the Redis cluster status goes from Updating
to Running
, and the newly created pod has a new role secondary
.
kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication
Example Output (3 Pods):
NAME READY STATUS RESTARTS AGE
redis-replication-redis-0 3/3 Running 0 9m47s
redis-replication-redis-1 3/3 Running 0 10m
redis-replication-redis-2 3/3 Running 0 4m48s
redis-replication-redis-sentinel-0 2/2 Running 0 16m
redis-replication-redis-sentinel-1 2/2 Running 0 16m
redis-replication-redis-sentinel-2 2/2 Running 0 17m
New replicas automatically join as secondary nodes.
kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication -L kubeblocks.io/role
Example Output:
NAME READY STATUS RESTARTS AGE ROLE
redis-replication-redis-0 3/3 Running 0 10m secondary
redis-replication-redis-1 3/3 Running 0 11m primary
redis-replication-redis-2 3/3 Running 0 5m27s secondary
redis-replication-redis-sentinel-0 2/2 Running 0 17m
redis-replication-redis-sentinel-1 2/2 Running 0 17m
redis-replication-redis-sentinel-2 2/2 Running 0 17m
Scale-in (Remove Replicas)
Expected Workflow:
- Selected replica (the one with the largest ordinal) is removed
- If removing a primary replica, automatic switchover occurs first
- Pod is terminated gracefully
- Cluster status changes from
Updating
toRunning
If the replica being scaled-in happens to be a primary replica, KubeBlocks will trigger a Switchover actions. And this pod will not be terminated until this Switchover action succeeds.
Option 1: Using Horizontal Scaling OpsRequest
Scale in the Redis cluster by removing ONE replica:
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
name: redis-replication-scale-in-ops
namespace: demo
spec:
clusterName: redis-replication
type: HorizontalScaling
horizontalScaling:
- componentName: redis
# Specifies the replica changes for scaling in components
scaleIn:
# Specifies the replica changes for the component.
# remove one replica from current component
replicaChanges: 1
Monitor progress:
kubectl get ops redis-replication-scale-in-ops -n demo -w
Expected Result:
NAME TYPE CLUSTER STATUS PROGRESS AGE
redis-replication-scale-in-ops HorizontalScaling redis-replication Running 0/1 8s
redis-replication-scale-in-ops HorizontalScaling redis-replication Running 1/1 24s
redis-replication-scale-in-ops HorizontalScaling redis-replication Succeed 1/1 24s
Option 2: Direct Cluster API Update
Alternatively, you can perform a direct update to the replicas
field in the Cluster resource:
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
componentSpecs:
- name: redis
serviceVersion: "7.2.4"
disableExporter: false
replicas: 1 # decrease replicas to scale-out
Or you can patch the cluster CR with command:
kubectl patch cluster redis-replication -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 1}]'
Verify Scale-In
Example Output (ONE Pod):
kubectl get pods -n demo -l app.kubernetes.io/instance=redis-replication,apps.kubeblocks.io/component-name=redis
NAME READY STATUS RESTARTS AGE
redis-replication-redis-0 3/3 Running 0 16m
Troubleshooting
If the scale-in operation gets stucked for quite a long time, please check these resources:
# Check agent logs on both current primary and candidate
kubectl logs -n demo <primary-pod> -c kbagent
kubectl logs -n demo <candidate-pod> -c kbagent
# Check cluster events for errors
kubectl get events -n demo --field-selector involvedObject.name=pg-cluster
# Check kubeblocks logs
kubectl -n kb-system logs deploy/kubeblocks
If you get errors like the following from the primary replica:
INFO Action Executed {"action": "switchover", "result": "exit code: 1: failed"}
INFO HTTP API Called {"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 7}
It could be a switchover error, and please check KubeBlocks logs for more details.
Best Practices
When performing horizontal scaling:
- Scale during low-traffic periods when possible
- Monitor cluster health during scaling operations
- Verify sufficient resources exist before scaling out
- Consider storage requirements for new replicas
Cleanup
To remove all created resources, delete the Redis cluster along with its namespace:
kubectl delete cluster redis-replication -n demo
kubectl delete ns demo
Summary
In this guide you learned how to:
- Perform scale-out operations to add replicas to a Redis cluster.
- Perform scale-in operations to remove replicas from a Redis cluster.
- Use both OpsRequest and direct Cluster API updates for horizontal scaling.
KubeBlocks ensures seamless scaling with minimal disruption to your database operations. with minimal disruption to your database operations.