KubeBlocks
BlogsEnterprise
⌘K
​
Blogs

Overview
Quickstart
Architecture

Operations

Lifecycle Management
Vertical Scaling
Horizontal Scaling
Volume Expansion
Switchover
Minor Version Upgrade
Manage Services

Backup & Restore

Backup
Restore

Observability

Observability for etcd Clusters
  1. Prerequisites
  2. Scale-out (Add Replicas)
    1. Verify Scale-Out
  3. Scale-in (Remove Replicas)
    1. Verify Scale-In
  4. Cleanup

Horizontal Scaling for etcd Clusters with KubeBlocks

This guide explains how to perform horizontal scaling (scale-out and scale-in) on an etcd cluster managed by KubeBlocks.

NOTE

etcd Quorum Requirements

etcd uses the Raft consensus algorithm. For correct operation:

  • Always maintain an odd number of nodes (3, 5, 7, etc.)
  • A cluster of N nodes tolerates at most ⌊N/2⌋ failures
  • Minimum recommended cluster size is 3 nodes
CAUTION

Known Limitation

After a scale-out operation completes, the KubeBlocks component controller may loop on the memberJoin health check, leaving the cluster in Updating state indefinitely even though the new member is fully started and functional. This is a known issue (#2541).

The cluster remains functional and accessible during this state. To restore the Running status, delete the cluster and recreate it at the desired replica count.

Prerequisites

    Before proceeding, verify your environment meets these requirements:

    • A functional Kubernetes cluster (v1.21+ recommended)
    • kubectl v1.21+ installed and configured with cluster access
    • Helm installed (installation guide)
    • KubeBlocks installed (installation guide)
    • etcd Add-on installed and an etcd cluster running (see Quickstart)

    Scale-out (Add Replicas)

    Expected Workflow:

    1. New pods are provisioned
    2. New members join the etcd cluster via etcdctl member add
    3. Data is replicated to new members
    4. OpsRequest transitions to Succeed

    Scale out the etcd cluster by adding 1 replica (from 3 to 4):

    apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: etcd-scale-out
  namespace: demo
spec:
  clusterName: etcd-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: etcd
    scaleOut:
      replicaChanges: 1

    Apply it:

    kubectl apply -f https://raw.githubusercontent.com/apecloud/kubeblocks-addons/refs/heads/main/examples/etcd/scale-out.yaml

    Monitor the progress:

    kubectl get ops etcd-scale-out -n demo -w
    Example Output
    NAME             TYPE                CLUSTER        STATUS    PROGRESS   AGE
etcd-scale-out   HorizontalScaling   etcd-cluster   Running   0/1        10s
etcd-scale-out   HorizontalScaling   etcd-cluster   Succeed   1/1        78s

    Update the replicas field in the Cluster resource:

    kubectl patch cluster etcd-cluster -n demo --type=json \
  -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 4}]'

    Verify Scale-Out

    kubectl get pods -n demo -l app.kubernetes.io/instance=etcd-cluster -L kubeblocks.io/role
    Example Output (4 Pods)
    NAME                  READY   STATUS    RESTARTS   AGE    ROLE
etcd-cluster-etcd-0   2/2     Running   0          5m     follower
etcd-cluster-etcd-1   2/2     Running   0          5m     follower
etcd-cluster-etcd-2   2/2     Running   0          5m     leader
etcd-cluster-etcd-3   2/2     Running   0          78s    follower

    Verify the new member is registered in the etcd cluster:

    kubectl exec -n demo etcd-cluster-etcd-0 -c etcd -- \
  etcdctl member list --endpoints=http://localhost:2379
    Example Output
    ..., started, etcd-cluster-etcd-0, ...
..., started, etcd-cluster-etcd-1, ...
..., started, etcd-cluster-etcd-2, ...
..., started, etcd-cluster-etcd-3, ...

    Scale-in (Remove Replicas)

    Expected Workflow:

    1. The member with the largest ordinal is removed from the etcd cluster via etcdctl member remove
    2. The corresponding pod is terminated
    3. OpsRequest transitions to Succeed

    Scale in the etcd cluster by removing 1 replica:

    apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: etcd-scale-in
  namespace: demo
spec:
  clusterName: etcd-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: etcd
    scaleIn:
      replicaChanges: 1

    Apply it:

    kubectl apply -f https://raw.githubusercontent.com/apecloud/kubeblocks-addons/refs/heads/main/examples/etcd/scale-in.yaml

    Monitor progress:

    kubectl get ops etcd-scale-in -n demo -w
    Example Output
    NAME            TYPE                CLUSTER        STATUS    PROGRESS   AGE
etcd-scale-in   HorizontalScaling   etcd-cluster   Running   0/1        10s
etcd-scale-in   HorizontalScaling   etcd-cluster   Succeed   1/1        25s

    kubectl patch cluster etcd-cluster -n demo --type=json \
  -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 3}]'

    Verify Scale-In

    kubectl get pods -n demo -l app.kubernetes.io/instance=etcd-cluster -L kubeblocks.io/role
    Example Output (3 Pods)
    NAME                  READY   STATUS    RESTARTS   AGE   ROLE
etcd-cluster-etcd-0   2/2     Running   0          8m    follower
etcd-cluster-etcd-1   2/2     Running   0          8m    follower
etcd-cluster-etcd-2   2/2     Running   0          8m    leader

    Cleanup

    kubectl delete cluster etcd-cluster -n demo
kubectl delete ns demo

    © 2026 KUBEBLOCKS INC