Horizontal Scaling for Elasticsearch Clusters with KubeBlocks

This guide explains how to perform horizontal scaling (scale-out and scale-in) on a Elasticsearch cluster managed by KubeBlocks. You'll learn how to use both OpsRequest and direct Cluster API updates to achieve this.

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Elasticsearch Cluster

KubeBlocks uses a declarative approach for managing Elasticsearch Clusters. Below is an example configuration for deploying a Elasticsearch Cluster with create a cluster with replicas for different roles.

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: es-multinode
  namespace: demo
spec:
  terminationPolicy: Delete
  componentSpecs:
    - name: dit
      componentDef: elasticsearch-8
      serviceVersion: 8.8.2
      configs:
        - name: es-cm
          variables:
            # use key `roles` to specify roles this component assume
            roles: data,ingest,transform
      replicas: 3
      disableExporter: false
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
        requests:
          cpu: "1"
          memory: "2Gi"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: master
      componentDef: elasticsearch-8
      serviceVersion: 8.8.2
      configs:
        - name: es-cm
          variables:
            # use key `roles` to specify roles this component assume
            roles: master
      replicas: 3
      disableExporter: false
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
        requests:
          cpu: "1"
          memory: "2Gi"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster es-multinode -n demo -w

Expected Output:

NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
es-multinode                        Delete               Creating   10s
es-multinode                        Delete               Updating   41s
es-multinode                        Delete               Running    42s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=es-multinode -n demo

Expected Output:

NAME                    READY   STATUS    RESTARTS   AGE
es-multinode-dit-0      3/3     Running   0          6m21s
es-multinode-dit-1      3/3     Running   0          6m21s
es-multinode-dit-2      3/3     Running   0          6m21s
es-multinode-master-0   3/3     Running   0          6m21s
es-multinode-master-1   3/3     Running   0          6m21s
es-multinode-master-2   3/3     Running   0          6m21s

Once the cluster status becomes Running, your Elasticsearch cluster is ready for use.

Scale-out (Add Replicas)

Expected Workflow:

1. New pod is provisioned, and transitions from Pending to Running.
2. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: es-multinode-scale-out-ops
  namespace: demo
spec:
  clusterName: es-multinode
  type: HorizontalScaling
  horizontalScaling:
  - componentName: dit
    # Specifies the replica changes for scaling in components
    scaleOut:
      # Specifies the replica changes for the component.
      # add one more replica to current component
      replicaChanges: 1

kubectl get ops es-multinode-scale-out-ops -n demo -w

NAME                         TYPE                CLUSTER        STATUS    PROGRESS   AGE
es-multinode-scale-out-ops   HorizontalScaling   es-multinode   Running   0/1        9s
es-multinode-scale-out-ops   HorizontalScaling   es-multinode   Running   1/1        16s
es-multinode-scale-out-ops   HorizontalScaling   es-multinode   Succeed   1/1        16s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: dit
      replicas: 4 # increase replicas to scale-out
...

kubectl patch cluster es-multinode -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 4}]'

Verify Scale-Out

After applying the operation, you will see a new pod created and the Elasticsearch cluster status goes from Updating to Running, and the newly created pod has a new role secondary.

kubectl get pods -n demo -l app.kubernetes.io/instance=es-multinode,apps.kubeblocks.io/component-name=dit

Example Output:

NAME                 READY   STATUS    RESTARTS   AGE
es-multinode-dit-0   3/3     Running   0          4m28s
es-multinode-dit-1   3/3     Running   0          5m27s
es-multinode-dit-2   3/3     Running   0          6m25s
es-multinode-dit-3   3/3     Running   0          1m25s

Scale-in (Remove Replicas)

Expected Workflow:

1. Selected replica (the one with the largest ordinal) is removed
2. Pod is terminated gracefully
3. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: es-multinode-scale-in-ops
  namespace: demo
spec:
  clusterName: es-multinode
  type: HorizontalScaling
  horizontalScaling:
  - componentName: dit
    # Specifies the replica changes for scaling in components
    scaleIn:
      # Specifies the replica changes for the component.
      # remove one replica from current component
      replicaChanges: 1

kubectl get ops es-multinode-scale-in-ops -n demo -w

NAME                        TYPE                CLUSTER        STATUS    PROGRESS   AGE
es-multinode-scale-in-ops   HorizontalScaling   es-multinode   Running   0/1        8s
es-multinode-scale-in-ops   HorizontalScaling   es-multinode   Running   1/1        24s
es-multinode-scale-in-ops   HorizontalScaling   es-multinode   Succeed   1/1        24s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: dit
      replicas: 3 # decrease replicas to scale-in

kubectl patch cluster es-multinode -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 3}]'

Verify Scale-In

kubectl get pods -n demo -l app.kubernetes.io/instance=es-multinode,apps.kubeblocks.io/component-name=dit

Example Output (three Pod):

NAME                 READY   STATUS    RESTARTS   AGE
es-multinode-dit-0   3/3     Running   0          8m20s
es-multinode-dit-1   3/3     Running   0          9m19s
es-multinode-dit-2   3/3     Running   0          10m

Best Practices

When performing horizontal scaling:

• Scale during low-traffic periods when possible
• Monitor cluster health during scaling operations
• Verify sufficient resources exist before scaling out
• Consider storage requirements for new replicas

Cleanup

To remove all created resources, delete the Elasticsearch cluster along with its namespace:

kubectl delete cluster es-multinode -n demo
kubectl delete ns demo

Summary

In this guide you learned how to:

• Perform scale-out operations to add replicas to a Elasticsearch cluster.
• Perform scale-in operations to remove replicas from a Elasticsearch cluster.
• Use both OpsRequest and direct Cluster API updates for horizontal scaling.

KubeBlocks ensures seamless scaling with minimal disruption to your database operations. with minimal disruption to your database operations.