Horizontal Scaling for MongoDB Clusters with KubeBlocks

This guide explains how to perform horizontal scaling (scale-out and scale-in) on a MongoDB cluster managed by KubeBlocks. You'll learn how to use both OpsRequest and direct Cluster API updates to achieve this.

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a MongoDB ReplicaSet Cluster

KubeBlocks uses a declarative approach for managing MongoDB Replication Clusters. Below is an example configuration for deploying a MongoDB ReplicaSet Cluster with one primary replica and two secondary replicas.

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mongo-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: mongodb
  topology: replicaset
  componentSpecs:
    - name: mongodb
      serviceVersion: "6.0.16"
      replicas: 3
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster mongo-cluster -n demo -w

Expected Output:

kubectl get cluster mongo-cluster -n demo
NAME            CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
mongo-cluster   mongodb              Delete               Creating   49s
mongo-cluster   mongodb              Delete               Running    62s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=mongo-cluster -L  kubeblocks.io/role -n demo

Expected Output:

NAME                      READY   STATUS    RESTARTS   AGE   ROLE
mongo-cluster-mongodb-0   2/2     Running   0          78s   primary
mongo-cluster-mongodb-1   2/2     Running   0          63s   secondary
mongo-cluster-mongodb-2   2/2     Running   0          48s   secondary

Once the cluster status becomes Running, your MongoDB cluster is ready for use.

Scale-out (Add Replicas)

Expected Workflow:

1. New pod is provisioned, and transitions from Pending to Running with secondary role
2. Data synced from primary to new replica
3. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: mongo-cluster-scale-out-ops
  namespace: demo
spec:
  clusterName: mongo-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: mongodb
    # Specifies the replica changes for scaling in components
    scaleOut:
      # Specifies the replica changes for the component.
      # add one more replica to current component
      replicaChanges: 1

kubectl get ops mongo-cluster-scale-out-ops -n demo -w

NAME                          TYPE                CLUSTER         STATUS    PROGRESS   AGE
mongo-cluster-scale-out-ops   HorizontalScaling   mongo-cluster   Running   0/1        9s
mongo-cluster-scale-out-ops   HorizontalScaling   mongo-cluster   Running   1/1        20s
mongo-cluster-scale-out-ops   HorizontalScaling   mongo-cluster   Succeed   1/1        20s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: mongodb
      replicas: 4 # increase replicas to scale-out
...

kubectl patch cluster mongo-cluster -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 4}]'

Verify Scale-Out

After applying the operation, you will see a new pod created and the MongoDB cluster status goes from Updating to Running, and the newly created pod has a new role secondary.

New replicas automatically join as secondary nodes.

kubectl get pods -n demo -l app.kubernetes.io/instance=mongo-cluster -L kubeblocks.io/role

Example Output:

NAME                      READY   STATUS    RESTARTS   AGE     ROLE
mongo-cluster-mongodb-0   2/2     Running   0          6m24s   primary
mongo-cluster-mongodb-1   2/2     Running   0          7m19s   secondary
mongo-cluster-mongodb-2   2/2     Running   0          5m57s   secondary
mongo-cluster-mongodb-3   2/2     Running   0          3m54s   secondary

Verify mongodb internal status using:

1. login in any mongodb replica

kubectl exec -it -n demo mongo-cluster-mongodb-0 -- /bin/bash
mongosh "mongodb://${MONGODB_ROOT_USER}:${MONGODB_ROOT_PASSWORD}@127.0.0.1:27017/admin"

2. check mongodb rs.status()

# login to mongodb and query
mongo-cluster-mongodb [direct: secondary] admin> rs.status()

Scale-in (Remove Replicas)

Expected Workflow:

1. Selected replica (the one with the largest ordinal) is removed
2. If removing a primary replica, automatic switchover occurs first
3. Pod is terminated gracefully
4. Cluster status changes from Updating to Running

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: mongo-cluster-scale-in-ops
  namespace: demo
spec:
  clusterName: mongo-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: mongodb
    # Specifies the replica changes for scaling in components
    scaleIn:
      # Specifies the replica changes for the component.
      # remove one replica from current component
      replicaChanges: 1

kubectl get ops mongo-cluster-scale-in-ops -n demo -w

NAME                         TYPE                CLUSTER         STATUS    PROGRESS   AGE
mongo-cluster-scale-in-ops   HorizontalScaling   mongo-cluster   Running   0/1        8s
mongo-cluster-scale-in-ops   HorizontalScaling   mongo-cluster   Running   1/1        24s
mongo-cluster-scale-in-ops   HorizontalScaling   mongo-cluster   Succeed   1/1        24s

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
spec:
  componentSpecs:
    - name: mongodb
      replicas: 1 # decrease replicas to scale-out

kubectl patch cluster mongo-cluster -n demo --type=json -p='[{"op": "replace", "path": "/spec/componentSpecs/0/replicas", "value": 1}]'

Verify Scale-In

Example Output (ONE Pod):

kubectl get pods -n demo -l app.kubernetes.io/instance=mongo-cluster,apps.kubeblocks.io/component-name=mongodb
NAME                        READY   STATUS   RESTARTS   AGE
mongo-cluster-mongodb-0     2/2     Running  0          18m

Troubleshooting

If the scale-in operation gets stuck for quite a long time, please check these resources:

# Check agent logs on both current primary and candidate
kubectl logs -n demo <primary-pod> -c kbagent
kubectl logs -n demo <candidate-pod> -c kbagent

# Check cluster events for errors
kubectl get events -n demo --field-selector involvedObject.name=pg-cluster

# Check kubeblocks logs
kubectl -n kb-system logs deploy/kubeblocks

If you get errors like the following from the primary replica:

INFO	Action Executed	{"action": "switchover", "result": "exit code: 1: failed"}
INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 7}

It could be a switchover error, and please check KubeBlocks logs for more details.

Best Practices

When performing horizontal scaling:

• Scale during low-traffic periods when possible
• Monitor cluster health during scaling operations
• Verify sufficient resources exist before scaling out
• Consider storage requirements for new replicas

Cleanup

To remove all created resources, delete the MongoDB cluster along with its namespace:

kubectl delete cluster mongo-cluster -n demo
kubectl delete ns demo

Summary

In this guide you learned how to:

• Perform scale-out operations to add replicas to a MongoDB cluster.
• Perform scale-in operations to remove replicas from a MongoDB cluster.
• Use both OpsRequest and direct Cluster API updates for horizontal scaling.

KubeBlocks ensures seamless scaling with minimal disruption to your database operations.