MongoDB Cluster Switchover

A switchover is a planned operation that transfers the primary role from one MongoDB instance to another. Unlike failover which occurs during failures, switchover provides:

• Controlled role transitions
• Minimal downtime (typically a few hundred milliseconds)
• Predictable maintenance windows

Switchover is ideal for:

• Node maintenance/upgrades
• Workload rebalancing
• Testing high availability
• Planned infrastructure changes

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a MongoDB Cluster

KubeBlocks uses a declarative approach for managing MongoDB Replication Clusters. Below is an example configuration for deploying a MongoDB ReplicaSet Cluster with one primary replica and two secondary replicas.

Apply the following YAML configuration to deploy the cluster:

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mongo-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: mongodb
  topology: replicaset
  componentSpecs:
    - name: mongodb
      serviceVersion: "6.0.16"
      replicas: 3
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster mongo-cluster -n demo -w

Expected Output:

kubectl get cluster mongo-cluster -n demo
NAME            CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
mongo-cluster   mongodb              Delete               Creating   49s
mongo-cluster   mongodb              Delete               Running    62s

Check the pod status and roles:

kubectl get pods -l app.kubernetes.io/instance=mongo-cluster -L  kubeblocks.io/role -n demo

Expected Output:

NAME                      READY   STATUS    RESTARTS   AGE   ROLE
mongo-cluster-mongodb-0   2/2     Running   0          78s   primary
mongo-cluster-mongodb-1   2/2     Running   0          63s   secondary
mongo-cluster-mongodb-2   2/2     Running   0          48s   secondary

Once the cluster status becomes Running, your MongoDB cluster is ready for use.

Check Roles

List the Pods and their roles (primary or secondary):

kubectl get pods -n demo -l app.kubernetes.io/instance=mongo-cluster,apps.kubeblocks.io/component-name=mongodb -L kubeblocks.io/role

Example Output:

NAME                      READY   STATUS    RESTARTS   AGE   ROLE
mongo-cluster-mongodb-0   2/2     Running   0          20m   primary
mongo-cluster-mongodb-1   2/2     Running   0          21m   secondary
mongo-cluster-mongodb-2   2/2     Running   0          19m   secondary

Performing a Planned Switchover

To initiate a planned switchover, create an OpsRequest resource as shown below:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: mongodb-switchover-ops
  namespace: demo
spec:
  clusterName: mongo-cluster
  type: Switchover
  switchover:
  - componentName: mongodb
    instanceName: mongo-cluster-mongodb-0

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: mongodb-switchover-targeted
  namespace: demo
spec:
  clusterName: mongo-cluster
  type: Switchover
  switchover:
  - componentName: mongodb
    # Specifies the instance whose role will be transferred.
    # A typical usage is to transfer the leader role in a consensus system.
    instanceName: mongo-cluster-mongodb-0
    # If CandidateName is specified, the role will be transferred to this instance.
    # The name must match one of the pods in the component.
    # Refer to ComponentDefinition's Swtichover lifecycle action for more details.
    candidateName: mongo-cluster-mongodb-1

Monitoring the Switchover

Monitor the switchover progress:

kubectl get ops mongodb-switchover-ops -n demo -w

Expected Result:

NAME                     TYPE         CLUSTER         STATUS    PROGRESS   AGE
mongodb-switchover-ops   Switchover   mongo-cluster   Succeed   1/1        33s

Verify the Switchover

After the switchover is executed, the specified instance will be promoted to the primary role, while the previously primary instance will take on the secondary role.

kubectl get pods -n demo -l app.kubernetes.io/instance=mongo-cluster -L kubeblocks.io/role

Expected Output:

NAME                      READY   STATUS    RESTARTS   AGE   ROLE
mongo-cluster-mongodb-0   2/2     Running   0          23m   secondary
mongo-cluster-mongodb-1   2/2     Running   0          24m   primary
mongo-cluster-mongodb-2   2/2     Running   0          23m   secondary

In this example:

• Pod 'mongo-cluster-mongodb-1' has been promoted to the primary role.
• Pod 'mongo-cluster-mongodb-0' has transitioned to the secondary role.

Troubleshooting

Common Switchover Issues

If the switchover operation gets stuck, check these resources:

# Check agent logs on both current primary and candidate
kubectl logs -n demo <primary-pod> -c kbagent
kubectl logs -n demo <candidate-pod> -c kbagent

# Check cluster events for errors
kubectl get events -n demo --field-selector involvedObject.name=mongo-cluster

# Check kubeblocks logs
kubectl -n kb-system logs deploy/kubeblocks

Summary

This guide demonstrated how to:

1. Deploy a MongoDB HA cluster
2. Perform both automatic and targeted Switchover
3. Verify role transitions

Key takeaways:

• Switchover enables controlled maintenance with minimal downtime (~100-500ms)
• KubeBlocks provides declarative operations for reliable role transitions
• Always verify:
  • Cluster status immediately after switchover
  • Application connectivity
  • Replication health
• Check logs for troubleshooting:
  • KubeBlocks operator (kb-system namespace)
  • kbagent on database pods