KubeBlocks
BlogsKubeBlocks Cloud
Overview
Quickstart

Topologies

MySQL Semi-Synchronous Cluster
MySQL Cluster with ProxySQL
MySQL Group Replication Cluster
MySQL Group Replication with ProxySQL
MySQL Cluster with Orchestrator
MySQL with Orchestrator & ProxySQL

Operations

Lifecycle Management
Vertical Scaling
Horizontal Scaling
Volume Expansion
Manage MySQL Services
Minor Version Upgrade
Modify MySQL Parameters
Planned Switchover in MySQL
Decommission MySQL Replica
Recovering MySQL Replica

Backup And Restores

Create BackupRepo
Create Full Backup
Scheduled Backups
Scheduled Continuous Backup
Restore MySQL Cluster
Restore with PITR

Custom Secret

Custom Password
Custom Password Policy

TLS

MySQL Cluster with TLS
MySQL Cluster with User-Provided TLS
MySQL Cluster with mTLS

Monitoring

Observability for MySQL Clusters

Advanced Pod Management

Custom Scheduling Policies
Custom Pod Resources
Pod Management Parallelism
Using OnDelete for Controlled Pod Updates
Gradual Rolling Update
  1. Why Decommission Pods with KubeBlocks?
  2. Prerequisites
  3. Deploy a MySQL Semi-Synchronous Cluster
  4. Verifying the Deployment
  5. Decommission a Specific Pod
    1. Option 1.: Using OpsRequest
      1. Monitor the Decommissioning Process
    2. Option 2.: Using Cluster API
    3. Verify the Decommissioning
  6. Summary

Decommission a Specific Pod in a KubeBlocks-Managed MySQL Clusters

This guide explains how to decommission (take offline) a specific Pod in a MySQL cluster managed by KubeBlocks. Decommissioning a Pod allows precise control over cluster resources without disrupting the cluster's overall functionality. This is particularly useful for workload rebalancing, node maintenance, or addressing specific failures.

Why Decommission Pods with KubeBlocks?

In traditional StatefulSet-based deployments, Kubernetes lacks the ability to decommission specific Pods. StatefulSets ensure the order and identity of Pods, and scaling down always removes the Pod with the highest ordinal number (e.g., scaling down from 3 replicas removes Pod-2 first). This limitation prevents precise control over which Pod to take offline, which can complicate maintenance, workload distribution, or failure handling.

KubeBlocks overcomes this limitation by enabling administrators to decommission specific Pods directly. This fine-grained control ensures high availability and allows better resource management without disrupting the entire cluster.

Prerequisites

Before proceeding, ensure the following:

  • Environment Setup:
    • A Kubernetes cluster is up and running.
    • The kubectl CLI tool is configured to communicate with your cluster.
    • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
  • Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:
kubectl create ns demo
namespace/demo created

Deploy a MySQL Semi-Synchronous Cluster

Deploy a 3-node MySQL semi-synchronous cluster (1 primary, 2 replicas):

kubectl apply -f - <<EOF
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: example-mysql-cluster
  namespace: demo
spec:
  clusterDef: mysql
  topology: semisync
  terminationPolicy: Delete
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.35
      replicas: 3
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
EOF

Verifying the Deployment

Monitor the cluster status until it transitions to the Running state:

kubectl get cluster example-mysql-cluster -n demo -w

Example Output:

NAME                     CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
example-mysql-cluster   mysql                Delete               Creating   8s
example-mysql-cluster   mysql                Delete               Running    2m41s

Once the cluster status becomes Running, your MySQL cluster is ready for use.

List the Pods in the cluster to verify all three Pods are running:

kubectl get pods -n demo -l app.kubernetes.io/instance=example-mysql-cluster

Example Output (3 Pods):

NAME                           READY   STATUS    RESTARTS   AGE
example-mysql-cluster-mysql-0   4/4     Running   0          4m30s
example-mysql-cluster-mysql-1   4/4     Running   0          4m30s
example-mysql-cluster-mysql-2   4/4     Running   0          49s

Decommission a Specific Pod

To decommission a specific Pod (e.g., 'example-mysql-cluster-mysql-1'), you can use one of the following methods:

Option 1.: Using OpsRequest

Create an OpsRequest to mark the Pod as offline:

kubectl apply -f - <<EOF
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-decommission-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: mysql
    scaleIn:
      onlineInstancesToOffline:
        - 'example-mysql-cluster-mysql-1'  # Specifies the instance names that need to be taken offline
EOF

Monitor the Decommissioning Process

Check the progress of the decommissioning operation:

kubectl describe ops example-mysql-cluster-decommission-ops -n demo

Example Output:

Status:
  Phase:              Succeed
  Progress:           1/1
  ...

Option 2.: Using Cluster API

Alternatively, update the Cluster resource directly to decommission the Pod:

kubectl apply -f - <<EOF
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: example-mysql-cluster
  namespace: demo
spec:
  clusterDef: mysql
  topology: semisync
  terminationPolicy: Delete
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.35
      replicas: 2    # <----- Reduce replicas from 3 to 2
      offlineInstances:
        - example-mysql-cluster-mysql-1   # <----- Specify Pod to be decommissioned
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
EOF

Verify the Decommissioning

After applying the updated configuration, verify the remaining Pods in the cluster:

kubectl get pods -n demo -l app.kubernetes.io/instance=example-mysql-cluster

Example Output:

NAME                                READY   STATUS    RESTARTS   AGE
example-mysql-cluster-mysql-0   4/4     Running   0          6m38s
example-mysql-cluster-mysql-2   4/4     Running   0          6m38s

Summary

In this guide, you learned:

  • The limitations of traditional StatefulSet-based scaling in Kubernetes.
  • How KubeBlocks enables precise decommissioning of specific Pods.
  • Two methods to decommission a Pod: using OpsRequest or directly updating the Cluster API.

By leveraging KubeBlocks, you can manage MySQL clusters with fine-grained control, ensuring high availability and flexibility for maintenance and workload distribution.

© 2025 ApeCloud PTE. Ltd.