Milvus Cluster Lifecycle Management

This guide demonstrates how to manage a Milvus Cluster's operational state in KubeBlocks, including:

• Stopping the cluster to conserve resources
• Starting a stopped cluster
• Restarting cluster components

These operations help optimize resource usage and reduce operational costs in Kubernetes environments.

Lifecycle management operations in KubeBlocks:

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploy a Milvus Cluster

Please refer to Deploying a Milvus Cluster with KubeBlocks to deploy a milvus cluster.

Cluster Lifecycle Operations

Stopping the Cluster

Stopping a Milvus Cluster in KubeBlocks will:

1. Terminates all running pods
2. Maintains cluster configuration

This operation is ideal for:

• Temporary cost savings
• Maintenance windows
• Development environment pauses

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: milvus-cluster-stop-ops
  namespace: demo
spec:
  clusterName: milvus-cluster
  type: Stop

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: milvus-cluster-stop-ops
  namespace: demo
spec:
  clusterName: milvus-cluster
  type: Stop

Verifying Cluster Stop

To confirm a successful stop operation:

1. Check cluster status transition:

   kubectl get cluster milvus-cluster -n demo -w
   

   Example Output:

   NAME             CLUSTER-DEFINITION  TERMINATION-POLICY   STATUS     AGE
   milvus-cluster   milvus              Delete               Stopping   6m33s
   milvus-cluster   milvus              Delete               Stopped    6m55s
   

2. Verify no running pods:

   kubectl get pods -l app.kubernetes.io/instance=milvus-cluster -n demo
   

   Example Output:

   No resources found in demo namespace.
   

Starting the Cluster

Starting a stopped Milvus Cluster:

1. Recreates all pods
2. Restores service endpoints

Expected behavior:

• Cluster returns to previous state
• No data loss occurs
• Services resume automatically

Verifying Cluster Start

To confirm a successful start operation:

1. Check cluster status transition:

   kubectl get cluster milvus-cluster -n demo -w
   

   Example Output:

   NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
   milvus-cluster   milvus               Delete               Updating   30m
   milvus-cluster   milvus               Delete               Updating   32m
   milvus-cluster   milvus               Delete               Updating   32m
   milvus-cluster   milvus               Delete               Running    33m
   milvus-cluster   milvus               Delete               Running    33m
   

2. Verify pod recreation:

   kubectl get pods -n demo -l app.kubernetes.io/instance=milvus-cluster
   

   Example Output:

   NAME                         READY   STATUS    RESTARTS   AGE
   milvus-cluster-datanode-0    1/1     Running   0          5m24s
   milvus-cluster-indexnode-0   1/1     Running   0          5m24s
   milvus-cluster-mixcoord-0    1/1     Running   0          5m24s
   milvus-cluster-proxy-0       1/1     Running   0          5m24s
   milvus-cluster-querynode-0   1/1     Running   0          5m24s
   milvus-cluster-querynode-1   1/1     Running   0          3m43s
   

Restarting Cluster

Restart operations provide:

• Pod recreation without full cluster stop
• Component-level granularity
• Minimal service disruption

Use cases:

• Configuration changes requiring restart
• Resource refresh
• Troubleshooting

Check Components

There are five components in Milvus Cluster. To get the list of components,

kubectl get cluster -n demo milvus-cluster -oyaml | yq '.spec.componentSpecs[].name'

Expected Output:

proxy
mixcoord
datanode
indexnode
querynode

Restart Proxy via OpsRequest API

List specific components to be restarted:

apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: milvus-cluster-restart-ops
  namespace: demo
spec:
  clusterName: milvus-cluster
  type: Restart
  restart:
  - componentName: proxy

Verifying Restart Completion

To verify a successful component restart:

1. Track OpsRequest progress:

   kubectl get opsrequest milvus-cluster-restart-ops -n demo -w
   

   Example Output:

   NAME                         TYPE      CLUSTER          STATUS    PROGRESS   AGE
   milvus-cluster-restart-ops   Restart   milvus-cluster   Running   0/1        4s
   milvus-cluster-restart-ops   Restart   milvus-cluster   Running   1/1        2m12s
   milvus-cluster-restart-ops   Restart   milvus-cluster   Running   1/1        2m12s
   milvus-cluster-restart-ops   Restart   milvus-cluster   Succeed   1/1        2m12s
   

2. Check pod status:

   kubectl get pods -n demo -l app.kubernetes.io/instance=milvus-cluster
   

   Note: Pods will show new creation timestamps after restart. Only pods belongs to component proxy have been restarted.

Once the operation is complete, the cluster will return to the Running state.

Summary

In this guide, you learned how to:

1. Stop a Milvus Cluster to suspend operations while retaining persistent storage.
2. Start a stopped cluster to bring it back online.
3. Restart specific cluster components to recreate their Pods without stopping the entire cluster.

By managing the lifecycle of your Milvus Cluster, you can optimize resource utilization, reduce costs, and maintain flexibility in your Kubernetes environment. KubeBlocks provides a seamless way to perform these operations, ensuring high availability and minimal disruption.