Skip to main content
Version: Preview

Manage Milvus with KubeBlocks

The popularity of generative AI (Generative AI) has aroused widespread attention and completely ignited the vector database (Vector Database) market.

Milvus is a highly flexible, reliable, and blazing-fast cloud-native, open-source vector database. It powers embedding similarity search and AI applications and strives to make vector databases accessible to every organization. Milvus can store, index, and manage a billion+ embedding vectors generated by deep neural networks and other machine learning (ML) models.

This tutorial illustrates how to create and manage a Milvus cluster by kbcli, kubectl or a YAML file. You can find the YAML examples and guides in the GitHub repository.

Before you start

  • Install kbcli if you want to manage the Milvus cluster with kbcli.

  • Install KubeBlocks by kbcli or by Helm.

  • Install and enable the milvus Addon by kbcli or by Helm.

  • To keep things isolated, create a separate namespace called demo throughout this tutorial.

    kubectl create namespace demo

Create a cluster

Steps

  1. Execute the following command to create a Milvus cluster.

    kbcli cluster create mycluster --cluster-definition=milvus-2.3.2 -n demo
note

If you want to customize your cluster specifications, kbcli provides various options, such as setting cluster version, termination policy, CPU, and memory. You can view these options by adding --help or -h flag.

kbcli cluster create milvus --help

kbcli cluster create milvus -h
  1. Check whether the cluster is created successfully.

    kbcli cluster list -n demo
    >
    NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
    mycluster demo milvus-2.3.2 Delete Running Jul 05,2024 17:35 UTC+0800
  2. Check the cluster information.

    kbcli cluster describe mycluster -n demo
    >
    Name: milvus Created Time: Jul 05,2024 17:35 UTC+0800
    NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
    demo milvus-2.3.2 Running Delete

    Endpoints:
    COMPONENT MODE INTERNAL EXTERNAL
    milvus ReadWrite milvus-milvus.default.svc.cluster.local:19530 <none>
    minio ReadWrite milvus-minio.default.svc.cluster.local:9000 <none>
    proxy ReadWrite milvus-proxy.default.svc.cluster.local:19530 <none>
    milvus-proxy.default.svc.cluster.local:9091

    Topology:
    COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
    etcd milvus-etcd-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    minio milvus-minio-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    milvus milvus-milvus-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    indexnode milvus-indexnode-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    mixcoord milvus-mixcoord-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    querynode milvus-querynode-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    datanode milvus-datanode-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800
    proxy milvus-proxy-0 <none> Running <none> <none> Jul 05,2024 17:35 UTC+0800

    Resources Allocation:
    COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
    milvus false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    etcd false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    minio false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    proxy false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    mixcoord false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    datanode false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    indexnode false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc
    querynode false 1 / 1 1Gi / 1Gi data:20Gi csi-hostpath-sc

    Images:
    COMPONENT TYPE IMAGE
    milvus milvus milvusdb/milvus:v2.3.2
    etcd etcd docker.io/milvusdb/etcd:3.5.5-r2
    minio minio docker.io/minio/minio:RELEASE.2022-03-17T06-34-49Z
    proxy proxy milvusdb/milvus:v2.3.2
    mixcoord mixcoord milvusdb/milvus:v2.3.2
    datanode datanode milvusdb/milvus:v2.3.2
    indexnode indexnode milvusdb/milvus:v2.3.2
    querynode querynode milvusdb/milvus:v2.3.2

    Show cluster events: kbcli cluster list-events -n demo milvus

Scale

Currently, KubeBlocks supports vertically scaling a Milvus cluster.

Before you start

Check whether the cluster status is Running. Otherwise, the following operations may fail.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo milvus-2.3.2 Delete Running Jul 05,2024 17:35 UTC+0800

Steps

  1. Set the --cpu and --memory values according to your needs and run the following command to perform vertical scaling.

    kbcli cluster vscale mycluster -n demo --cpu=1 --memory=1Gi --components=milvus 

    Please wait a few seconds until the scaling process is over.

  2. Validate the vertical scaling operation.

    • View the OpsRequest progress.

      KubeBlocks outputs a command automatically for you to view the OpsRequest progress. The output includes the status of this OpsRequest and Pods. When the status is Succeed, this OpsRequest is completed.

      kbcli cluster describe-ops mycluster-verticalscaling-rpw2l -n demo
    • Check the cluster status.

      kbcli cluster list mycluster -n demo
      >
      NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
      mycluster demo Delete Updating Jul 05,2024 17:35 UTC+0800
      • STATUS=Updating: it means the vertical scaling is in progress.
      • STATUS=Running: it means the vertical scaling operation has been applied.
      • STATUS=Abnormal: it means the vertical scaling is abnormal. The reason may be that the number of the normal instances is less than that of the total instance or the leader instance is running properly while others are abnormal.

        To solve the problem, you can manually check whether this error is caused by insufficient resources. Then if AutoScaling is supported by the Kubernetes cluster, the system recovers when there are enough resources. Otherwise, you can create enough resources and troubleshoot with kubectl describe command.

  3. After the OpsRequest status is Succeed or the cluster status is Running again, check whether the corresponding resources change.

    kbcli cluster describe mycluster -n demo

Volume Expansion

Before you start

Check whether the cluster status is Running. Otherwise, the following operations may fail.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo milvus-2.3.2 Delete Running Jul 05,2024 17:35 UTC+0800

Steps

  1. Set the --storage value according to your need and run the command to expand the volume.

    kbcli cluster volume-expand mycluster -n demo --storage=40Gi --components=milvus -t data

    The volume expansion may take a few minutes.

  2. Validate the volume expansion operation.

    • View the OpsRequest progress.

      KubeBlocks outputs a command automatically for you to view the details of the OpsRequest progress. The output includes the status of this OpsRequest and PVC. When the status is Succeed, this OpsRequest is completed.

      kbcli cluster describe-ops mycluster-volumeexpansion-5pbd2 -n demo
    • View the cluster status.

      kbcli cluster list mycluster
      >
      NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
      mycluster demo milvus-2.3.2 Delete Running Jul 05,2024 17:35 UTC+0800
      • STATUS=Updating: it means the volume expansion is in progress.
      • STATUS=Running: it means the volume expansion operation has been applied.
  3. After the OpsRequest status is Succeed or the cluster status is Running again, check whether the corresponding resources change.

    kbcli cluster describe mycluster -n demo

Restart

  1. Configure the values of components and ttlSecondsAfterSucceed and run the command below to restart a specified cluster.

    kbcli cluster restart mycluster -n demo --components="milvus" --ttlSecondsAfterSucceed=30
    • components describes the component name that needs to be restarted.
    • ttlSecondsAfterSucceed describes the time to live of an OpsRequest job after the restarting succeeds.
  2. Validate the restarting.

    Run the command below to check the cluster status to check the restarting status.

    kbcli cluster list mycluster -n demo
    >
    NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
    milvus demo milvus-2.3.2 Delete Running Jul 05,2024 18:35 UTC+0800
    • STATUS=Updating: it means the cluster restart is in progress.
    • STATUS=Running: it means the cluster has been restarted.

Stop/Start a cluster

You can stop/start a cluster to save computing resources. When a cluster is stopped, the computing resources of this cluster are released, which means the pods of Kubernetes are released, but the storage resources are reserved. You can start this cluster again by snapshots if you want to restore the cluster resources.

Stop a cluster

  1. Configure the name of your cluster and run the command below to stop this cluster.

    kbcli cluster stop mycluster -n demo
  2. Check the status of the cluster to see whether it is stopped.

    kbcli cluster list mycluster -n demo

Start a cluster

  1. Configure the name of your cluster and run the command below to start this cluster.

    kbcli cluster start mycluster -n demo
  2. Check the status of the cluster to see whether it is running again.

    kbcli cluster list mycluster -n demo

Delete a cluster

Termination policy

note

The termination policy determines how a cluster is deleted.

terminationPolicyDeleting Operation
DoNotTerminateDoNotTerminate blocks delete operation.
HaltHalt deletes Cluster resources like Pods and Services but retains Persistent Volume Claims (PVCs), allowing for data preservation while stopping other operations. Halt policy is deprecated in v0.9.1 and will have same meaning as DoNotTerminate.
DeleteDelete extends the Halt policy by also removing PVCs, leading to a thorough cleanup while removing all persistent data.
WipeOutWipeOut deletes all Cluster resources, including volume snapshots and backups in external storage. This results in complete data removal and should be used cautiously, especially in non-production environments, to avoid irreversible data loss.

To check the termination policy, execute the following command.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo milvus-2.3.2 Delete Running Jul 05,2024 17:35 UTC+0800

Steps

Run the command below to delete a specified cluster.

kbcli cluster delete mycluster -n demo