Deploying a Milvus Cluster with KubeBlocks

Milvus Cluster is a distributed deployment for production workloads with multiple specialized components:

Access Layer

• Stateless proxies that handle client connections and request routing

Compute Layer

• Query Nodes: Execute search operations
• Data Nodes: Handle data ingestion and compaction
• Index Nodes: Build and maintain vector indexes

Coordination Layer

• Root Coordinator: Manages global metadata
• Query Coordinator: Orchestrates query execution
• Data Coordinator: Manages data distribution
• Index Coordinator: Oversees index building

Storage Layer

• Metadata Storage (ETCD): Cluster metadata and configuration
• Object Storage (MinIO/S3): Persistent vector data storage
• Log Storage (Pulsar): Message queue for change data capture

Prerequisites

Before proceeding, ensure the following:

• Environment Setup:
  • A Kubernetes cluster is up and running.
  • The kubectl CLI tool is configured to communicate with your cluster.
  • KubeBlocks CLI and KubeBlocks Operator are installed. Follow the installation instructions here.
• Namespace Preparation: To keep resources isolated, create a dedicated namespace for this tutorial:

kubectl create ns demo
namespace/demo created

Deploying the Milvus Cluster

Step 1. Deploy an ETCD Cluster

ETCD cluster is for metadata storage

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: etcdm-cluster
  namespace: demo
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: etcd
      componentDef: etcd-3  # Prefix Match Component Definition Name
      serviceVersion: 3.5.6
      replicas: 1
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Step 2. Deploy a minio Cluster

Minio is for object storage

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: miniom-cluster
  namespace: demo
spec:
  terminationPolicy: WipeOut
  componentSpecs:
    - name: minio
      componentDef: minio  # Prefix Match Component Definition Name
      env:
        - name: MINIO_BUCKETS # specify buckets to be created when initializing the cluster
          value: milvus
      replicas: 2
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

Step 3. Deploy a Kafka Cluster

Kafka is for log storage

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafkam-cluster
  namespace: demo
spec:
  terminationPolicy: Delete
  clusterDef: kafka
  topology: separated
  componentSpecs:
    - name: kafka-broker
      serviceVersion: 3.3.2
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      env:
        - name: KB_KAFKA_BROKER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
        - name: KB_KAFKA_CONTROLLER_HEAP
          value: "-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64"
          # Whether to enable direct Pod IP address access mode.
          # - If set to 'true', Kafka clients will connect to Brokers using the Pod IP address directly.
          # - If set to 'false', Kafka clients will connect to Brokers using the Headless Service's FQDN, and service `advertised-listener` must be set with "podService: true".
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "false"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: kafka-controller
      serviceVersion: 3.3.2
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi

Deploy a Milvus Cluster

The cluster will be created with the following components:

• Proxy
• Data Node
• Index Node
• Query Node
• Mixed Coordinator

And each component will be created with serviceRef to the corresponding service: etcd, minio, and kafka created previously.

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  namespace: demo
  name: milvus-cluster
spec:
  terminationPolicy: Delete
  # The value must be `milvus` to create a Milvus Cluster
  clusterDef: milvus
  # Valid options are: [standalone,cluster]
  topology: cluster
  componentSpecs:
    - name: proxy
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      configs:
        - name: config
          variables:
            mq_type: kafka # pulsar, kafka as mq
            minio_bucket: milvus # a valid bucket name in minio
            minio_root_path: files # root path for milvus data
            minio_use_path_style: "true" # set to true for minio use path style
      # Defines a list of ServiceRef for a Component
      serviceRefs:
        - name: milvus-meta-storage # Specifies the identifier of the service reference declaration, defined in `componentDefinition.spec.serviceRefDeclarations[*].name`
          namespace: demo        # namepspace of referee cluster, update on demand
          # References a service provided by another KubeBlocks Cluster
          clusterServiceSelector:
            cluster: etcdm-cluster  # ETCD Cluster Name, update the cluster name on demand
            service:
              component: etcd       # component name, should be etcd
              service: headless     # Refer to default headless Service
              port: client          # Refer to port name 'client', port 2379
        - name: milvus-log-storage
          namespace: demo
          clusterServiceSelector:
            cluster: kafkam-cluster # Kafka Cluster Name
            service:
              component: kafka-broker
              service: headless
              port: kafka-client         # port 9092
        - name: milvus-object-storage
          namespace: demo
          clusterServiceSelector:
            cluster: miniom-cluster # Minio Cluster Name
            service:
              component: minio
              service: headless
              port: api            # port 9000
            credential:            # Specifies the SystemAccount to authenticate and establish a connection with the referenced Cluster.
              component: minio     # for component 'minio'
              name: root          # the name of the credential (SystemAccount) to reference, using account 'admin' in this case
    - name: mixcoord
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      serviceRefs:
        - name: milvus-meta-storage
          namespace: demo
          clusterServiceSelector:
            cluster: etcdm-cluster
            service:
              component: etcd
              service: headless
              port: client
        - name: milvus-log-storage
          namespace: demo
          clusterServiceSelector:
            cluster: kafkam-cluster
            service:
              component: kafka-broker
              service: headless
              port: kafka-client
        - name: milvus-object-storage
          namespace: demo
          clusterServiceSelector:
            cluster: miniom-cluster
            service:
              component: minio
              service: headless
              port: api
            credential:
              component: minio
              name: root
      disableExporter: true
      configs:
        - name: config
          variables:
            mq_type: kafka
            minio_bucket: milvus
            minio_root_path: files
            minio_use_path_style: "true"
    - name: datanode
      replicas: 1
      disableExporter: true
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      serviceRefs:
        - name: milvus-meta-storage
          namespace: demo
          clusterServiceSelector:
            cluster: etcdm-cluster
            service:
              component: etcd
              service: headless
              port: client
        - name: milvus-log-storage
          namespace: demo
          clusterServiceSelector:
            cluster: kafkam-cluster
            service:
              component: kafka-broker
              service: headless
              port: kafka-client
        - name: milvus-object-storage
          namespace: demo
          clusterServiceSelector:
            cluster: miniom-cluster
            service:
              component: minio
              service: headless
              port: api
            credential:
              component: minio
              name: root
      disableExporter: true
      configs:
        - name: config
          variables:
            mq_type: kafka
            minio_bucket: milvus
            minio_root_path: files
            minio_use_path_style: "true"
    - name: indexnode
      replicas: 1
      disableExporter: true
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      serviceRefs:
        - name: milvus-meta-storage
          namespace: demo
          clusterServiceSelector:
            cluster: etcdm-cluster
            service:
              component: etcd
              service: headless
              port: client
        - name: milvus-log-storage
          namespace: demo
          clusterServiceSelector:
            cluster: kafkam-cluster
            service:
              component: kafka-broker
              service: headless
              port: kafka-client
        - name: milvus-object-storage
          namespace: demo
          clusterServiceSelector:
            cluster: miniom-cluster
            service:
              component: minio
              service: headless
              port: api
            credential:
              component: minio
              name: root
      disableExporter: true
      configs:
        - name: config
          variables:
            mq_type: kafka
            minio_bucket: milvus
            minio_root_path: files
            minio_use_path_style: "true"
    - name: querynode
      replicas: 1
      disableExporter: true
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      serviceRefs:
        - name: milvus-meta-storage
          namespace: demo
          clusterServiceSelector:
            cluster: etcdm-cluster
            service:
              component: etcd
              service: headless
              port: client
        - name: milvus-log-storage
          namespace: demo
          clusterServiceSelector:
            cluster: kafkam-cluster
            service:
              component: kafka-broker
              service: headless
              port: kafka-client
        - name: milvus-object-storage
          namespace: demo
          clusterServiceSelector:
            cluster: miniom-cluster
            service:
              component: minio
              service: headless
              port: api
            credential:
              component: minio
              name: root
      disableExporter: true
      configs:
        - name: config
          variables:
            mq_type: kafka
            minio_bucket: milvus
            minio_root_path: files
            minio_use_path_style: "true"

Service References are specified using serviceRefs as follows, please update cluster names and ports accordingly w.r.t your running environment.

# Defines a list of ServiceRef for a Component
serviceRefs:
  - name: milvus-meta-storage # Specifies the identifier of the service reference declaration, defined in `componentDefinition.spec.serviceRefDeclarations[*].name`
    namespace: demo        # namepspace of referee cluster, update on demand
    # References a service provided by another KubeBlocks Cluster
    clusterServiceSelector:
      cluster: etcdm-cluster  # ETCD Cluster Name, update the cluster name on demand
      service:
        component: etcd       # component name, should be etcd
        service: headless     # Refer to default headless Service
        port: client          # NOTE: Refer to port name 'client', for port number '3501'
  - name: milvus-log-storage
    namespace: demo
    clusterServiceSelector:
      cluster: kafkam-cluster # Kafka Cluster Name
      service:
        component: kafka-broker
        service: headless
        port: kafka-client          # NOTE: Refer to port name 'kafka-client', for port number '9092'
  - name: milvus-object-storage
    namespace: demo
    clusterServiceSelector:
      cluster: miniom-cluster # Minio Cluster Name
      service:
        component: minio
        service: headless
        port: http           # NOTE: Refer to port name 'http', for port number '9000'
      credential:            # Specifies the SystemAccount to authenticate and establish a connection with the referenced Cluster.
        component: minio     # for component 'minio'
        name: root          # NOTE: the name of the credential (SystemAccount) to reference, using account `root` in this case

If the serviceRef info is invalid, the cluster won't be able to create and you may check KubeBlocks logs to find the error.

k -n kb-system logs deployments/kubeblocks

For example, if the Kafka Cluster Service kafkam-cluster-kafka-broker-headless name specified in the serviceRef is not found, the error message will be:

build error: requeue after: 1s as: build synthesized component for milvus-cluster-querynode failed: build service references failed: Service "kafkam-cluster-broker-headless" not found

Verifying the Deployment

Check the Cluster Status

Once the cluster is deployed, check its status:

kubectl get cluster milvus-cluster  -n demo -w

Example Output:

NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
milvus-cluster   milvus               Delete               Running   4m38s

Verify Component and Pod Status

kubectl get component -l app.kubernetes.io/instance=milvus-cluster -n demo

Example Output:

NAME                       DEFINITION                       SERVICE-VERSION   STATUS    AGE
milvus-cluster-datanode    milvus-datanode-1.0.0            v2.3.2            Running   5m8s
milvus-cluster-indexnode   milvus-indexnode-1.0.0           v2.3.2            Running   5m8s
milvus-cluster-mixcoord    milvus-mixcoord-1.0.0            v2.3.2            Running   5m8s
milvus-cluster-proxy       milvus-proxy-1.0.0               v2.3.2            Running   5m8s
milvus-cluster-querynode   milvus-querynode-1.0.0           v2.3.2            Running   5m8s

Check pods:

kubectl get pods -l app.kubernetes.io/instance=milvus-cluster -n demo

Example Output:

NAME                         READY   STATUS    RESTARTS   AGE
milvus-cluster-datanode-0    1/1     Running   0          5m30s
milvus-cluster-indexnode-0   1/1     Running   0          5m31s
milvus-cluster-mixcoord-0    1/1     Running   0          5m32s
milvus-cluster-proxy-0       1/1     Running   0          5m32s
milvus-cluster-querynode-0   1/1     Running   0          5m31s
milvus-cluster-querynode-1   1/1     Running   0          3m51s

Cleanup

To remove all resources created during this tutorial:

kubectl delete cluster milvus-cluster -n demo
kubectl delete cluster etcdm-cluster -n demo
kubectl delete cluster miniom-cluster -n demo
kubectl delete cluster kafkam-cluster -n demo
kubectl delete ns demo