Skip to main content
Version: release-0.9

Manage Qdrant with KubeBlocks

The popularity of generative AI (Generative AI) has aroused widespread attention and completely ignited the vector database (Vector Database) market. Qdrant (read: quadrant) is a vector similarity search engine and vector database. It provides a production-ready service with a convenient API to store, search, and manage points—vectors with an additional payload Qdrant is tailored to extended filtering support. It makes it useful for all sorts of neural-network or semantic-based matching, faceted search, and other applications.

KubeBlocks supports the management of Qdrant. This tutorial illustrates how to create and manage a Qdrant cluster by kbcli, kubectl or a YAML file. You can find the YAML examples in the GitHub repository.

Before you start

  • Install kbcli if you want to manage the Qdrant cluster with kbcli.

  • Install KubeBlocks by kbcli or by Helm.

  • Install and enable the Qdrant Addon by kbcli or by Helm.

  • To keep things isolated, create a separate namespace called demo throughout this tutorial.

    kubectl create namespace demo

Create a cluster

Steps

  1. Execute the following command to create a Qdrant cluster.

    kbcli cluster create qdrant mycluster -n demo

    If you want to customize your cluster specifications, kbcli provides various options, such as setting cluster version, termination policy, CPU, and memory. You can view these options by adding --help or -h flag.

    kbcli cluster create qdrant --help

    kbcli cluster create qdrant -h
  2. Check whether the cluster is created.

    kbcli cluster list -n demo
    >
    NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
    mycluster demo qdrant Delete Running Aug 15,2023 23:03 UTC+0800
  3. Check the cluster information.

    kbcli cluster describe mycluster -n demo
    >
    Name: mycluster Created Time: Aug 15,2023 23:03 UTC+0800
    NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY
    demo qdrant Running Delete

    Endpoints:
    COMPONENT MODE INTERNAL EXTERNAL
    qdrant ReadWrite mycluster-qdrant-qdrant.default.svc.cluster.local:6333 <none>
    mycluster-qdrant-qdrant.default.svc.cluster.local:6334

    Topology:
    COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME
    qdrant mycluster-qdrant-0 <none> Running <none> x-worker3/172.20.0.3 Aug 15,2023 23:03 UTC+0800
    qdrant mycluster-qdrant-1 <none> Running <none> x-worker2/172.20.0.5 Aug 15,2023 23:03 UTC+0800
    qdrant mycluster-qdrant-2 <none> Running <none> x-worker/172.20.0.2 Aug 15,2023 23:04 UTC+0800

    Resources Allocation:
    COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS
    qdrant false 1 / 1 1Gi / 1Gi data:20Gi standard

    Images:
    COMPONENT TYPE IMAGE
    qdrant qdrant docker.io/qdrant/qdrant:latest

    Data Protection:
    AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME
    Disabled <none> <none> 7d <none> <none>

    Show cluster events: kbcli cluster list-events -n demo mycluster

Connect to a Qdrant cluster

Qdrant provides both HTTP and gRPC protocols for client access on ports 6333 and 6334 respectively. Depending on where the client is, different connection options are offered to connect to the Qdrant cluster.

note

If your cluster is on AWS, install the AWS Load Balancer Controller first.

  • If your client is inside a K8s cluster, run kbcli cluster describe mycluster -n demo to get the ClusterIP address of the cluster or the corresponding K8s cluster domain name.
  • If your client is outside the K8s cluster but in the same VPC as the server, run kbcli cluster expose mycluster -n demo --enable=true --type=vpc to get a VPC load balancer address for the database cluster.
  • If your client is outside the VPC, run kbcli cluster expose mycluster -n demo --enable=true --type=internet to open a public network reachable address for the database cluster.

Monitor the database

The monitoring function of Qdrant is the same as other engines. For details, refer to the monitoring tutorial.

Scale

The scaling function for Qdrant is also supported.

Scale horizontally

Horizontal scaling changes the amount of pods. For example, you can scale out replicas from three to five.

From v0.9.0, besides replicas, KubeBlocks also supports scaling in and out instances, refer to Horizontal Scale in API docs for more details and examples.

Before you start

Check whether the cluster status is Running. Otherwise, the following operations may fail.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo qdrant Delete Running Aug 15,2023 23:03 UTC+0800

Steps

  1. Set the --replicas value according to your needs and perform the horizontal scaling.

    kbcli cluster hscale mycluster -n demo --replicas=5 --components=qdrant
    • --components describes the component name ready for horizontal scaling.
    • --replicas describes the replica amount of the specified components. Edit the amount based on your demands to scale in or out replicas.

    Please wait a few seconds until the scaling process is over.

  2. Validate the horizontal scaling operation.

    • View the OpsRequest progress.

      KubeBlocks outputs a command automatically for you to view the OpsRequest progress. The output includes the status of this OpsRequest and Pods. When the status is Succeed, this OpsRequest is completed.

      kbcli cluster describe-ops mycluster-horizontalscaling-xpdwz -n demo
    • View the cluster satus.

      kbcli cluster list mycluster -n demo
      >
      NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
      mycluster demo qdrant Delete Running Jul 24,2023 11:38 UTC+0800
      • STATUS=Updating: it means horizontal scaling is in progress.
      • STATUS=Running: it means horizontal scaling has been applied.
  3. After the OpsRequest status is Succeed or the cluster status is Running again, check whether the corresponding resources change.

    kbcli cluster describe mycluster -n demo

Scale vertically

You can vertically scale a cluster by changing resource requirements and limits (CPU and storage). For example, you can change the resource class from 1C2G to 2C4G by performing vertical scaling.

Before you start

Check whether the cluster status is Running. Otherwise, the following operations may fail.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo qdrant Delete Running Aug 15,2023 23:03 UTC+0800

Steps

  1. Set the --cpu and --memory values according to your needs and run the following command to perform vertical scaling.

    kbcli cluster vscale mycluster -n demo --cpu=0.5 --memory=512Mi --components=qdrant 

    Please wait a few seconds until the scaling process is over.

  2. Validate the vertical scaling operation.

    • View the OpsRequest progress.

      KubeBlocks outputs a command automatically for you to view the OpsRequest progress. The output includes the status of this OpsRequest and Pods. When the status is Succeed, this OpsRequest is completed.

      kbcli cluster describe-ops mycluster-verticalscaling-rpw2l -n demo
      >
      NAME TYPE CLUSTER STATUS PROGRESS AGE
      mycluster-verticalscaling-rpw2l VerticalScaling mycluster Running 1/5 44s
    • Check the cluster status.

      kbcli cluster list mycluster -n demo
      >
      NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
      mycluster demo Delete Updating Aug 15,2023 23:03 UTC+0800
      • STATUS=Updating: it means the vertical scaling is in progress.
      • STATUS=Running: it means the vertical scaling operation has been applied.
      • STATUS=Abnormal: it means the vertical scaling is abnormal. The reason may be that the number of the normal instances is less than that of the total instance or the leader instance is running properly while others are abnormal.

        To solve the problem, you can manually check whether this error is caused by insufficient resources. Then if AutoScaling is supported by the Kubernetes cluster, the system recovers when there are enough resources. Otherwise, you can create enough resources and troubleshoot with kubectl describe command.

  3. After the OpsRequest status is Succeed or the cluster status is Running again, check whether the corresponding resources change.

    kbcli cluster describe mycluster -n demo

Volume Expansion

Before you start

Check whether the cluster status is Running. Otherwise, the following operations may fail.

kbcli cluster list mycluster -n demo
>
NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster demo qdrant Delete Running Aug 15,2023 23:03 UTC+0800

Steps

  1. Set the --storage value according to your need and run the command to expand the volume.

    kbcli cluster volume-expand mycluster -n demo --storage=40Gi --components=qdrant -t data

    The volume expansion may take a few minutes.

  2. Validate the volume expansion operation.

    • View the OpsRequest progress.

      KubeBlocks outputs a command automatically for you to view the details of the OpsRequest progress. The output includes the status of this OpsRequest and PVC. When the status is Succeed, this OpsRequest is completed.

      kbcli cluster describe-ops mycluster-volumeexpansion-5pbd2 -n demo
      >
      NAME TYPE CLUSTER STATUS PROGRESS AGE
      mycluster-volumeexpansion-5pbd2 VolumeExpansion mycluster Running 1/1 67s
    • View the cluster status.

      kbcli cluster list mycluster -n demo
      >
      NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
      mycluster demo qdrant Delete Updating Aug 15,2023 23:03 UTC+0800
      • STATUS=Updating: it means the volume expansion is in progress.
      • STATUS=Running: it means the volume expansion operation has been applied.
  3. After the OpsRequest status is Succeed or the cluster status is Running again, check whether the corresponding resources change.

    kbcli cluster describe mycluster -n demo

Restart

  1. Configure the values of components and ttlSecondsAfterSucceed and run the command below to restart a specified cluster.

    kbcli cluster restart mycluster -n demo --components="qdrant" --ttlSecondsAfterSucceed=30
    • components describes the component name that needs to be restarted.
    • ttlSecondsAfterSucceed describes the time to live of an OpsRequest job after the restarting succeeds.
  2. Validate the restarting.

    Run the command below to check the cluster status to check the restarting status.

    kbcli cluster list mycluster -n demo
    >
    NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
    mycluster demo qdrant Delete Running Aug 15,2023 23:03 UTC+0800
    • STATUS=Updating: it means the cluster restart is in progress.
    • STATUS=Running: it means the cluster has been restarted.

Stop/Start a cluster

You can stop/start a cluster to save computing resources. When a cluster is stopped, the computing resources of this cluster are released, which means the pods of Kubernetes are released, but the storage resources are reserved. You can start this cluster again by snapshots if you want to restore the cluster resources.

Stop a cluster

  1. Configure the name of your cluster and run the command below to stop this cluster.

    kbcli cluster stop mycluster -n demo
  2. Check the status of the cluster to see whether it is stopped.

    kbcli cluster list mycluster -n demo

Start a cluster

  1. Configure the name of your cluster and run the command below to start this cluster.

    kbcli cluster start mycluster -n demo
  2. Check the status of the cluster to see whether it is running again.

    kbcli cluster list mycluster -n demo

Backup and restore

The backup and restore operations for Qdrant are the same as those of other clusters and you can refer to the backup and restore documents for details. Remember to use --method parameter.