KubeBlocks is an open-source Kubernetes database Operator that supports rich database engines and comprehensive database maintenance operations and services. This test aims to verify whether the maintenance operations and services results of the engines supported by KubeBlocks meet expectations.

These are the engines covered by the test:

• ApeCloud Mysql
• Clickhouse
• Damengdb
• Doris
• ElasticSearch
• Etcd
• GaussDB
• Greatdb
• Greptimedb
• Hadoop
• Hive
• Influxdb
• Kafka
• Kingbase
• Loki
• Mariadb
• Milvus
• Minio
• Mogdb
• Mongodb
• Mongodb Sharding
• Mssql
• Mysql
• Mysql Proxysql
• Mysql Orchestrator
• Nacos
• Nebula
• OceanBase
• OceanBase Proxy
• OpenSearch
• Oracle
• Orchestrator
• Orioledb
• Postgresql
• Qdrant
• Rabbitmq
• Redis
• Redis Cluster
• Risingwave
• RocketMQ
• Starrocks
• TdSql
• Tdengine
• Tidb
• Vanilla Postgresql
• Vastbase
• Victoria Metrics
• Weaviate
• Yashandb
• Zookeeper

KubeBlocks database maintenance operations have different ways to perform, most of the which can be realized by submitting a OpsRequest operation, you may refer to https://github.com/apecloud/kubeblocks/tree/main/examples for details.

Working Principle:

KubeBlocks deploys database engine components using the Kubernetes Operator pattern and creates database clusters based on custom resources (such as ComponentDefinition and ComponentVersion). The controller automatically schedules and maintains the state of the database instances according to the configuration.

Practical Significance:

• Verify if the database cluster can successfully initialize and enter a normal running state.
• Ensure that the database service is accessible after startup and that the data persistence mechanism is effective.
• Provide foundational validation for subsequent operational and maintenance activities.

Test Examples

```yaml
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: example-mysql-cluster
  namespace: demo
spec:
  clusterDef: mysql
  topology: semisync
  terminationPolicy: Delete
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.35
      replicas: 2
      resources:
        limits:
          cpu: '0.5'
          memory: 0.5Gi
        requests:
          cpu: '0.5'
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: ""
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
```

Working Principle:

The restart behavior of the database instance is controlled by modifying the state fields of the cluster resource objects. KubeBlocks supports restarting all Pods in the cluster. During a restart, each Pod will be deleted one by one and then recreated.

You can restart all pods of the cluster. When an exception occurs in a database, you can try to restart it.

Practical Significance:

• Verify whether the database restart process is safe and reliable.
• Ensure that data corruption or loss does not occur during the restart process.

Test Examples

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-restart-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: Restart
  restart:
  - componentName: mysql
```

Working Principle:

The stop/start behavior of the database instance is controlled by modifying the state fields of the cluster resource objects. When stopping, all database Pods will be gracefully shut down.

When a cluster is stopped, the computing resources of this cluster are released, which means the pods of Kubernetes are released, but the storage resources are reserved. You can start this cluster again to restore it to the state it was in before it was stopped.

Practical Significance:

• Verify whether the database stop/start process is safe and reliable.
• Ensure that data corruption or loss does not occur during the stop/start process.
• Provide the ability to stop non-active clusters on demand to save resource consumption.

Test Examples

Stop

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-stop-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: Stop
```

Start

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-start-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: Start
```

Working Principle:

You can vertically scale a cluster by changing resource requests and limits (CPU and Memory), and KubeBlocks coordinates with the underlying InstanceSets controller to complete vertical scaling.

• In-Place Update: The cluster does not restart, minimizing the impact of instance updates on system availability.
• Non-In-Place Update: KubeBlocks controller detects changes and triggers rolling or hot update strategies to ensure configuration changes take effect without affecting service availability.

Practical Significance:

• Verify whether the system can dynamically adjust CPU/Memory resources to respond to load changes.
• Ensure that data loss or service interruption does not occur during vertical scaling.

Test Examples

Vertical ScaleOut

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-vscale-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: VerticalScaling
  verticalScaling:
  - componentName: mysql
    requests:
      cpu: '1'
      memory: 1Gi
    limits:
      cpu: '1'
      memory: 1Gi
```

Vertical ScaleIn

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-vscale-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: VerticalScaling
  verticalScaling:
  - componentName: mysql
    requests:
      cpu: '0.5'
      memory: 0.5Gi
    limits:
      cpu: '0.5'
      memory: 0.5Gi
```

Working Principle:

KubeBlocks supports ScaleOut and ScaleIn operations and supports scaling both replicas and instances.

• ScaleOut: It supports scaling out the specified replicas and makes the offline instances online again.
• ScaleIn: It supports scaling in the specified replicas and offloading specified instances.

You can perform the horizontal scale by modifying the cluster in a declarative API style or creating an OpsRequest:

• With the declarative API style, users can directly modify the Cluster YAML file to specify the number of replicas for each component and instance template. If the new number of replicas is greater than the current number of Pods, it indicates a scale-out; conversely, if the new number of replicas is less than the current number of Pods, it indicates a scale-in.
• Another approach is to specify the replica count increment in the OpsRequest. The controller will calculate the desired number of replicas based on the current number of Pods in the Cluster's components and the increment value, and perform scaling accordingly.

Practical Significance:

• Verify whether the system can dynamically adjust resources to respond to load changes.
• Ensure that new nodes correctly join the cluster and share the load after expansion.
• Ensure that no data loss or service interruption occurs during contraction.

Test Examples

ScaleOut

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-scale-out-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: mysql
    scaleOut:
      replicaChanges: 1
```

ScaleIn

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-scale-in-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: HorizontalScaling
  horizontalScaling:
  - componentName: mysql
    scaleIn:
      replicaChanges: 1
```

Working Principle:

Volume expansion is triggered by modifying the requested storage size of the PVC, and KubeBlocks ensures that the database file system and data directories expand synchronously.

Practical Significance:

• Verify the elasticity and scalability of the storage layer.
• Ensure that storage capacity can be expanded without downtime when data grows.
• Enhance the long-term availability and stability of the database cluster.

Test Examples

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-expand-volume-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: VolumeExpansion
  volumeExpansion:
  - componentName: mysql
    volumeClaimTemplates:
    - name: data
      storage: 30Gi
```

Working Principle:

Reconstruction of an abnormal database instance involves deleting the target Pod and rebuilding the database instance based on the original data.

• In-Place Rebuild: Rebuild the instance with the same pod name.
• Non-In-Place Rebuild: Create a new instance by horizontalScaling and remove the instance after the new instance is ready.

Practical Significance:

• Verify the capability of reconstructing individual database instances.
• Ensure that the rebuilt instance can quickly recover and rejoin the cluster.
• Improve the self-healing capabilities of the database system.

Test Examples

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-rebuildinstance-ops
  namespace: demo
spec:
  type: RebuildInstance
  clusterName: example-mysql-cluster
  force: true
  rebuildFrom:
    - componentName: mysql
      instances:
        - name: example-mysql-cluster-mysql-0
      backupName: backup-demo-example-mysql-cluster-20250528130909
      inPlace: false
      restoreEnv:
        - name: BACKUP_FOR_STANDBY
          value: "true"
```

Working Principle:

Database switchover refers to the process of switching the role of the primary database to the standby database in a database cluster, making the standby database the new primary database instance.

Switchover operations are usually performed when the primary database fails, undergoes maintenance, or is upgraded to ensure high availability and continuity of database services.

Practical Significance:

• Verify whether the master-slave switchover process is controllable.
• Ensure that data consistency remains unaffected during the switchover.
• Support planned maintenance and load balancing scenarios.

Test Examples

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-switchover-ops
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  type: Switchover
  switchover:
  - componentName: mysql
    instanceName: example-mysql-cluster-mysql-0
```

Working Principle:

KubeBlocks provides the backup and restore function to ensure the safety and reliability of your data. The backup and restore function of KubeBlocks relies on BackupRepo and before using the full backup and restore function, you need to configure BackupRepo first.

KubeBlocks adopts physical backup which takes the physical files in a database as the backup object. You can choose one backup option based on your demands to back up the cluster data on demand or by schedule.

• On-demand backup Based on different backup options, on-demand backup can be further divided into backup tool and snapshot backup.
  • Backup tool: You can use the backup tool of the corresponding data product, such as MySQL XtraBackup and PostgreSQL pg_basebackup. KubeBlocks supports configuring backup tools for different data products.
  • Snapshot backup: If your data is stored in a cloud disk that supports snapshots, you can create a data backup by snapshots. Snapshot backup is usually faster than a backup tool, and thus is recommended.
• Scheduled backup: You can specify retention time, backup method, time, and other parameters to customize your backup schedule.

As for the restore function, KubeBlocks supports restoring data from the backup set.

Practical Significance:

• Verify the effectiveness of data protection mechanisms.
• Ensure that data can be quickly restored in disaster recovery scenarios.
• Meet enterprise-level SLA requirements for data security.

Test Examples

Backup

```yaml
apiVersion: dataprotection.kubeblocks.io/v1alpha1
kind: Backup
metadata:
  name: example-mysql-cluster-backup
  namespace: demo
spec:
  backupMethod: xtrabackup
  backupPolicyName: example-mysql-cluster-mysql-backup-policy
  deletionPolicy: Delete
```

Restore

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-restore
  namespace: demo
spec:
  clusterName: example-mysql-cluster-restored
  force: false
  restore:
    backupName: example-mysql-cluster-backup
    backupNamespace: demo
  type: Restore
```

Working Principle:

KubeBlocks supports dynamic configuration.

This feature simplifies the process of configuring parameters, which saves you from manually configuring database parameters as KubeBlocks handles the updates and configurations automatically to adapt to the new specifications. This saves time and effort and reduces performance issues caused by incorrect configuration.

But it's also important to note that the dynamic parameter configuration doesn't apply to all parameters. Some parameters may require manual configuration.

Additionally, if you have manually modified database parameters before, KubeBlocks may overwrite your customized configurations when updating the database configuration template. Therefore, when using the dynamic configuration feature, it is recommended to back up and record your custom configuration so that you can restore them if needed.

Practical Significance:

• Verify the ability to dynamically adjust database parameters.
• Support flexible optimization of database performance based on different business requirements.
• Improve operational efficiency and reduce the risk of human error in configuration.

Test Examples

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: mysql-reconfigure-dynamic
  namespace: demo
spec:
  clusterName: example-mysql-cluster
  force: false
  reconfigures:
  - componentName: mysql
    parameters:
      - key: max_connections
        value: '100'
  preConditionDeadlineSeconds: 0
  type: Reconfiguring
```

Working Principle:

Control whether the database service is exposed externally by creating or deleting Service resources. Multiple methods are supported, including LoadBalancer and NodePort.

Practical Significance:

• Verify the security and immediacy of network service exposure/hiding.
• Support temporarily opening access permissions for debugging or integration testing.
• Enhance the granularity of database service access control.

Test Examples

Enable

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-expose-enable-ops
  namespace: demo
spec:
  type: Expose
  clusterName: example-mysql-cluster
  expose:
  - componentName: mysql
    services:
    - name: internet
      serviceType: LoadBalancer
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb
        service.beta.kubernetes.io/aws-load-balancer-internal: "false"  # or "true" for an internal VPC IP
      roleSelector: primary
    switch: Enable
```

Disable

```yaml
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: example-mysql-cluster-expose-disable-ops
  namespace: demo
spec:
  type: Expose
  clusterName: example-mysql-cluster
  expose:
  - componentName: mysql
    services:
    - name: internet
      roleSelector: primary
      serviceType: LoadBalancer
    switch: Disable
  preConditionDeadlineSeconds: 0
```

Working Principle:

Update cluster configuration (e.g., Monitor toggle, TerminationPolicy, etc.), and the KubeBlocks controller will detect changes and trigger rolling or hot update strategies to ensure configuration changes take effect while maintaining service availability.

Practical Significance:

• Support dynamic adjustment of cluster parameters, enhancing operational flexibility.
• Verify service continuity and consistency during configuration updates.
• Reduce manual intervention and achieve automated configuration management.

Working Principle:

Use standard clients to connect to the database service to verify whether the cluster can be accessed normally.

Practical Significance:

• Verify that the database service port is reachable and the authentication mechanism is functioning correctly.
• Confirm that the cluster is in a healthy state and capable of accepting connection requests.
• Serve as a foundational verification for other operational and maintenance activities.

Working Principle:

Conduct baseline performance tests on the database cluster using built-in or external testing tools to evaluate metrics such as throughput and response time.

Practical Significance:

• Verify the performance of the database under typical workloads.
• Used for performance comparisons between different versions or configurations.
• Support capacity planning and performance tuning.

After the maintenance operations, the database status is normal and the data is complete, thus the test is passed.