High availability

KubeBlocks integrates the open-source Patroni solution to realize high availability and adopts Noop as the switch policy.

Before you start

• Install kbcli.

• Install KubeBlocks: You can install KubeBlocks by kbcli or by Helm.

• Create a PostgreSQL Replication Cluster.

• Check the Switch Policy and the role probe.

  • Check whether the switch policy is Noop.

    kubectl get cluster pg-cluster -o yaml
    >
    spec:
      componentSpecs:
      - name: postgresql
        componentDefRef: postgresql
        switchPolicy:
          type: Noop

  • Check whether the following role probe parameters exist to verify the role probe is enabled.

    kubectl get cd postgresql -o yaml
    >
    probes:
      roleProbe:
        failureThreshold: 3
        periodSeconds: 2
        timeoutSeconds: 1

Steps

1. View the initial status of the PostgreSQL cluster.

   kbcli cluster describe pg-cluster

   

   Currently, pg-cluster-postgresql-0 is the primary pod and pg-cluster-postgresql-1 is the secondary pod.

2. Simulate a primary pod exception.

   # Enter the primary pod
   kubectl exec -it pg-cluster-postgresql-0  -- bash
   
   # Delete the data directory of PostgreSQL to simulate an exception
   root@postgres-postgresql-0:/home/postgres# rm -fr /home/postgres/pgdata/pgroot/data

3. View logs to observe how the roles of pods switch when an exception occurs.

   # View the primary pod logs
   kubectl logs pg-cluster-postgresql-0

   In the logs, the leader lock is released from the primary pod and an HA switch occurs.

   2023-04-18 08:06:52,338 INFO: Lock owner: pg-cluster-postgresql-0; I am pg-cluster-postgresql-0
   2023-04-18 08:06:52,460 INFO: Leader key released
   2023-04-18 08:06:52,552 INFO: released leader key voluntarily as data dir empty and currently leader
   2023-04-18 08:06:52,553 INFO: Lock owner: pg-cluster-postgresql-1; I am pg-cluster-postgresql-0
   2023-04-18 08:06:52,553 INFO: trying to bootstrap from leader 'pg-cluster-postgresql-1'

   # View secondary pod logs
   kubectl logs pg-cluster-postgresql-1

   In the logs, the original secondary pod has obtained the lock and become the leader.

   2023-04-18 08:07:14,441 INFO: no action. I am (pg-cluster-postgresql-1), the leader with the lock
   2023-04-18 08:07:24,354 INFO: no action. I am (pg-cluster-postgresql-1), the leader with the lock

4. Connect to the PostgreSQL cluster to view the replication information.

   kbcli cluster connect pg-cluster

   postgres=# select * from pg_stat_replication;

   

   From the output, pg-cluster-postgresql-0 has been assigned as the secondary's pod.

5. Describe the cluster and check the instance role.

   kbcli cluster describe pg-cluster

   

   After the failover, pg-cluster-postgresql-0 becomes the secondary pod and pg-cluster-postgresql-1 becomes the primary pod.