High availability
KubeBlocks integrates the open-source Patroni solution to realize high availability and adopts Noop as the switch policy.
Before you start
Check whether the following role probe parameters exist to verify the role probe is enabled.
kubectl get cd postgresql -o yaml
>
probes:
roleProbe:
failureThreshold: 3
periodSeconds: 2
timeoutSeconds: 1
Steps
View the initial status of the PostgreSQL cluster.
kbcli cluster describe pg-cluster
Currently,
pg-cluster-postgresql-0
is the primary pod andpg-cluster-postgresql-1
is the secondary pod.Simulate a primary pod exception.
# Enter the primary pod
kubectl exec -it pg-cluster-postgresql-0 -- bash
# Delete the data directory of PostgreSQL to simulate an exception
root@postgres-postgresql-0:/home/postgres# rm -fr /home/postgres/pgdata/pgroot/dataView logs to observe how the roles of pods switch when an exception occurs.
# View the primary pod logs
kubectl logs pg-cluster-postgresql-0In the logs, the leader lock is released from the primary pod and an HA switch occurs.
2023-04-18 08:06:52,338 INFO: Lock owner: pg-cluster-postgresql-0; I am pg-cluster-postgresql-0
2023-04-18 08:06:52,460 INFO: Leader key released
2023-04-18 08:06:52,552 INFO: released leader key voluntarily as data dir empty and currently leader
2023-04-18 08:06:52,553 INFO: Lock owner: pg-cluster-postgresql-1; I am pg-cluster-postgresql-0
2023-04-18 08:06:52,553 INFO: trying to bootstrap from leader 'pg-cluster-postgresql-1'# View secondary pod logs
kubectl logs pg-cluster-postgresql-1In the logs, the original secondary pod has obtained the lock and become the leader.
2023-04-18 08:07:14,441 INFO: no action. I am (pg-cluster-postgresql-1), the leader with the lock
2023-04-18 08:07:24,354 INFO: no action. I am (pg-cluster-postgresql-1), the leader with the lockConnect to the PostgreSQL cluster to view the replication information.
kbcli cluster connect pg-cluster
postgres=# select * from pg_stat_replication;
From the output,
pg-cluster-postgresql-0
has been assigned as the secondary pod.Describe the cluster and check the instance role.
kbcli cluster describe pg-cluster
After the failover,
pg-cluster-postgresql-0
becomes the secondary pod andpg-cluster-postgresql-1
becomes the primary pod.