ZooKeeper Monitoring with Prometheus Operator

This guide demonstrates how to configure comprehensive monitoring for ZooKeeper clusters in KubeBlocks using:

1. Prometheus Operator for metrics collection
2. Built-in ZooKeeper metrics exporter (port 7000)
3. Grafana for visualization

Prerequisites

Before proceeding, verify your environment meets these requirements:

• A functional Kubernetes cluster (v1.21+ recommended)
• kubectl v1.21+ installed and configured with cluster access
• Helm installed (installation guide)
• KubeBlocks installed (installation guide)
• ZooKeeper Add-on installed and a ZooKeeper cluster running (see Quickstart)

Install Monitoring Stack

1. Install Prometheus Operator

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring \
  --create-namespace

2. Verify Installation

kubectl get pods -n monitoring

NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          114s
prometheus-grafana-75bb7d6986-9zfkx                      3/3     Running   0          2m
prometheus-kube-prometheus-operator-7986c9475-wkvlk      1/1     Running   0          2m
prometheus-kube-state-metrics-645c667b6-2s4qx            1/1     Running   0          2m
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   0          114s

Deploy a ZooKeeper Cluster

Deploy a ZooKeeper cluster with metrics enabled (enabled by default):

kubectl apply -f https://raw.githubusercontent.com/apecloud/kubeblocks-addons/refs/heads/main/examples/zookeeper/cluster.yaml

ZooKeeper exposes Prometheus metrics on port 7000 at /metrics path on each pod.

Verify Cluster Status

kubectl get cluster zookeeper-cluster -n demo

NAME                CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
zookeeper-cluster                        Delete               Running   3m

Verify Metrics Endpoint

kubectl exec -n demo zookeeper-cluster-zookeeper-0 -- \
  curl -s http://localhost:7000/metrics | head -10

# HELP read_final_proc_time_ms read_final_proc_time_ms
# TYPE read_final_proc_time_ms summary
read_final_proc_time_ms{quantile="0.5",} NaN
read_final_proc_time_ms{quantile="0.9",} NaN
read_final_proc_time_ms{quantile="0.99",} NaN
read_final_proc_time_ms_count 0.0
read_final_proc_time_ms_sum 0.0
# HELP zookeeper_version Version of Zookeeper.
# TYPE zookeeper_version gauge
zookeeper_version{version="3.9.2-...",} 1.0

Configure Metrics Collection

Create PodMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: zk-cluster-pod-monitor
  namespace: demo
  labels:               # Must match 'prometheus.spec.podMonitorSelector'
    release: prometheus
spec:
  jobLabel: app.kubernetes.io/managed-by
  podTargetLabels:
  - app.kubernetes.io/instance
  - app.kubernetes.io/managed-by
  - apps.kubeblocks.io/component-name
  - apps.kubeblocks.io/pod-name
  podMetricsEndpoints:
    - path: /metrics
      port: metrics       # Port named 'metrics' on port 7000
      scheme: http
  namespaceSelector:
    matchNames:
      - demo
  selector:
    matchLabels:
      app.kubernetes.io/instance: zookeeper-cluster
      apps.kubeblocks.io/component-name: zookeeper

Apply it:

kubectl apply -f https://raw.githubusercontent.com/apecloud/kubeblocks-addons/refs/heads/main/examples/zookeeper/pod-monitor.yaml

PodMonitor Configuration Guide

Verify Monitoring Setup

Check Prometheus Targets

kubectl port-forward svc/prometheus-kube-prometheus-prometheus -n monitoring 9090:9090

Open: http://localhost:9090/targets

Look for targets with app_kubernetes_io_instance="zookeeper-cluster". Since jobLabel: app.kubernetes.io/managed-by is set, the job name will be kubeblocks. All 3 ZooKeeper pod targets (one per ensemble member) should show state UP, each scraping port 7000.

Test Metrics Collection

curl -sG "http://localhost:9090/api/v1/query" \
  --data-urlencode 'query=up{app_kubernetes_io_instance="zookeeper-cluster"}' | jq

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "up",
          "app_kubernetes_io_instance": "zookeeper-cluster",
          "app_kubernetes_io_managed_by": "kubeblocks",
          "apps_kubeblocks_io_component_name": "zookeeper",
          "apps_kubeblocks_io_pod_name": "zookeeper-cluster-zookeeper-0",
          "container": "zookeeper",
          "endpoint": "metrics",
          "instance": "10.0.6.134:7000",
          "job": "kubeblocks",
          "namespace": "demo",
          "pod": "zookeeper-cluster-zookeeper-0"
        },
        "value": [1746000000, "1"]
      }
    ]
  }
}

Visualize in Grafana

Access Grafana

kubectl port-forward svc/prometheus-grafana -n monitoring 3000:80

Open http://localhost:3000 and log in:

• Username: admin
• Password: prom-operator (default)

Key ZooKeeper Metrics to Monitor

Cleanup

kubectl delete cluster zookeeper-cluster -n demo
kubectl delete podmonitor zk-cluster-pod-monitor -n demo
kubectl delete ns demo