Configure monitoring
This tutorial takes Oracle MySQL as an example and explains how to configure monitoring in KubeBlocks. You can refer to the full PR.
Before you start
- Knowledge about basic Kubernetes concepts, such as Pod and Sidecar.
- Finish Tutorial 1.
- Knowledge about basic monitoring system concepts, such as Prometheus and Grafana.
Introduction
Monitoring is an essential part of Kubernetes observability. It helps developers check the system's operational status to quickly identify issues.
Kubeblocks currently integrates Prometheus and Grafana as add-ons. In this tutorial, you will learn how to integrate the Prometheus/Grafana solution.
Prometheus Overview
Prometheus provides an open-source monitoring solution that integrates metric collection, metric storage, and alert capabilities.
It is widely used in cloud-native, containerized, and microservices architectures. With Prometheus, developers and operations teams can monitor the performance and health status of applications in real-time, so as to quickly identify and resolve issues to ensure application reliability and availability. Prometheus is usually used with Grafana to create powerful monitoring and observability solutions.
Grafana Overview
Grafana is an open-source analytics and monitoring platform widely used for visualizing time series data. It allows users to create interactive and customizable dashboards to monitor and analyze data from various sources.
📎 Table 1. Terminology
| Term | Description |
|---|---|
| Prometheus Exporter | Prometheus Exporter is a component that collects monitoring data and provides data to external entities using the Prometheus monitoring specification. For more details, refer to Prometheus Exporter List. |
| Prometheus Metrics | Prometheus Metrics are data points used for monitoring and performance analysis. They typically include request counts, response times, CPU usage, and memory usage. |
| Grafana Dashboard | Grafana Dashboard is a visualization interface used to present data. It is commonly used for monitoring and analyzing various time series data. |
Prometheus in KubeBlocks has already been configured with scraping jobs, so developers only need to configure the Exporter. In KubeBlocks, the Exporter is deployed as a sidecar alongside the main container of the database engine in the same Pod.
Configure Exporter
First, choose an Exporter. This tutorial is based on Oracle MySQL, so a MySQL Exporter is needed.
Configure Exporter version
Modify ClusterVersion (clusterversion.yaml).
You can find an appropriate Exporter from open-source communities(e.g., Prometheus in Docker).
componentVersions:
- componentDefRef: mysql-compdef
versionsContext:
containers:
- name: mysql-container
image: ...
imagePullPolicy: ..
- name: mysql-exporter
image: prom/mysqld-exporter:v0.14.0
Specify the image of mysql-exporter as prom/mysqld-exporter with the version 0.14.0.
Add an Exporter container
Modify clusterdefinition.yaml and configure mysql-exporter in Sidecar form.
podSpec:
containers:
# mysql container and other containers ->
- name: mysql-exporter
ports:
- name: metrics
containerPort: 9104
protocol: TCP
env:
- name: "MYSQL_MONITOR_USER"
valueFrom:
secretKeyRef:
name: $(CONN_CREDENTIAL_SECRET_NAME)
key: username
- name: "MYSQL_MONITOR_PASSWORD"
valueFrom:
secretKeyRef:
name: $(CONN_CREDENTIAL_SECRET_NAME)
key: password
- name: "DATA_SOURCE_NAME"
value: "$(MYSQL_MONITOR_USER):$(MYSQL_MONITOR_PASSWORD)@(localhost:3306)/"
As shown, a new container has been added to the original PodSpec. DATA_SOURCE_NAME is an environment variable specific to the mysql-exporter.
Different Exporters require different environment variables, and they need to be configured based on specific features of each Exporter.
As mentioned, this tutorial uses mysql exporter 0.14.0. In the latest mysql exporter version 0.15.0, the variable DATA_SOURCE_NAME is no longer supported.
Configure monitor parameters
Modify clusterdefinition.yaml and configure monitor parameters.
componentDefs:
- name: mysql-compdef
characterType: mysql
service: ..
monitor:
exporterConfig:
scrapePort: 9104 # Listening port of the Exporter, used by Prometheus to pull data
scrapePath: "/metrics" # Path of the Exporter path, used by Prometheus to pull data
KubeBlocks supports multiple monitoring solutions. To use the open-source Prometheus/Grafana solution, configure the listening port and metrics path in monitor, which should correspond to the container-port specified in 2. Add an Exporter container.
Configure Grafana Dashboard
Obtain Grafana Dashboard configurations
Grafana Dashboard can help users monitor, analyze, and understand data in real-time. For popular databases, various dashboard configuration files (in JSON format) can be easily found.
Add to your cluster
Import the downloaded JSON files on the Grafana Dashboard page or configure them in your cluster template.
The latter option is more versatile, as the same configuration can be reused for any cluster generated through the template.
Therefore, two files are added to the existing Helm chart.
- dashboards: Save dashboard JSON files.
- grafana: Create a ConfigMap to store the contents of dashboard JSON files.
tree oracle-mysql
.
├── Chart.yaml
├── dashboards
│ └── mysql.json
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── clusterdefinition.yaml
│ └── clusterversion.yaml
│ └── grafana
│ └── configmap-dashboards.yaml
└── values.yaml
4 directories, 8 files
Monitor cluster data
Prepare the environment and enable Prometheus monitoring components
Run kbcli addon list to check if the following add-ons are enabled (status: Enabled):
kbcli addon list
>
...
grafana Helm Enabled true
alertmanager-webhook-adaptor Helm Enabled true
prometheus Helm Enabled alertmanager true
...
If not (status: Disabled), enable them one by one.
kbcli addon enable prometheus
kbcli addon enable alertmanager-webhook-adaptor
kbcli addon enable grafana
Then you can have access to the integrated three dashboards:
kbcli dashboard list
>
NAME NAMESPACE PORT CREATED-TIME
kubeblocks-grafana kb-system 13000 Jul 24,2023 11:38 UTC+0800
kubeblocks-prometheus-alertmanager kb-system 19093 Jul 24,2023 11:38 UTC+0800
kubeblocks-prometheus-server kb-system 19090 Jul 24,2023 11:38 UTC+0800
Create a database cluster
Install a cluster template.
helm install oracle-mysql ./path-to-your-helm-chart/oracle-mysqlEnable monitoring function.
Opt 1. Enable when creating a cluster
kbcli cluster create mycluster --cluster-definition='oracle-mysql' --monitor='true'Opt 2. Enable in an existing cluster
kbcli cluster update mycluster --monitor='true'
Open the dashboard
# View available dashboards
kbcli dashboard list
>
NAME NAMESPACE PORT CREATED-TIME
kubeblocks-grafana default 3000 Jan 13,2023 10:53 UTC+0800
kubeblocks-prometheus-alertmanager default 9093 Jan 13,2023 10:53 UTC+0800
kubeblocks-prometheus-server default 9090 Jan 13,2023 10:53 UTC+0800
# Select Grafana and open the web console in the default browser
kbcli dashboard open kubeblocks-grafana
Sign in to the dashboard
If the dashboard requires a login, use the following username and password.
Username: admin
Password: kubeblocks
Summary
This tutorial explains how to quickly adapt the Prometheus/Grafana solution to monitor your database cluster. KubeBlocks will also introduce a monitoring solution based on OpenTelemetry in the future. Stay tuned for updates.
References
Appendix
A.1 Disable cluster monitoring
In KubeBlocks, you can enable or disable monitoring for a specific cluster using kbcli.
kbcli cluster update mycluster --monitor='false'