Configure alert

Configure IM alert

Alerts are mainly used for daily error response to improve system availability. KubeBlocks has a built-in set of common alert rules and integrates multiple notification channels. The alert capability of KubeBlocks can meet the operation and maintenance requirements of production-level online clusters.

note

The alert function is the same for all.

Alert rules

The built-in generic alert rules of KubeBlocks meet the needs of various data products and provide an out-of-the-box experience without further configurations. These alert rules provide the best practice for cluster operation and maintenance, which further improve alert accuracy and reduce the probability of false negatives and false positives by experience-based smoothing windows, alert thresholds, alert levels, and alert indicators.

Taking PostgreSQL as an example, the alert rules have built-in common abnormal events, such as instance down, instance restart, slow query, connection amount, deadlock, and cache hit rate.

The following example shows PostgreSQL alert rules (refer to Prometheus for syntax). When the amount of active connections exceeds 80% of the threshold and lasts for 2 minutes, Prometheus triggers a warning alert and sends it to AlertManager.

alert: PostgreSQLTooManyConnections
  expr: |
    sum by (namespace,app_kubernetes_io_instance,pod) (pg_stat_activity_count{datname!~"template.*|postgres"})
    > on(namespace,app_kubernetes_io_instance,pod)
    (pg_settings_max_connections - pg_settings_superuser_reserved_connections) * 0.8
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "PostgreSQL too many connections (> 80%)"
    description: "PostgreSQL has too many connections and the value is {{ $value }}. (instance: {{ $labels.pod }})"

You can view all the built-in alert rules in Alerts Tab of Prometheus Dashboards. Run the commands below to open Prometheus Dashboards.

# View dashboards list
kbcli dashboard list

# Open Prometheus Dashboards
kbcli dashboard open kubeblocks-prometheus-server # Here is an example and fill in the actual name based on the above dashboard list

Configure IM alert

The alert message notification of KubeBlocks mainly adopts the AlertManager native capability. After receiving the Prometheus alert, KubeBlocks performs steps including deduplication, grouping, silence, suppression, and routing, and finally sends it to the corresponding notification channel.

AlertManager integrates a set of notification channels, such as Email and Slack. KubeBlocks extends new IM class notification channels with AlertManger Webhook.

This tutorial takes configuring Feishu as the notification channel as an example.

Before you start

To receive alerts, you need to deploy monitoring add-ons and enable cluster monitoring first. Refer to Monitor database for details.

Steps:

1. Configure alert channels.

   Refer to the following guides to configure your alert channels.

   • Feishu custom bot
   • DingTalk custom bot
   • WeChat Enterprise custom bot
   • Slack
   note

   • Each notification channel has its interface call volume and frequency limits and when the limits are reached, the channel will limit traffic and you cannot receive alerts.
   • The SLA of the service provided by a single channel cannot guarantee the alerts are sent successfully. Therefore, it is recommended to configure multiple notification channels to ensure availability.

2. Configure the receiver.

   To improve ease of use, kbcli develops the alert subcommand to simplify the receiver configuration. You can set the notification channels and receivers by the alert subcommand. This subcommand also supports condition filters, such as cluster names and severity levels. After the configuration succeeds, it takes effect dynamically without the service restarting.

   1. Add an alert receiver.

      kbcli alert add-receiver --webhook='xxx' --cluster=xx --severity=xx

      Example

      The following commands show how to add a receiver to Feishu based on different requirements. The webhook address below is an example and you need to replace it with the actual address before running the command.

      # Signature authentication is disabled
      kbcli alert add-receiver \
      --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo'
      
      # Signature authentication is enabled and sign is used as the value of token
      kbcli alert add-receiver \
      --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo,token=sign'
      
      # Only receive the alerts from a cluster named mysql-cluster
      kbcli alert add-receiver \
      --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo' --cluster=mysql-cluster
      
      # Only receive the critical alerts from a cluster named mysql-cluster
      kbcli alert add-receiver \
      --webhook='url=https://open.feishu.cn/open-apis/bot/v2/hook/foo' --cluster=mysql-cluster --severity=critical
      note

      For the detailed command description, run kbcli alert add-receiver -h.

   2. View the receiver list to verify whether the new receiver is added.

      You can also view the notification configurations by this command.

      kbcli alert list-receivers

   3. (Optional) Delete the notification channel and receiver if you want to disable the alert function.

      kbcli alert delete-receiver <receiver-name>

IM alert troubleshooting

If you cannot receive alert notices, run the commands below to troubleshoot the logs of AlertManager and AlertManager-Webhook-Adaptor add-ons.

# Find the corresponding Pod of AlertManager and get Pod name
kubectl get pods -n kb-system -l 'app=prometheus,component=alertmanager'

# Search AlertManeger logs
kubectl logs <pod-name> -n kb-system -c prometheus-alertmanager

# Find the corresponding Pod of AlertManager-Webhook-Adaptor and get Pod name
kubectl get pods -n kb-system -l 'app.kubernetes.io/name=alertmanager-webhook-adaptor'

# Search AlertManager-Webhook-Adaptor logs
kubectl logs <pod-name> -n kb-system -c alertmanager-webhook-adaptor

Configure email alert

KubeBlocks also supports email alert.

1. Configure SMTP server.

   kbcli alert config-smtpserver 
   --smtp-from alert-test@apecloud.com \
   --smtp-smarthost smtp.feishu.cn:587 \
   --smtp-auth-username alert-test@apecloud.com \
   --smtp-auth-password 123456abc \
   --smtp-auth-identity alert-test@apecloud.com

2. View the SMTP server list to verify whether the above server is added successfully.

   You can also view the configuration details by this command.

   kbcli alert list-smtpserver

3. Add email receiver.

   kbcli alert add-receiver --email='user1@kubeblocks.io'

   KubeBlocks email alert now supports receiving emails from a specified cluster and of a certain severity. You can set this function by using --cluster and --severity flags.

   • --cluster: means only receiving emails from a specified cluster.

     kbcli alert add-receiver --email='user1@kubeblocks.io,user2@kubeblocks.io' --cluster=mycluster

   • --severity: means only receiving emails from a specified cluster and the alert severity is warning.

     kbcli alert add-receiver --email='user1@kubeblocks.io,user2@kubeblocks.io' --cluster=mycluster --severity=warning