Version: release-0.8

Advanced logs

Currently, KubeBlocks supports viewing the running logs, error logs, and slow logs of the running database cluster through kbcli cluster logs. This method is lightweight and only relies on the log files and tail commands in the target container, without installing any software or plug-ins in the cluster or client.

Its drawbacks are also very obvious. It strongly depends on the survival of the container and the current status of log files. If the container is already abnormal or the log file is deleted, this method cannot work as expected. The above method is called the cluster logs function.

The advanced logs function can better retain log clues at problem sites, support long-term remote backup of log data, and facilitate integration with external systems.

The advanced logs function supports storing log data to the Loki server in the cluster and the remote AWS S3 service. The data stored in Loki supports logcli client and Grafana console access and display. The data stored in S3 supports importing and integrating with other systems for further processing and analysis.

Add-on introduction

Agamotto, Loki, Prometheus, and other components related to observability are managed by KubeBlocks as add-ons. Each of these add-ons has its specialties.

Agamotto is a collection add-on that collects performance and logs.
Loki is a storage add-on that stores log data.
Grafana is a front-end display add-on that displays the observed data.
Logcli is a front-end search add-on that provides the native LogQL query capability. It is installed by kbcli plugin.

Before you start

Enable advanced logs function

Enable Loki

The Loki add-on stores the log data and accepts the search request from the front end. Refer to the Loki official documentation for details.

caution

Before enabling the advanced logs function, deploy a Loki node first and then apply Agamotto collection to avoid log data loss.

View the Loki add-on status to check whether it is enabled.
```
kbcli addon list | grep loki
```

If this add-on is not enabled, run the command below to enable this add-on.

This command runs Loki in the statefulset mode. KubeBlocks deploys a single-node Loki of the single binary type, i.e. the target in the configuration is all, uses a 10-GB PV to store data, and starts a loki-gateway service to receive data.

kbcli addon enable loki
>
NAME                           TYPE   PROVIDER    STATUS     AUTO-INSTALL   AUTO-INSTALLABLE-SELECTOR   
loki                           Helm   apecloud    Enabled    false   

Check whether this Pod is running.

kubectl get pods | grep loki
>
NAME                                            READY   STATUS    RESTARTS   AGE
kb-addon-loki-0                                 1/1     Running   0          47h
loki-gateway-5f4895c7b-5khv7                    1/1     Running   0          47h

Run Grafana to view the status of Loki. For example,
(Optional) Disable the Loki add-on.
```
kbcli disabled loki
```

By default, Loki saves the log data of the last 3 days, and the data older than 72 hours will be automatically deleted. You can use the following commands to adjust the storage policy. Note that when adjusting the policy, you need to pay attention to the host's disk resources and Loki PVC. Currently, Loki does not support size-based retention.

# The new retention_period must be an integer multiple of 24
kbcli addon enable loki --set loki.loki.limits_config.retention_period=168h

Run kubectl get pv to view the PV information of the current Loki instance.

kubectl get pv
>
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                               STORAGECLASS      REASON   AGE
pvc-ed20ec94-9a58-46e4-9c28-b692cba70e79   8Gi        RWO            Delete           Bound    kb-system/storage-kb-addon-loki-0   csi-hostpath-sc            28h

If you want to check the disk usage of the current Loki node, you can use the exec command to log in to the container and use the du command to check the disk usage. For example,

kubectl exec -it kb-addon-loki-0 -n kb-system -- /bin/sh

Run the du command.

du -sh /var/loki/*
>
0K  /var/loki/boltdb-shipper-active
0K  /var/loki/boltdb-shipper-cache
4M    /var/loki/chunks
0K    /var/loki/compactor
0K   /var/loki/retention
0K    /var/loki/rules
5M    /var/loki/wal

Enable Agamotto

After the log collection function is enabled, Agamotto detects two localhost folders, /var/log/pods and /var/log/kubeblocks. All files that fit the following patterns will be collected.

/var/log/pods directory: it stores the standard output of the storage container and the error logs. The file path pattern follows /var/log/pods/**/**/*.log.
/var/log/kubeblocks directory: it can customize the storage of log files storage. By default, KubeBlocks stores the logs of database engines, such as error log and slow log. The file path pattern follows /var/log/kubeblocks/**/**/*.log.

View the Agamotto add-on status to check whether it is enabled.
```
kbcli addon list | grep agamotto
```

If this add-on is not enabled, run the command below to enable this add-on.

kbcli addon enable agamotto
>
NAME                           TYPE   PROVIDER    STATUS     AUTO-INSTALL   AUTO-INSTALLABLE-SELECTOR   
agamotto                       Helm   apecloud    Enabled    false  

Check whether this Pod is running.

kubectl get pods | grep agamotto
>
NAME                                            READY   STATUS    RESTARTS   AGE
kb-addon-agamotto-kvbr5                         1/1     Running   0          44h

Run the command to upgrade the log function of Agamotto.
```
kbcli addon enable agamotto --set log.enabled=true
```
Upload logs to S3.
By default, the logs function only restores the log data on the Loki server of the current Kubernetes cluster and does not upload the log data to the remote S3 server.
Fill in the command below with Access Key, Secret Key, region, and bucket, and run this command to enable uploading log data to S3.
```
kbcli addon enable agamotto --set log.enabled=true,log.s3.enabled=true,log.s3.accessKey=user_ak,log.s3.secretKey=user_sk,log.s3.region=user_region,log.s3.bucket=user_bucket
```
The target bucket of the remote S3 creates a directory named after the filename, which stores the real-time data Partition of the corresponding log (partitioned by 5 minutes or 5 MB).
(Optional) Disable the Agamotto add-on.
```
kbcli addon disable agamotto
```

Enable Grafana

note

In the production environment, all monitoring add-ons are disabled by default when installing KubeBlocks and Grafana is also included. You can enable Grafana but it is highly recommended to build your monitoring system or purchase a third-party monitoring service for safety and stability concerns. For integrating a third-party monitoring service, refer to Integrate a third party monitoring service.

Check whether the Loki and Grafana services are enabled.

kbcli dashboard list
>
NAME                 NAMESPACE   PORT    CREATED-TIME                 
kubeblocks-grafana   default     13000   Jul 08,2023 15:37 UTC+0800   
kubeblocks-logs      default     13100   Jul 08,2023 15:36 UTC+0800  

Open the dashboard of the container logs.
```
kbcli dashboard open kubeblocks-logs
```
Or you can go to the Explore page and select Loki to search the logs.

Enable LogCLI

kbcli supports the LogCLI plug-in for your convenience of querying logs by command line.

note

For more LogCLI and LogQL grammar, refer to the Loki official documentation.

Install the LogCLI plug-in.

Set the environment variable.

export PATH="${KBCLI_ROOT:-$HOME/.kbcli}/plugins/bin:$PATH"

Add the block-index of ApeCloud as the krew index.

kbcli plugin index add default https://github.com/apecloud/block-index.git

View the index status.

kbcli plugin index list
>
INDEX     URL                                           
default   https://github.com/apecloud/block-index.git   

Install the LogCLI plug-in.
```
kbcli plugin install default/logcli 
```
View the plug-in status and check whether LogCLI is installed successfully.
```
kbcli plugin list | grep logcli
```

Use the LogCLI.

Set the environment variable of LogCLI. For example, set the LOKI_ADDR pointing to the localhost 3100 port.
```
export LOKI_ADDR=http://localhost:3100
```
Export the Loki service to the localhost. Note that port/namespace should be the same as the settings in the system. For example,
```
kubectl port-forward svc/loki-gateway 3100:80 -n kb-system
```

Use the LogCLi tool to query log data.

kbcli logcli query '{exporter="OTLP"}'
>
2023/07/10 17:30:18 http://localhost:3100/loki/api/v1/query_range?direction=BACKWARD&end=1688981418606367000&limit=30&query=%7Bexporter%3D%22OTLP%22%7D&start=1688977818606367000
2023/07/10 17:30:18 Common labels: {exporter="OTLP", host_ip="0.0.0.0", host_name="minikube", log_file_name="0.log", restart_num="0"}
2023-07-10T17:30:18+08:00 {container="nginx", log_file_path="/var/log/pods/kb-system_loki-gateway-748cbf767f-6jpdw_54c6db59-10d5-419b-963e-0f116358e14a/nginx/0.log", namespace="kb-system", pod="loki-gateway-748cbf767f-6jpdw", pod_id="54c6db59-10d5-419b-963e-0f116358e14a"}                                                             {"log":"172.17.0.1 - - [10/Jul/2023:09:30:18 +0000]  204 \"POST /loki/api/v1/push HTTP/1.1\" 0 \"-\" \"Go-http-client/1.1\" \"-\"\n","stream":"stderr","time":"2023-07-10T09:30:18.230393596Z"}
2023-07-10T17:30:18+08:00 {container="nginx", log_file_path="/var/log/pods/kb-system_loki-gateway-748cbf767f-6jpdw_54c6db59-10d5-419b-963e-0f116358e14a/nginx/0.log", namespace="kb-system", pod="loki-gateway-748cbf767f-6jpdw", pod_id="54c6db59-10d5-419b-963e-0f116358e14a"}

MySQL Slow logs and error logs

By default, in the ClusterDefinition of MySQL, there is a sidecar container named metrics in the podSpec definition of the mysql component.

This metrics container detects the error logs and slow logs output by MySQL in real-time, and dumps the logs to the host's /var/log/KubeBlocks directory. To prevent taking up too much host disk space, the default size of each file is 10 MB, and a maximum of 2 files can be saved.

In addition, the Agamotto component deployed on the host detects the /var/log/KubeBlocks folder in real-time and collects the log data in it to complete the entire log data collection link.

- name: metrics
  image:  {{ .Values.metrics.image.registry | default "docker.io" }}/{{ .Values.metrics.image.repository }}:{{ .Values.metrics.image.tag }}
  imagePullPolicy: {{ .Values.metrics.image.pullPolicy | quote }}
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
  env:
    - name: DB_TYPE
      value: MySQL
    - name: ENDPOINT
      value: {{ printf "localhost:3306" }}
    - name: MYSQL_USER
      valueFrom:
        secretKeyRef:
          name: $(CONN_CREDENTIAL_SECRET_NAME)
          key: username
          optional: false
    - name: MYSQL_PASSWORD
      valueFrom:
        secretKeyRef:
          name: $(CONN_CREDENTIAL_SECRET_NAME)
          key: password
          optional: false
  command:
    - "/bin/agamotto"
    - "--config=/opt/agamotto/agamotto-config.yaml"
  ports:
    - name: http-metrics
      containerPort: {{ .Values.metrics.service.port }}
  volumeMounts:
    - name: agamotto-configuration
      mountPath: /opt/agamotto
    - name: data
      mountPath: /data/mysql
    - name: log-data
      mountPath: /var/log/kubeblocks
      readOnly: false

Advanced logs

Add-on introduction​

Before you start​

Enable advanced logs function​

Enable Loki​

Enable Agamotto​

Enable Grafana​

Enable LogCLI​

MySQL Slow logs and error logs​