Store and Manage Logs of Percona Operator Pods with PMM and Grafana Loki

While it is convenient to view the log of MySQL or MongoDB pods with kubectl logs, sometimes the log is purged when the pod is deleted, which makes searching historical logs a bit difficult. Grafana Loki, an aggregation logging tool from Grafana, can be installed in the existing Kubernetes environment to help store historical logs and build a comprehensive logging stack.

What Is Grafana Loki

Grafana Loki is a set of tools that can be integrated to provide a comprehensive logging stack solution. Grafana-Loki-Promtail is the major component of this solution. The Promtail agent collects and ships the log to the Loki datastore and then visualizes logs with Grafana. While collecting the logs, Promtail can label, convert and filter the log before sending. Next step, Loki receives the logs and indexes the metadata of the log. At this step, the logs are ready to be visualized in the Grafana and the administrator can use Grafana and Loki’s query language, LogQL, to explore the logs. 

Installing Grafana Loki in Kubernetes Environment

We will use the official helm chart to install Loki. The Loki stack helm chart supports the installation of various components like promtail, fluentd, Prometheus and Grafana. 

For a quick introduction, we will install only loki-stack and promtail. We will use the Grafana pod, that is deployed by Percona Monitoring and Management to visualize the log. 

Let’s see what has been installed:

Promtail and loki-stack pods have been created, together with a service that loki-stack will use to publish the logs to Grafana for visualization.

You can see the promtail pods are deployed by a daemonset  which spawns a promtail pod in every node. This is to make sure the logs from every pod in all the nodes are collected and shipped to the loki-stack  pod for centralizing storing and management.

Integrating Loki With PMM Grafana

Next, we will add Loki as a data source of PMM Grafana, so we can use PMM Grafana to visualize the logs. You can do it from the GUI or with kubectl CLI. 

Below is the step to add data source from PMM GUI:

Navigate to https://<PMM-IP-addres>:9443/graph/datasources  then select Add data source 

Then Select Loki

Next, in the Settings, in the HTTP URL box, enter the DNS records of the loki-stack  service 

You can also use the below command to add the data source. Make sure to specify the name of PMM pod, in this command, monitoring-0  is the PMM pod. 

Exploring the Logs in PMM Grafana

Now, you can explore the pod logs in Grafana UI, navigate to Explore, and select Loki in the dropdown list as the source of metrics:

In the Log Browser, you can select the appropriate labels to form your first LogQL query, for example, I select the following attributes

  {app="percona-xtradb-cluster", component="pxc", container="logs",pod="cluster1-pxc-1"}  

Click Show logs  and you can see all the logs of cluster-pxc-1 pod

You can perform a simple filter with |= “message-content”. For example, filtering all the messages related to State transfer  by

Conclusion

Deploying Grafana Loki in the Kubernetes environment is feasible and straightforward. Grafana Loki can be integrated easily with Percona Monitoring and Management to provide both centralized logging and comprehensive monitoring when running Percona XtraDB Cluster and Percona Server for MongoDB in Kubernetes.

It would be interesting to know if you have any thoughts while reading this, please share your comments and thoughts.