Percona Monitoring and Management and Datadog Integration

So your company is using Datadog on the global level. Still, as DBA, you want to use Percona Monitoring and Management (PMM) for better database insights and maintain a common policy of Datadog as a standard alerting or high-level observability tool.

Percona Monitoring and Management is not intended to replace Datadog; rather, they work better together. The first can provide you with in-depth database monitoring without vendor lock-in or licensing costs, while the second provides you with end-to-end observability beyond database infrastructure.

In this blog post, I’ll show you a simple example of collecting all rich data about databases using PMM and a way to send some of it to Datadog for general dashboards with main metrics to monitor. We’ll use here “up” metrics for different databases to reflect database status.

Setting up environments

Let’s prepare our environment. I’ll use a free Datadog account for this experiment.

Note: while it’s https://www.datadoghq.com/, you may end up on  https://app.datadoghq.eu/monitors/create, and this is an EU domain change (This com->eu caused me several hours of investigating why it’s not working as expected)

Next, I need an installed and working PMM with several databases added to it. To install PMM, I’ll follow the instructions from the official documentation in installing pmm and add several databases to PMM.

Integrate the tools

Now it’s time to start connecting both tools.

Preparing Datadog side:  You can find a complete list of Available Datadog agents for different OS inside the user account.

Because I am using an EC2 instance for my PMM installation, I’ll use an Amazon Linux agent.

I also need to select an API key for this agent, and after it’s set, I can have the command I need to execute inside my ES2 with PMM installed inside it.

I have a command like:

After execution, I can see the logs of Datadog agent installation and configuration and at the end:

To confirm that the agent has been added and working, I’ll go to the Infrastructure dashboard in Datadog UI and see the recently added host in the Hostname list with the status “Active.”

The next step is to extract data from PMM. We will use the Federation mechanism for this.

I will experiment with *_up metrics that usually reflect service status in Prometheus exporters.

Info: While using VictoriaMetrics inside PMM, we expose usual Prometheus endpoints to simplify integrations. 

So the federated URL for exposing “*_up” metrics will be https://PMM_SERVER/prometheus/federate?match[]={__name__=~%22.*_up%22}

I can test in on my PMM while being logged into it.

The next step I need to do is to feed the Datadog agent with data from PMM.
I’ll use the OpenMetrics integration available in Datadog.

To do this, I must create a file /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml from a conf.yaml.example located in the same folder. And modify several lines in the file:

To force the Datadog agent on this node to read data from PMM URL:

To get all pmm metrics in the same and conveniently separated namespace.

The most “hard to find and understand” option is:

To force the Datadog agent to collect and store data. (some strange behavior if the type is not defined).

Time to verify the integration.

First, I need to restart the Datadog agent to get a new collection applied.

And if it’s ok – recheck the status by command: 

In the output, I’m searing for the section: 

 A non-zero number for samples confirms the data was properly collected. 

Now it’s time to go into Datadog UI and verify data there. 

Metrics Summary is the best place to check this and search for PMM-related metrics.

By clicking on pmm.mysql_up metric, I can confirm that data are collected for all three MySQL servers added to the PMM.

Dashboarding data in Datadog

The main Idea of having data in Datadog is to create dashboards based on this data.  While I’ve confirmed the presence of metrics – I can go straight to the next step – creating the dashboard by clicking the “Save to Dashboard” button. After following the modal window question, I can create a new dashboard from this metric. By editing the formula to add another metric (pg_up), changing the style to the area, and coloring different metrics in different palettes (so I can easily see what type of services is absent), I can get a better dashboard.

Now I can see all five databases I have in PMM presented in the Datadog dashboard. 

Conclusion

You now know how to pass data to Datadog from PMM and do a simple dashboard. This post (I hope) will help you to play with PMM and integrate it with other tools. If there are better ways to integrate tools than this approach – please, let us know how you do this or your challenges. I will be happy to listen to you on our forum. We are looking for your feedback.

Percona Monitoring and Management is a best-of-breed open source database monitoring solution. It helps you reduce complexity, optimize performance, and improve the security of your business-critical database environments, no matter where they are located or deployed.

Download Percona Monitoring and Management Today

Subscribe

Connect with

Login

Label

0 Comments