Application Performance Monitoring with Elastic APM

• Home
• Blog
• ELK Stack
• Application Performance Monitoring with Elastic APM

As its name implies, Elastic APM is an application performance monitoring system which is built on top of the Elastic Stack (Elasticsearch, Logstash, Kibana, Beats). Similar to other APM solutions that you may have heard of, Elastic APM allows you to track key performance-related information such as requests, responses, database transactions, errors, etc.

The Elastic APM solution is comprised of 4 main building blocks — Elasticsearch for data storage and indexing, Kibana for analyzing and visualizing the data, and two APM-specific components — the APM server and the APM agent.

APM agents are responsible for collecting the performance data and sending it to the APM server. The different agents are instrumented as a library in our applications. The APM server is responsible for receiving the data, creating documents from it and sending the data forth into Elasticsearch for storage.

In this tutorial, I will describe how to set up Elastic APM for a basic node.js app on a single Ubuntu 16.04 machine on AWS. We’ll take a look at getting the data pipeline up and running, some of the visualizations features and for Logz.io users – how to ship the APM data into your accounts.

Installing Elasticsearch

As explained above, Elasticsearch and Kibana are two key building blocks in this solution. Just in case you don’t have them set up already, here are instructions for installing version 6.4 of these components.

Elasticsearch requires Java 1.8, so that’s our first step:

sudo apt-get update
 
sudo apt-get install default-jre

Next, we need to add Elastic’s signing key so that our packages can be verified:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key 
add -

For installing on Debian, we need to install the apt-transport-https package:

sudo apt-get install apt-transport-https

The next step is to add the repository definition to our system:

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo 
tee -a /etc/apt/sources.list.d/elastic-6.x.list

All that’s left to do is to update our repositories and install Elasticsearch:

sudo apt-get update
sudo apt-get install elasticsearch

To start Elasticsearch we will use:

sudo service elasticsearch start

To confirm that everything is working as expected, we will point curl or our browser to http://localhost:9200 to get  the following output:

{
  "name" : "L6Wqjy-",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "OgJByBkaQAK_j74YATW6GQ",
  "version" : {
    "number" : "6.4.3",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "fe40335",
    "build_date" : "2018-10-30T23:17:19.084789Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Installing Kibana

As with Elasticsearch, we will use a simple apt command to install Kibana:

sudo apt-get install kibana

Next, we will start Kibana with:

sudo service kibana start

Opening up Kibana in our browser with: http://localhost:5601 , we will be presented with the Kibana homepage:

Installing the APM Server

As explained above, the APM server collects the performance data tracked by the agents and forwards it to Elasticsearch.

To install the APM server using apt, we will use:

sudo apt-get install apm-server

And, to start the server:

sudo service apm-server start

We can query Elasticsearch to make sure a new apm-server index is created:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
 
 
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana                         7CdGpSpuTDqSpzDGfV5XmQ   1   0         17            1     58.8kb         58.8kb
yellow open   apm-6.4.3-onboarding-2018.11.12 4XSxUrWnQryD9yRCwPmCRQ   1   1          1            0      5.9kb

Installing the APM agent and instrumenting our node.js app

Our next step is to create a simple demo app, install the APM agent for node.js and instrument it in our app’s index.js file.

First, let’s create the application folder and create a package.json file for tracking dependencies.

sudo mkdir /demoapp
cd /demoapp
npm init

This last command will result in a series of questions and we can just press enter to use the default settings.

To install express and add it to our package.json, we’ll use:

npm install express --save-dev

While we’re at it, let’s also install the APM agent:

npm install elastic-apm-node --save

Next, we will create the app’s main file which we call index.js (sometimes this file is called server.js or app.js ):

sudo vim index.js

We need to start the APM agent before requiring any other modules in our node.js app, so at the very top of our new file we’ll insert the following snippet:

var apm = require('elastic-apm-node').start({
  serviceName: 'demoapp',
  serverUrl: 'http://localhost:8200'
})

We then paste some basic ‘hello world’ express code:

const express = require('express')
const app = express()
app.get('/', (req, res) => {
 res.send('HEY!')
})
app.listen(3000, () => console.log('Server running on port 3000'))

To start the server, we will use:

node index.js

The output we get shows our node server running as expected:

Server running on port 3000

To simulate some transactions, we can browse to port 3000 on the server we installed the app:

Analyzing in Kibana

So, we’ve installed all the building blocks of our APM solution. It’s time for analyzing the data and visualizing it in Kibana.

Querying Elasticsearch to list indices, we can see that two apm-* indices have been created:

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
 
health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   apm-6.4.3-transaction-2018.11.13 ikIjni6qSDmgix2K9zC8vQ   1   1          5            0       37kb           37kb
yellow open   apm-6.4.3-onboarding-2018.11.13  -boQQic_S2ePPPtJ3HP83w   1   1          1            0      5.9kb          5.9kb
green  open   .kibana                          m4IxzDFrT9CkiB8EqpD3Kw   1   0         17            1     49.3kb         49.3kb

There are two ways of loading these indices in Kibana — one is via the Index Patterns tab on the Management page in Kibana.

The other way is using the APM page where clicking on Setup Instructions displays a tutorial detailing some of the instructions provided above. This option is a good way to verify that the APM server and agent are working as expected.

At the very bottom of the page there is an option to Load Kibana objects . Using this option will load the index patterns, visualizations, and dashboards.

Once you’ve defined the index patterns, you will see your transactions data as well as exceptions (if you have any) in the discover tab.

On the APM page, you will see the ‘demoapp’ service we defined in the index.js file listed under Services.

Clicking it open up a dashboard displaying details on requests made to the app (response times and requests per minute).

You can click on a request to get more details on the request, the response, as well as system and user related info.

Of course, you can use Kibana’s visualization features to build your own detailed dashboard or use the default dashboards provided in Kibana.

Shipping APM data into Logz.io

For Logz.io users, shipping the performance data above into a Logz.io account is easy and requires adjusting the APM server configuration file at /etc/apm-server/apm-server.yml .

First, we’re going to download the required SSL certificate:

wget 
https://raw.githubusercontent.com/logzio/public-certificates/master/COMODORSADomain
ValidationSecureServerCA.crt

We will then copy it into the correct location:

sudo mkdir -p /etc/pki/tls/certs
sudo cp COMODORSADomainValidationSecureServerCA.crt 
/etc/pki/tls/certs/

We can now edit the APM server configuration file:

sudo vim /etc/apm-server/apm-server.yml

We will then add the following snippets (detailed instructions are available within Logz.io, under Log Shipping → Beats ):

fields:
  logzio_codec: json
  token:<yourAccountToken>
fields_under_root: true
 
output.logstash:
  enabled: true
  hosts: ["listener.logz.io:5015"]
  ssl.certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

Be sure to keep to YAML syntax and comment out the default Elasticsearch output. Your account token can be found under Settings → General .

Restarting the APM server, you should begin to see the transactions appear in Logz.io:

Summing it up

I only touched upon the core functionality of Elastic APM with a simple node.js app. Monitoring the performance of a full-fledged application, with multiple services, is an entirely different story of course, but Elastic APM should be considered a good open source alternative to other commercial APM tools in the market.

Those accustomed to working with Elasticsearch, Kibana and Beats will find it especially easy to install and handle. Setup is easy and the default Kibana objects make it easy to get started. To those new to the ELK Stack, however, I would first familiarize myself with some basic concepts before diving into Elastic APM.