Application Logs: Pitfalls and Insights

 2 years ago
source link: https://blog.knoldus.com/application-logs/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Reading Time: 3 minutes


In this blog article,we aim to give reader a sense of what is the need of monitoring application logs and a methodology of how we can monitor them using an example.

Lets say you have a use case where you would like to monitor important aspects of your application for e.g. temperature” as a measure” or “facet” for multiple devices emitting data for complete IoT system for monitoring home temperatures. You would like to raise an alert in scenarios where abnormal conditions arises. Abnormal conditions here refers to scenarios where spike in temperature occurs beyond specific threshold limit.Hold those thoughts, because we would be using this example in subsequent sections to monitor data emitted by different sensors.

First, let us understand the necessity of monitoring logs.

Why do we need to monitor logs?

1.) For validating the behaviour of your application with the expected behaviour.

2) Proper management of system resources that your application consumes like (CPU, Memory etc.)

3) Keeping track of essential fields inside your application like temperature, no. of files received etc. depending on your application.

4) Gather intelligence and improve you business as well.

QuickStart guide to obtain Metrics from Application Logs
1. Background of Grok Exporter

Grok provides functionality to parse unstructured log data into something structured and queryable. Grok is heavily used in Logstash to provide log data as input for ElasticSearch. It is easy to extend Grok with custom patterns.

The grok_exporter aims at porting Grok from the application metric to Prometheus monitoring. The goal is to use Grok patterns for extracting Prometheus metrics from arbitrary log files.

Note :- You can use any exporter for extracting metrics from your logs. For the scope of this example I have used grok_exporter.

Download it from https://github.com/fstab/grok_exporter/releases.

2.Write Parser Using GROK

Write a Parser in config.yml at ~/grok_exporter-0.2.7.linux-amd64/example/ that reads logs and converts it into metrics.

global: config_version: 2 input: type: file path: temperatureSpike.log readall: true # Read from the beginning of the file? False means we start at the end of the file and read only new lines. grok: patterns_dir: ./patterns/ metrics: – type: gauge name: device_temperature_detail help: Temperature Detail of different devices represented by device ID. match: 'Device %{USERNAME:deviceId} has %{NUMBER:temperature} temperature' value: '{{.temperature}}' labels: system_id: '{{.deviceId}}'

server: host: localhost port: 9144

Now run ./grok_exporter –config ./example/config.yml inside grok directory. You should be able to see metrics at localhost:9144/metrics as depicted in below image:-

Metrics exposed by the Parser

In the screenshot above you can see that my metrics Device-Id and Temperature are exposed at localhost:9144/metrics.

Note : You can write your custom pattern inside patterns directory and use in config.yml.

3. Use Prometheus for Visualizing your custom metrics

Start Prometheus using comand

sudo systemctl start prometheus 

Check its status using below command :-

sudo systemctl status prometheus

Once prometheus is up and running. You should be able to see prometheus UI – localhost:9090/graph. In screenshot attached below, you will be able to see metrics that you created using parser inside prometheus.

Prometheus depicting metrices exported using grok

Refer to link below for more details on prometheus :-



Now visualize your metrics using grafana –

  • Start grafana using below command :-
./bin/grafana-server web.

Create an alerting rule corresponding to abnormal conditions.You should be able to see dashboard in case of any spike in temperature

Refer to this link below for more information on grafana.


Result of Monitoring Logs

Figure 4: DEVICE44 shows the spike here

Above graph clearly depicts spike in temperature for “Device44”. You can set an alerting rule in grafana to get notification on your choice of channels like mail, slack etc.


In this post, we have gone through a way out of how we can trace our application logs and generate alerts as soon as an unexpected condition arises.Stay Tuned for more interesting blogs like this.

About Joyk

Aggregate valuable and interesting links.
Joyk means Joy of geeK