API Analysis with the ELK Stack

• Home
• Blog
• ELK Stack
• API Analysis with the ELK Stack

Pulling in data exposed via API is not one of the most common use cases for ELK Stack users but it is definitely one I’ve come across in the past. Developers wrapping their database services with REST API, for example, might be interested in analyzing this data for business intelligence purposes.

Whatever the reason, the ELK Stack offers some easy ways to integrate with this API. One of these methods is the Logstash HTTP poller input plugin. In the example below, I’ll be pulling in some weather data exposed via public API using this plugin.

To follow these steps and build your own pipeline for HTTP API analysis in ELK you’ll need to set up the following:

• Your own ELK Stack or a Logz.io account. To install ELK, take a look at our ELK guide . To create a Logz.io account, simply go to https://logz.io/freetrial and sign up for free.
• Access to a weather API service. For this article, I’m using OpenWeatherMap , which provides free access to current weather stats via API but you can hook up to any API using the same methodology.

The dataset

As mentioned above, I’m collecting current weather data using OpenWeatherMap API. The dataset includes statistics on existing weather conditions all over the world collected from 40,000 weather stations.

The data itself is available in JSON, XML or HTML but the default format is JSON, which suits our use case perfectly because we’re using Elasticsearch to index the data. Calling the data is done in a variety of ways — you can call by providing a city name (the method we’ll use here), city ID, geographic coordinates, or ZIP code. You can also call for multiple cities within specific coordinates.

The example below is calling for weather stats within a bounding box that includes the longtitude and latitude coordinates:

http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10

The data returned by the API provides us with all the stats we need for gauging current weather conditions, including temperature, humidity, atmospheric pressure, wind speed and direction, and more.

Here’s an example:

{"coord":
{"lon":145.77,"lat":-16.92},
"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],
"base":"cmc stations",
"main":{"temp":293.25,"pressure":1019,"humidity":83,"temp_min":289.82,"temp_max":295.37},
"wind":{"speed":5.1,"deg":150},
"clouds":{"all":75},
"rain":{"3h":3},
"dt":1435658272,
"sys":{"type":1,"id":8166,"message":0.0166,"country":"AU","sunrise":1435610796,"sunset":1435650870},
"id":2172797,
"name":"Cairns",
"Cod":200}

Configuring Logstash

The method for importing data returned via API into the ELK Stack is by using the Logstash http_poller input plugin , which can aggregate, decode and ship data from a defined endpoint URL.  

If you haven’t installed Logstash yet as prescribed above, do so now. Then, create a new Logstash pipeline configuration file:

sudo vim /etc/logstash/conf.d/weather.conf

Logstash input

The input section defines the http_poller input plugin — the URL endpoint to poll, a request timeout, a CRON-based schedule (every 5 minutes) and the codec to use (JSON). The metadata_target setting is optional, and adds some fields to the responses on the performance of the poller.

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

The API we’re inserting here contains the following query parameters after the call URL:

• q=London,uk – asks to return weather data for London
• APPID= – OpenWeatherMap API key
• units=metrics – converting the units format to celsius

http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7
341764f682c2242e744c4f167b0&units=metric

Logstash filter

Since the API responses are in JSON, there is little processing or parsing that is required from Logstash. And so we can leave the filter section empty for now.

filter {}

Logstash output

I think this last section is pretty self-explanatory, right?

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}

Starting Logstash, you will see a new Elasticsearch index created with the weather data. Define the new index pattern in Kibana to begin analysis:

Shipping to Logz.io

To ship the data to Logz.io, a few adjustments need to be made to the Logstash configuration file. A mutate filter containing the Logz.io account token needs to be added, and we need to change the output to point to Logz.io listeners instead of Elasticsearch.

The amended configuration file looks like this (be sure to enter your token in the designated position:

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/find?lat=42.3&lon=-71.05&cnt=20&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}
 
filter {
  mutate {
    add_field => { "token" => "logzioAccountToken" }
  }
}
 
output {
   tcp {
    host => "listener.logz.io"
    port => 5050
    codec => json_lines
  }
}

Restart Logstash.

You should see the weather data load in Logz.io automatically within a minute or two:

Forget your favorite weather app, hook up your ELK with a weather API and use the dashboard below to stay up to date!