Show EchoJS: CLI utility for filtering and aggregation of JSONL streams

jsonlash

CLI utility for filtering and aggregation of JSONL streams. No matter which service for logging you use (LogDNA, Papertrail, Loggly, etc.) simply pipe log into jsonlash set up filters and aggregators and see aggregated data in realtime.

Usage

Installation

Install from NPM globally:

npm install -g jsonlash

After installation you can simply run jsonlash from your terminal with -h parameter to display help page:

jsonlash -h

Basic usage with filtering

We currently use Log DNA as logging service so I am going to use it in examples but it's going to work with any JSONL stream. So pipe your log stream to jsonlash:

logdna tail | jsonlash

Now it will simply print out the log as it comes. So lets filter the API logs that are in form:

{
    "msg": "API call",
    "req": {
        "duration": 590,
        "method": "GET",
        "route": "V2.datasets.items",
        ...
    }
    ...
}

Filtering is done using -f [FILTER] parameter:

logdna tail | jsonlash -f 'msg=API call'

We can add more filters to filter out only requests with POST method and duration over 1000ms. And also add parameter -e to expand printed JSONs to be more readable:

logdna tail | jsonlash -f 'msg=API call' -f 'req.method=POST' -f 'req.duration>1000' -e

Aggregations

Lets continue with API logs example. To group log lines by request method and compute average and maximal duration call:

logdna tail | jsonlash -f 'msg=API call' -a req.method --max req.duration --avg req.duration

and output will be a table with data aggregated in realtime:

Examples

1.

Aggregate logs by two fields req.method and req.routeName and compute average duration and maximum duration

... | jsonlash -a req.method -a req.routeName --max req.duration --avg req.duration

2.

Filter out requests taking more than a 10s, grouped them by req.routeName and compute how many users requested each of them:

... | jsonlash -f 'req.duration>10000' -a req.routeName --uni req.userId

Command reference

This is a simple command line tool to filter and aggregate JSONL (json-lines) streams.

USAGE
  $ jsonlash

OPTIONS
  -a, --aggregate=aggregate  aggregate JSONL items
  -d, --debug                debug mode, shows JSON parsing errors
  -e, --expand               expand outputted JSON
  -f, --filter=filter        filter JSONL items
  -h, --help                 show CLI help
  -v, --version              show CLI version
  --avg=avg                  aggregate average value over all occurances of given field
  --max=max                  aggregate maximum value over all occurances of given field
  --min=min                  aggregate minimum value over all occurances of given field
  --sum=sum                  aggregate sum over all occurances of given field
  --uni=uni                  aggregate number of unique occurances of given field

DESCRIPTION
  Simply pipe in any JSONL stream and with filter and/or aggregation flag.