What’s New in Elastic Stack 7.3

• Home
• Blog
• ELK Stack
• What’s New in Elastic Stack 7.3

As if the temperature this summer was not high enough, this new major release of the Elastic Stack promises turns it up a notch with some hot new features. Bundling new ETL capabilities in Elasticsearch, a bunch of improvements in Kibana and a lot of new integration goodness in Filebeat and Metricbeat, Elastic Stack 7.3 is worth 5 minutes of your time to stay up to date.

Elasticsearch

As the heart of the stack, and per usual, I’m going to start with Elasticsearch. There are a lot of new enhancements and improvements on top of existing functionality so I tried to focus on new stuff.

Dataframes

What is probably the biggest Elasticsearch news in this 7.3  release, Dataframes is a new way to summarize and aggregate data in a more analysis-friendly and resource-efficient way. 

Using what is called a “transform” operation, users, in essence, transform an Elasticsearch index into a different format by first defining a “pivot” — a set of definitions instructing Elasticsearch how to summarize the data. The pivot is defined by first selecting one or more fields used to group your data and then the aggregation type (not all aggregation types are currently supported). The result, as said, is the data frame — a summary of your original time series data stored in another index. Transforms can run once or continuously. 

Data Frames is a beta feature which is licensed under the basic license.

New voting-only node type

A new “voting-only master-eligible” node type has been developed. Despite what is implied by the name, this node cannot actually act as a master in the cluster. What it can do is vote when electing a master and this can be useful as a tie-breaker. Because of this, is also takes up less resources and can run on a smaller machine.

Voting-only master-eligible nodes are licensed under the basic license. 

Flattened object type

Another interesting piece of Elasticsearch news is the support for flattened object types. 

Up until now, objects with a large number of fields had to be indexed into separate fields. This, of course, made mapping much more complicated and could potentially also affect the performance of the cluster. 

The new flattened type maps the entire object into a single field, indexing all subfields into one field as keywords (which can then be more easily queried and visualized). For now, only basic searches and aggregations can be used. 

The flattened object type is licensed under the basic license. 

Search improvements

The most important development in Elasticsearch search is a new aggregation type called rare_terms . This aggregation was developed to help identify terms with low document counts, an aggregation that promises to aid security-related searches that often focus on those least occurring events. 

Outlier detection

As the name of this feature implies, Outlier detection helps you identify outliers — data points with different values from those of normal data points. The way this is done is by analyzing the numerical fields for each document and annotating their “unusualness” in an outlier score which can be used for analysis and visualization. 

Outlier detection promises to be of use for both operational and security use cases, helping users detect security threats as well as unusual system performance, and is licensed under the basic license.

Logstash

This old horse is still the cornerstone of many data pipelines, despite the advent of alternative aggregators and enhancements made to Filebeat. Version 7.3 includes two interesting news items — improvements to pipeline-to-pipeline communication and better JMS support. 

Pipeline-to-pipeline communication 

The use case for this feature, as its name implies, is to enable users to connect between different processing pipelines on the same Logstash instance. By doing so, users can break up complicated pipelines into more modular units which can help boost performance and also allows more modular handling of the processing. 

Elastic has taken care of all the outstanding issues in this feature and is now encouraging users to give it a try. Pipeline-to-pipeline communication is still in beta.

JMS input

Logstash 7.3 now bundles the JMS input plugin by default. This plugin, used for ingesting data from JMS deployments, was greatly improved in the previous release of the stack, with the introduction of failover mechanisms, better performance, TLS and more. This article explains how to use this plugin to allow Logstash to act as a queue or topic consumer.

Kibana

Kibana 7.0 was such a huge leap in terms of the changes applied compared to previous versions that one can hardly expect changes of the same order of magnitude to be introduced in each major release. Still, Kibana 7.3 has some interesting new developments worth pointing out. 

Maps goes GA

I have previously mentioned Maps but now that this feature is fully available, I think it this is a great opportunity to dive deeper into this feature. Most Kibana users are familiar with the Coordinate Map and Region Map visualizations that can be used to geographically visualize data. Maps takes geographic visualization to an entirely new level, allowing users to add multiple layers on top of the map to visualize additional geospatial elements.

In this 7.3 release, other than going GA, Maps adds new customization options for layers, new ways to import geospatial data, top hits aggregation and enhanced tooltips. 

Maps is licensed under Elastic’s basic license. 

Logs

Kibana’s live tailing page, Logs, now has the ability to highlight specific details in the logs and also includes integration with Elastic APM, allowing users to move automatically from a log message to a trace and thus remain within the context of an event. Logs and APM are licensed under the basic license. 

Misc. usability enhancements

Kibana 7.3 adds a long list of minor but important usability improvements that are worth noting such as the ability to delete and restore Elasticsearch snapshots in the Snapshot and Restore management UI (basic license), export a saved search on a dashboard directly to CSV (basic license), show values directly inside bar charts, and use KQL and auto-complete in filter aggregations. 

Kerberos support

Other big Kibana news in 7.3 is support for a new SSO authentication type – Kerberos. Of course, Kibana already supports other SSO methods, namely SAML and OpenID Connect, all available for Platinum subscribers only and apparently not available for cloud offerings yet.

Beats

Beats have come a long way since first being introduced. Specifically, a lot of functionality has been added to the top two beats in the family — Filebeat and Metricbeat, to support better integration with popular data sources and version 7.3 continues this development line.

Enhanced Kubernetes monitoring

Kubernetes users using the Elastic Stack to monitor their clusters will be thrilled to hear that Metricbeat now includes new metricsets to monitor kube-controller-manager, kube-proxy and kube-scheduler. 

Automating Functionbeat

AWS users can now use a CloudFormation template for deploying Functionbeat. This ability promises to help automate data collection and shipping of data from AWS services instead of manually spawning up Functionbeat. Functionebeat is licensed under the Basic license.

Shipping from Google Cloud 

It appears like the new AWS module in Metricbeat was just the beginning of new integrations between cloud services and the stack. Filebeat now allows ingesting data from Google Cloud using a new Google Pub/Sub input and also support a new module for shipping Google VPC flow logs. These features are in beta and licensed under the basic license

Database support

A series of new features have been added to Filebeat and Metricbeat to better support monitoring specific databases, including Oracle, Amazon RDS, CockroachDB and Microsoft SQL.

Some endnotes

So, as usual, a lot of goodness is yet another feature-packed release. 

Interestingly, the vast majority of the new Elasticsearch, Logstash, Kibana and Beats code is under Elastic’s basic license and is highlighted as such in the respective release notes. This adds some clarity into licensing and usage limitations. I made sure to mention these features and also any feature in beta, but before looking into upgrading be sure to verify these conditions as well as breaking changes.

Enjoy!

Love ELK but hate maintaining it? Try Logz.io's hosted ELK solution.

Request a Demo