What’s New in Elastic Stack 6.5

• Home
• Blog
• ELK Stack
• What’s New in Elastic Stack 6.5

Elastic Stack 6.5 is out!

Every new version of the Elastic Stack is packed with new features and updates, and as always, I’m happy to dive a bit deeper into the new release to provide our readers with a wrap up of what’s new. 

Interestingly enough, and as reflected in the announcements surrounding this release, this release is all about Kibana. That’s not to say the other components in the stack were left out – to the contrary, and I will cover them all, don’t you worry. But Kibana definitely takes the limelight with some exciting new changes and additions in the UI.

Also worthy of note is that a large number of the new features are either experimental or in beta mode. Some require a paid subscription, others are open source or under X-Pack’s basic license. I tried mentioning the relevant licensing for each feature but the official docs are somewhat confusing. I recommend checking out the official release notes for each of the stack’s components before upgrading.

Elasticsearch

As deserved by being the heart of the stack, I will start with Elasticsearch. This is despite the fact that, as mentioned already, Kibana includes the most meaningful changes.

Elasticsearch 6.5 is based on Apache Lucene 7.5 and includes support for Java (JDK) 11 and G1 Garbage Collector ( G1GC) . Below is an overview of the other major changes in this version.

Reduced snapshots

If you rely on Elasticsearch Snapshots for backing up your data you’ll be happy to hear about a new feature that promises to reduce the disk space used by snapshots by 50%!

This new feature allows users to take a “source-only” snapshot that contains only the _source and index metadata. You have all the necessary data required to restore and reindex the data if necessary, and you save on disk space. The one catch is the time to restore — which will be longer and if you need the data to be searchable, it will require a full reindex.

SQL

A lot of Elasticsearch users were excited to hear about the new SQL capabilities announced in Elasticsearch 6.3. The ability to execute SQL queries (X-Pack Basic) on data indexed in Elasticsearch had been on the wishlist of many users, and in version 6.5 additional SQL functions are supported, such as ROUND, TRUNCATE, IN, CONVERT, CONCAT, LEFT, RIGHT, REPEAT, POSITION, LOCATE, REPLACE and INSERT. The ability to query across indices has also been added.

Complementing the existing JDBC driver, Elasticsearch 6.5 now ships with a new ODBC driver which allows further integrability with 3rd party applications. Note, this is an alpha release, and only available as an MSI for Windows.

Cross-cluster replication

Following a number of failover features in recent releases, one can safely claim that Elasticsearch is much more fault-tolerant than what it used to be. In version 6.5, Elasticsearch now offers cross-cluster data replication for replicating data across multiple datacenters. This feature follows the steps of other minor updates to Elasticsearch, specifically soft deletes and sequence numbers, and gives users a much easier way to load data into multiple clusters across datacenters. Keep in mind that this feature is in beta mode and only available for paid subscriptions.

Security

Elasticsearch now supports structured audit logs. This sounds like a given, but the fact is that until this version, Elasticsearch audit logs were not formatted in a particularly friendly fashion. The new audit logs (Linux: /etc/var/log/elasticsearch/elasticsearch_audit.log ) are structured as JSON messages, with ordered attributes.

Another security enhancement is support for authorizations realms, enabling an authentication realm to delegate authorization (lookup and assignment of roles) to other realms. This is another feature requiring a paid subscription.

Kibana

Those who are acquainted with Kibana will agree with me that three brand new pages (or apps) is unprecedented, especially in a minor release. Kibana 6.5 ships with new Infrastructure, Logs and Canvas pages, as well as other smaller updates.

Infrastructure page

To those of you using the stack for monitoring your infrastructure, this one promises to be a biggie. Reminding me a lot of other ITIM tools in the market, this new page (X-Pack Basic) in Kibana offers users an easier way to gain visibility into the different components constructing their infrastructure.

Users can select an element and drill further to view not only metrics but also relevant log data. This feature is still in beta, and only supports server, Docker containers and Kubernetes.

Logs page

If you’re a Logz.io user, you might have heard of Live Tail — the ability to see a live feed of your data coming into the system from all your data sources. Instead of ssh’ing into a machine and using tail -f for tailing specific log files, you can see all your data streaming in from across your system, in real time.

In Kibana 6.5, a new “Logs” page (X-Pack Basic) offers similar capabilities.  The main caveat here is that only “logging indices” (e.g. logstash-*, filebeat-*, etc.) can be used.

Canvas

Canvas is Adobe Photoshop for the world of machine data analytics. I had the pleasure of covering the technology preview here , and am amazed to see how much this project has progressed (love the easy way to add new elements). Canvas takes a while to load, but once it does…

A picture is worth a 1000 words, and you can read the post I linked to above to understand what exactly can be done in Canvas. It’s great to see it baked and pre-packaged into Kibana even as a beta.

Spaces

Another game changing feature is Spaces — the ability to organize your Kibana objects in separate workspaces. Spaces can be created via the UI or using dedicated API. This will be especially useful for those using RBAC (requires a subscription) as they could assign users and roles to the different spaces. Users can create as many spaces as you like and easily switch between them.

Rollup UI

If you use Elasticsearch Rollups (X-Pack Basic, available since Elasticsearch 6.4) for aggregating historical data, you’ll probably find this new feature a pleasure to work with. Instead of using the API, you can use the new UI to create, start, stop, and remove rollup jobs.

Logstash

The main news with this old horse is the Java execution engine which has advanced into beta mode (was announced as experimental in version 6.3). Replacing the old Ruby execution engine, this puppy promises better performance, reduced memory usage and overall — an entirely faster experience. The Java execution engine does not work out of the box right now, so you need to make a small adjustment in your logstash configuration file. Expected GA – version 7.0. Run, Logstash, run!

Other Logstash news is the GA of an SNMP input plugin (for collecting metrics from network devices over SNMP) and an App Search output plugin for feeding data into Elastic’s App Search service.

Beats

New beats

First off, Functionbeat is a new serverless beat that can be deployed on AWS Lambda to move logs from AWS CloudWatch to an Elasticsearch instance of your choice. For triggering the function, you can use either CloudWatch logs or SQS events. This beat is in beta mode.

Second, Journalbeat is designed for handling the logs collected by journald on Linux distros. This beat is planned to become an input in Filebeat and so is defined as being in experimental mode.

Other notable news in the realm of new beats/modules is Heartbeart (for periodic pings on the status of services ) going GA and a new module in Filebeat to support Suricata data.

Config UI

The major news in the world of Beats is the new UI in Kibana (X-Pack Basic) for managing the configuration of your beats. You need to “enroll” your beat from your terminal using a secure token retrieved from Kibana and use a wizard to apply specific configurations (e.g. input, output, etc.). Enabling the beat itself is still done via the terminal of course.

I’m still not convinced how useful this is, especially in highly complex environments with multiple beats deployed. The switching back and forth between Kibana and the terminal is not especially user-friendly, but this is just a beta so I’m pretty sure we’ll see changed applied in version 7.0.

Endnotes

No doubt, this is a lot of news to digest — and I did not even cover the APM component.

Elastic and the community are doing an amazing job in supporting the development of the stack to support the changes we’re seeing in the industry, especially around modern architecture components such as Kubernetes and Docker. Logging and monitoring continue to be a challenge for even the most skilled engineer.

As always, and especially with minor versions, be careful before upgrading. As I specified above, a large amount of the features listed are either in beta or experimental mode. Keep that in mind before upgrading. Read the breaking changes and release notes carefully.

Enjoy!