Logstash 7.0.0 released
The new major version of Logstash is now available to download here! This 7.0.0 milestone brings a better out of the box experience to our users through:
Refer to the Logstash breaking changes and release notes for a list of bug fixes and other changes.
Default Java pipeline execution
Logstash was created by Jordan Sissel some time ago as a tool — written in Ruby — to get data from point A to point B. Logstash's accessible text DSL and simple three stage — input/filter/output — execution model proved to be a friendly way for users to move data from any source to any destination.
Originally, Logstash allowed the users to decide which of the Ruby implementations to use: CRuby, JRuby, Rubinius or any other. As the project evolved, JRuby became the de facto runtime due to its performance characteristics and its ability to leverage a vast set of mature and diverse Java libraries.
JRuby is a marvelous piece of technology in its ability to closely implement the Ruby Spec while still squeezing a lot of performance from the JVM. But as we leaned more and more on JRuby, we also identified some concerns. Notably, growing use of Java code in Logstash meant a computational and memory overhead when crossing the barrier between the two languages. This and the increasing number of Java based plugins led us into a path of moving pieces of Logstash's internals to Java. The rewrite has opened up the possibility for Java based plugins to interact directly with a Java based pipeline while also providing better performance and lower memory footprint.
But that's not all! The original Ruby execution engine reads the Logstash config files using Treetop to parse and convert them to executable Ruby source code which is then interpreted and executed at runtime by the JRuby runtime. The new Java execution engine replaces the Ruby source code generation phase with compilation directly to JVM bytecode using Janino. Some execution optimizations can be applied at this stage and the startup and reload times for some large configurations have been reduced by as much as 20x.
Index lifecycle management
Version 6.6.0 the Elastic stack introduced ILM to give users a more flexible control over management actions on your indices, based on criteria such as shard size and performance requirements. A policy is added to the index template in order to drive these actions, which can be changed per index after it is created. In version 6.7.0 Logstash's Elasticsearch output plugin offered a capability — disabled by default — to deploy an ILM policy in the same way it has been doing for years regarding index templates.
Now in version 7.0.0, this Elasticsearch output plugin will automatically detect if the Elasticsearch cluster supports ILM and use deploy the ILM policy if it does. This behaviour can be changed by changing your `elasticsearch { ilm_enabled => auto }` to `true` or `false`.
Plugins, plugins, plugins
A major release is a rare opportunity to introduce desirable breaking changes that have been held off. To this effect we have deprecated settings, made deprecated settings obsolete, and removed existing obsolete settings in many of our plugins. For a complete list please check the breaking changes page.
Wrapping up
We hope this set of improvements in Logstash 7.0.0 is useful to you. Give it a try by downloading Logstash here, and please reach out on our Discuss forum if you have any questions or feedback. Cheers!