Logstash breaking changes

We’ve removed deprecated SSL settings in some Logstash plugins, and have replaced them with updated settings. If your plugin configuration contains any of these obsolete options, the plugin may fail to start.

`logstash-input-beats`

Setting	Replaced by
cipher_suites	ssl_cipher_suites
ssl	ssl_enabled
ssl_peer_metadata	ssl_peer_metadata option of enrich
ssl_verify_mode	ssl_client_authentication
tls_min_version	ssl_supported_protocols
tls_max_version	ssl_supported_protocols

`logstash-input-elastic_agent`

Setting	Replaced by
cipher_suites	ssl_cipher_suites
ssl	ssl_enabled
ssl_peer_metadata	ssl_peer_metadata option of enrich
ssl_verify_mode	ssl_client_authentication
tls_min_version	ssl_supported_protocols
tls_max_version	ssl_supported_protocols

`logstash-input-elasticsearch`

Setting	Replaced by
ca_file	ssl_certificate_authorities
ssl	ssl_enabled
ssl_certificate_verification	ssl_verification_mode

`logstash-input-elastic_serverless_forwarder`

Setting	Replaced by
ssl	ssl_enabled

`logstash-input-http`

Setting	Replaced by
cipher_suites	ssl_cipher_suites
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
ssl	ssl_enabled
ssl_verify_mode	ssl_client_authentication
tls_max_version	ssl_supported_protocols
tls_min_version	ssl_supported_protocols
verify_mode	ssl_client_authentication

`logstash-input-http_poller`

Setting	Replaced by
cacert	ssl_certificate_authorities
client_cert	ssl_certificate
client_key	ssl_key
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
keystore_type	ssl_keystore_password
truststore	ssl_truststore_path
truststore_password	ssl_truststore_password
truststore_type	ssl_truststore_type

`logstash-input-tcp`

Setting	Replaced by
ssl_cert	ssl_certificate
ssl_enable	ssl_enabled
ssl_verify	ssl_client_authentication in server mode and ssl_verification_mode in client mode

`logstash-filter-elasticsearch`

Setting	Replaced by
ca_file	ssl_certificate_authorities
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
ssl	ssl_enabled

`logstash-filter-http`

Setting	Replaced by
cacert	ssl_certificate_authorities
client_cert	ssl_certificate
client_key	ssl_key
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
keystore_type	ssl_keystore_type
truststore	ssl_truststore_path
truststore_password	ssl_truststore_password
truststore_type	ssl_truststore_type

`logstash-output-elasticsearch`

Setting	Replaced by
cacert	ssl_certificate_authorities
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
ssl	ssl_enabled
ssl_certificate_verification	ssl_verification_mode
truststore	ssl_truststore_path
truststore_password	ssl_truststore_password

`logstash-output-http`

Setting	Replaced by
cacert	ssl_certificate_authorities
client_cert	ssl_certificate
client_key	ssl_key
keystore	ssl_keystore_path
keystore_password	ssl_keystore_password
keystore_type	ssl_keystore_password
truststore	ssl_truststore_path
truststore_password	ssl_truststore_password
truststore_type	ssl_truststore_type

`logstash-output-tcp`

Setting	Replaced by
ssl_cacert	ssl_certificate_authorities
ssl_cert	ssl_certificate
ssl_enable	ssl_enabled
ssl_verify	ssl_client_authentication in server mode and ssl_verification_mode in client mode

We've improved memory configuration for certain Logstash plugins. Input plugins such as elastic_agent, beats, tcp, and http allocate buffers in Java memory to read events from the network. The default allocation method is direct memory rather than heap memory to simplify configuration, and to help facilitate debugging memory usage problems through the analysis of heap dumps. If you need to re-enable the previous behavior, change the pipeline.buffer.type setting in logstash.yml. Check out off-heap-buffers-allocation for details. #16500

We have removed the Logstash modules framework, and encourage users to try Elastic Integrations This includes the netflow, azure and arcsight modules, and the modules framework as a whole. #16794

We have removed support for previously deprecated configuration settings:

• http.* prefixed settings for the Logstash API. Settings prefixed by http.* have been replaced by the equivalent settings prefixed with api.*. #16552

• event_api.tags.illegal Any events that include field named tags automatically rename the field _tags to avoid any clash with the reserved Logstash tags field. Instead, Logstash generates _tagsparsefailure in the event tags and the illegal value is written to the _tags field. #16461

The ingest converter, which was used to convert ingest pipelines to Logstash pipelines, has been removed. #16453

The logstash-filter-elastic_integration plugin offers similar functionality, and can help you use Logstash to extend Elastic integrations.

JDK17 is the minimum version of the JDK required to run Logstash. For the best experience, we still recommend running Logstash using the bundled-jdk. See Logstash JVM requirements for details. #16443

The base image for Logstash on Docker has been changed from Ubuntu to UBI9. If you create a Docker image based on the Logstash image and rely on it being Ubuntu based, you need to change your derived image to take account of this change. For example, if your derived docker image installs additional packages using a package manager, UBI9 uses microdnf, rather than apt. #16599

We've changed the default behavior to prevent users from accidentally running Logstash as a superuser. If you try to run Logstash as a superuser, it logs an error and fails to start, ensuring that users cannot run Logstash with elevated privileges by accident.

You can change the value of the allow_superuser setting to true in logstash.yml if you want to restore the previous behavior and allow Logstash to run with superuser privileges. #16558

To continue using deprecated internal collection to monitor Logstash, set xpack.monitoring.allow_legacy_collection to true in logstash.yml. We encourage you to move to agent-driven monitoring, the latest, supported way to monitor Logstash #16586

We've improved the way we deal with duplicate message fields in json documents. Some code paths that log in json produce log events that include multiple instances of the message field. (The JSON codec plugin is one example.) While this approach produces JSON that is technically valid, many clients do not parse this data correctly, and either crash or discard one of the fields.

We recently introduced the option to fix duplicates, and made it the default behavior for 9.0 and later. To re-enable the previous behavior, change the log.format.json.fix_duplicate_message_fields setting in logstash.yml to false.

Check out Logging in json format can write duplicate message fields for more details about the issue. #16578

We’ve removed the Logstash Enterprise_search integration plugin, and its component App Search and Workplace Search plugins from the default Logstash install. These plugins will receive only security updates and critical fixes moving forward.

We recommend using our native Elasticsearch tools for your Search use cases. For more details, please visit the Search solution and use case documentation.