Logstash output
editLogstash output
editThe Logstash output is currently only supported for Elastic Agents in standalone mode. Fleet-managed agents are not supported.
The Logstash output uses an internal protocol to send events directly to Logstash over TCP. Logstash provides additional parsing, transformation, and routing of data collected by Elastic Agent.
Compatibility: This output works with all compatible versions of Logstash. Refer to the Elastic Support Matrix.
This example configures a Logstash output called default in the
elastic-agent.yml file:
|
The Logstash server and the port ( |
To receive the events in Logstash, you also need to create a Logstash configuration pipeline. The Logstash configuration pipeline listens for incoming Elastic Agent connections, processes received events, and then sends the events to Elasticsearch.
The following example configures a Logstash pipeline that listens on port 5044 for
incoming Elastic Agent connections and routes received events to Elasticsearch:
input {
elastic_agent {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
data_stream => "true"
}
}
For more information about configuring Logstash, refer to Configuring Logstash and Elastic Agent input plugin.
Logstash output configuration settings
editThe logstash output supports the following settings, grouped by category.
Many of these settings have sensible defaults that allow you to run Elastic Agent with
minimal configuration.
Commonly used settings
edit| Setting | Description |
|---|---|
|
(boolean) Enables or disables the output. If set to |
|
|
(boolean) Configures escaping of HTML in strings. Set to Default: |
|
|
(list) The list of known Logstash servers to connect to. If load balancing is disabled, but multiple hosts are configured, one host is selected randomly (there is no precedence). If one host becomes unreachable, another one is selected randomly. All entries in this list can contain a port number. If no port is specified,
|
|
|
(string) The URL of the SOCKS5 proxy to use when connecting to the Logstash
servers. The value must be a URL with a scheme of If the SOCKS5 proxy server requires client authentication, embed a username and password in the URL as shown in the example. When using a proxy, hostnames are resolved on the proxy server instead of on the
client. To change this behavior, set outputs:
default:
type: logstash
hosts: ["remote-host:5044"]
proxy_url: socks5://user:password@socks5-proxy:2233
|
|
|
(boolean) Determines whether Logstash hostnames are resolved locally when using a
proxy. If Default: |
Authentication settings
editSettings for authenticating with Logstash.
To use SSL, you must also configure the Elastic Agent input plugin for Logstash to use SSL/TLS.
There are a number of SSL configuration settings available depending on whether you are configuring the client, server, or both. See the following tables for available settings:
- Table 4, “Common configuration options”. These settings are valid in both client and server configurations.
- Table 5, “Client configuration options”
- Table 6, “Server configuration options”
For more information about using certificates, refer to Encrypt traffic in a self-managed cluster.
Table 4. Common configuration options
| Setting | Description |
|---|---|
|
(string) This configures a certificate pin that you can use to ensure that a specific certificate is part of the verified chain. The pin is a base64 encoded string of the SHA-256 of the certificate. This check is not a replacement for the normal SSL validation, but it adds
additional validation. If this setting is used with |
|
|
(list) The list of cipher suites to use. The first entry has the highest priority. If this option is omitted, the Go crypto library’s default suites are used (recommended). Note that TLS 1.3 cipher suites are not individually configurable in Go, so they are not included in this list. The following cipher suites are available:
Here is a list of acronyms used in defining the cipher suites:
|
|
|
(list) The list of curve types for ECDHE (Elliptic Curve Diffie-Hellman ephemeral key exchange). The following elliptic curve types are available:
|
|
|
(boolean) Enables or disables the SSL configuration. Default: SSL settings are disabled if either |
|
|
(list) List of allowed SSL/TLS versions. If the SSL/TLS server supports none
of the specified versions, the connection will be dropped during or after the
handshake. The list of allowed protocol versions include: Default: |
Table 5. Client configuration options
| Setting | Description |
|---|---|
|
(string) The path to the certificate for SSL client authentication. This
setting is only required if Example: ssl.certificate: "/path/to/cert.pem" When this setting is configured, the Specify a path, or embed a certificate directly in the ssl.certificate: |
-----BEGIN CERTIFICATE-----
CERTIFICATE CONTENT APPEARS HERE
-----END CERTIFICATE-----
|
|
|
(list) The list of root certificates for verifications (required). If
Example: ssl.certificate_authorities: ["/path/to/root/ca.pem"] Specify a list of files that Elastic Agent will read, or embed a certificate directly
in the ssl.certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
CERTIFICATE CONTENT APPEARS HERE
-----END CERTIFICATE-----
|
|
|
(string) The client certificate key used for client authentication. Only
required if Example: ssl.key: "/path/to/cert.key" Specify a path, or embed the private key directly in the ssl.key: |
-----BEGIN PRIVATE KEY-----
KEY CONTENT APPEARS HERE
-----END PRIVATE KEY-----
|
|
|
(string) The passphrase used to decrypt an encrypted key stored in the
configured |
|
|
(string) Controls the verification of server certificates. Valid values are:
Default: |
Table 6. Server configuration options
| Setting | Description |
|---|---|
|
(string) The path to the certificate for SSL server authentication. If the certificate is not specified, startup will fail. Example: ssl.certificate: "/path/to/server/cert.pem" When this setting is configured, the Specify a path, or embed a certificate directly in the ssl.certificate: |
-----BEGIN CERTIFICATE-----
CERTIFICATE CONTENT APPEARS HERE
-----END CERTIFICATE-----
|
|
|
(list) The list of root certificates for client verifications is only required
if Example: ssl.certificate_authorities: ["/path/to/root/ca.pem"] Specify a list of files that Elastic Agent will read, or embed a certificate directly
in the ssl.certificate_authorities:
- |
-----BEGIN CERTIFICATE-----
CERTIFICATE CONTENT APPEARS HERE
-----END CERTIFICATE-----
|
|
|
(string) Configures client authentication. The valid options are:
Default: |
|
|
(string) The server certificate key used for authentication (required). Example: ssl.key: "/path/to/server/cert.key" Specify a path, or embed the private key directly in the ssl.key: |
-----BEGIN PRIVATE KEY-----
KEY CONTENT APPEARS HERE
-----END PRIVATE KEY-----
|
|
|
(string) The passphrase used to decrypt an encrypted key stored in the
configured |
|
|
(string) Configures the type of TLS renegotiation to support. The valid options are:
Default: |
|
|
(string) Controls the verification of client certificates. Valid values are:
Default: |
Performance tuning settings
editSettings that may affect performance.
| Setting | Description |
|---|---|
|
(string) The number of seconds to wait before trying to reconnect to Logstash
after a network error. After waiting Default: |
|
|
(string) The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. Default: |
|
|
(int) The maximum number of events to bulk in a single Logstash request. Events can be collected into batches. Elastic Agent will split batches larger than
Specifying a larger batch size can improve performance by lowering the overhead of sending events. However big batch sizes can also increase processing times, which might result in API errors, killed connections, timed-out publishing requests, and, ultimately, lower throughput. Set this value to Default: |
|
|
(int) The gzip compression level. Set this value to Increasing the compression level reduces network usage but increases CPU usage. Default: |
|
|
If Default: Example: outputs:
default:
type: logstash
hosts: ["localhost:5044", "localhost:5045"]
loadbalance: true
|
|
|
(int) The number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped. Set Default: |
|
|
(int) The number of batches to send asynchronously to Logstash while waiting
for an ACK from Logstash. The output becomes blocking after the specified number of
batches are written. Specify Default: |
|
|
(boolean) If Default: |
|
|
(string) The number of seconds to wait for responses from the Logstash server before timing out. Default: |
|
|
(string) Time to live for a connection to Logstash after which the connection will be reestablished. This setting is useful when Logstash hosts represent load balancers. Because connections to Logstash hosts are sticky, operating behind load balancers can lead to uneven load distribution across instances. Specify a TTL on the connection to achieve equal connection distribution across instances. Default: The |
|
|
(int) The number of workers per configured host publishing events to {output-type}. This is best used with load balancing mode enabled. Example: If you have two hosts and three workers, in total six workers are started (three for each host). Default: |