Migrate your configuration

edit

Although Filebeat is based on Logstash Forwarder, Filebeat uses YAML for its configuration file, rather than the JSON+comments language used by Logstash Forwarder. This means that you will need to migrate your existing configuration files to use the YAML syntax. Filebeat has a main configuration file called filebeat.yml, but Filebeat also accepts reading multiple configuration files from a conf.d directory and has similar restrictions to Logstash Forwarder. If you specify additional config files, you need to place them in a directory other than the directory where the main Filebeat config file resides. You specify the location of the config files by using the config_dir option to configure the path to the directory. In most cases, you can do a one-to-one conversion to create a Filebeat config file for each Logstash Forwarder config file.

Before migrating your config files, we recommend that you first read the Configuring Filebeat section to understand the Filebeat options.

Migrate the "files" section

edit

To migrate the files section from the Logstash Forwarder configuration, create an inputs section in the Filebeat config file. For example, assuming that you start with this configuration in Logstash Forwarder:

  # The list of files configurations
  "files": [
    # An array of hashes. Each hash tells what paths to watch and
    # what fields to annotate on events from those paths.
    {
      "paths": [
        "/var/log/messages",
        "/var/log/*.log"
      ],

      # A dictionary of fields to annotate on each event.
      "fields": {
        "type": "syslog",
        "service": "apache",
        "zone": "us-east-1"
      }
    }, {
      # A path of "-" means stdin.
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }, {
      "paths": [
        "/var/log/apache/httpd-*.log"
      ],
      "fields": { "type": "apache" }
    }
  ]

The equivalent inputs section would look like this:

filebeat.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
  fields:
    service: apache
    zone: us-east-1
  fields_under_root: true
- type: stdin 
- type: log
  paths:
    - /var/log/apache2/httpd-*.log

The explicit type option was introduced to differentiate between normal files and stdin. In the future, additional types might be supported.

As you can see, apart from the new type options, which were before implicitly defined via the type custom field, the remaining options can be migrated mechanically.

The Filebeat configuration gives you more control over how each input behaves by allowing you to configure options that were previously global in Logstash Forwarder and set them separately for each input. See Configuring Filebeat.

Migrate the "network" section

edit

Like Logstash Forwarder, Filebeat can communicate directly with Logstash. Filebeat can also insert log entries directly into Elasticsearch. This results in an output section that is a bit more complex, as you can see in the following example. You’ll find, however, that you can easily translate the Logstash part of the configuration from the equivalent Logstash Forwarder configuration.

The following snippet shows the network section of the Logstash Forwarder configuration:

  # The network section covers network configuration :)
  "network": {
    # A list of downstream servers listening for our messages.
    # logstash-forwarder will pick one at random and only switch if
    # the selected one appears to be dead or unresponsive
    "servers": [ "localhost:5043" ],

    # The path to your client ssl certificate (optional)
    "ssl certificate": "./logstash-forwarder.crt",
    # The path to your client ssl key (optional)
    "ssl key": "./logstash-forwarder.key",

    # The path to your trusted ssl CA file. This is used
    # to authenticate your downstream server.
    "ssl ca": "./logstash-forwarder.crt",

    # Network timeout in seconds. This is most important for
    # logstash-forwarder determining whether to stop waiting for an
    # acknowledgement from the downstream server. If an timeout is reached,
    # logstash-forwarder will assume the connection or server is bad and
    # will connect to a server chosen at random from the servers list.
    "timeout": 15
  }

The equivalent in Filebeat would look like this:

output.logstash:
  hosts: 
    - localhost:5043
  timeout: 15
  ssl.certificate_authorities: 
    - ./logstash-forwarder.crt
  ssl.certificate: ./logstash-forwarder.crt
  ssl.key: ./logstash-forwarder.key

When multiple hosts are defined, the default behavior in Filebeat is to pick a random host for new connections, similar to the Logstash Forwarder behavior. Filebeat can optionally do load balancing. For more details, see the loadbalance configuration option.

Note that if the ssl settings are missing, then SSL is disabled. SSL is automatically enabled when you add any of the ssl options. For more information about specific configuration options, see Specify SSL settings.

Changed configuration file options

edit

With the refactoring of the configuration file, the following options were removed or renamed:

Config Option Action

deadTime

deadTime was renamed to ignore_older. Filebeat keeps the files that it’s reading open until they are older than the timespan specified by ignore_older. If a file is changed, Filebeat reopens it.

netTimeout

netTimeout was removed and is replaced by the timeout option in libbeat.

log-to-syslog and syslog

Both options were removed and replaced by logging options in libbeat.

For more information about these options, see Configuring Filebeat.

A complete example

edit

Let’s see a simple, but complete example of a Logstash Forwarder configuration and its equivalent for Filebeat.

Logstash Forwarder configuration:

{
  "files": [
    {
      "paths": [
        "/var/log/*.log"
      ],
      "fields": {
        "type": "syslog",
        "service": "test01"
      }
    }
  ],
  "network": {
    "servers": [ "localhost:5043" ],
  }
}

Filebeat configuration:

filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
  fields:
    service: test01
output.elasticsearch:
  hosts: ["http://localhost:5043"]