Filtering and Enhancing the Exported Data

When your use case requires only a subset of the data exported by Filebeat or you need to add metadata, you can use Filebeat config options to filter the data, or you can define processors.

You can specify configuration options in the filebeat section of the config file to define regular expressions that match the lines you want to include and/or exclude from the output. The supported options are include_lines, exclude_lines, and exclude_files.

For example, you can use the include_lines option to export any lines that start with "ERR" or "WARN":

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/myapp/*.log
  include_lines: ["^ERR", "^WARN"]

The disadvantage of this approach is that you need to implement a configuration option for each filtering criteria that you need.

See Filebeat configuration options for more information about each option.

You can define processors in your configuration to process events before they are sent to the configured output. The libbeat library provides processors for reducing the number of exported fields, and processors for enhancing events with additional metadata. Each processor receives an event, applies a defined action to the event, and returns the event. If you define a list of processors, they are executed in the order they are defined in the configuration file.

event -> processor 1 -> event1 -> processor 2 -> event2 ...

The processors are defined in the Filebeat configuration file.

For example, the following configuration drops all the DEBUG messages.

processors:
 - drop_event:
     when:
        regexp:
           message: "^DBG:"

To drop all the log messages coming from a certain log file:

processors:
 - drop_event:
     when:
        contains:
           source: "test"

See Processors Configuration for more information.