grok

If this filter is successful, add any arbitrary fields to this event. Field names can be dynamic and include parts of the event using the %{field}.

Example:

    filter {
      grok {
        add_field => { "foo_%{somefield}" => "Hello world, from %{host}" }
      }
    }
[source,ruby]
    # You can also add multiple fields at once:
    filter {
      grok {
        add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
        }
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add field foo_hello if it is present, with the value above and the %{host} piece replaced with that value from the event. The second example would also add a hardcoded field.

If this filter is successful, add arbitrary tags to the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      grok {
        add_tag => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also add multiple tags at once:
    filter {
      grok {
        add_tag => [ "foo_%{somefield}", "taggedy_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would add a tag foo_hello (and the second example would of course add a taggedy_tag tag).

Break on first match. The first successful match by grok will result in the filter being finished. If you want grok to try all patterns (maybe you are parsing different things), then set this to false.

If true, keep empty captures as event fields.

A hash of matches of field ⇒ value

For example:

    filter {
      grok { match => { "message" => "Duration: %{NUMBER:duration}" } }
    }

If you need to match multiple patterns against a single field, the value can be an array of patterns

    filter {
      grok { match => { "message" => [ "Duration: %{NUMBER:duration}", "Speed: %{NUMBER:speed}" ] } }
    }

If true, only store named captures from grok.

The fields to overwrite.

This allows you to overwrite a value in a field that already exists.

For example, if you have a syslog line in the message field, you can overwrite the message field with part of the match like so:

    filter {
      grok {
        match => { "message" => "%{SYSLOGBASE} %{DATA:message}" }
        overwrite => [ "message" ]
      }
    }

In this case, a line like May 29 16:37:11 sadness logger: hello world will be parsed and hello world will overwrite the original message.

Specify a pattern to parse with. This will match the message field.

If you want to match other fields than message, use the match setting. Multiple patterns is fine.

Logstash ships by default with a bunch of patterns, so you don’t necessarily need to define this yourself unless you are adding additional patterns. You can point to multiple pattern directories using this setting Note that Grok will read all files in the directory matching the patterns_files_glob and assume its a pattern file (including any tilde backup files)

    patterns_dir => ["/opt/logstash/patterns", "/opt/logstash/extra_patterns"]

Pattern files are plain text with format:

    NAME PATTERN

For example:

    NUMBER \d+

Glob pattern, used to select the pattern files in the directories specified by patterns_dir

Call the filter flush method at regular interval. Optional.

If this filter is successful, remove arbitrary fields from this event. Fields names can be dynamic and include parts of the event using the %{field} Example:

    filter {
      grok {
        remove_field => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also remove multiple fields at once:
    filter {
      grok {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the field with name foo_hello if it is present. The second example would remove an additional, non-dynamic field.

If this filter is successful, remove arbitrary tags from the event. Tags can be dynamic and include parts of the event using the %{field} syntax.

Example:

    filter {
      grok {
        remove_tag => [ "foo_%{somefield}" ]
      }
    }
[source,ruby]
    # You can also remove multiple tags at once:
    filter {
      grok {
        remove_tag => [ "foo_%{somefield}", "sad_unwanted_tag"]
      }
    }

If the event has field "somefield" == "hello" this filter, on success, would remove the tag foo_hello if it is present. The second example would remove a sad, unwanted tag as well.

If true, make single-value fields simply that value, not an array containing that one value.

Append values to the tags field when there has been no successful match