Formatters

edit

Our log formatters allow you to change the way various logging frameworks log to IO (file/console) to use ECS json.

ECS Aware Message Templates

edit

Our log formatters allow you to set ECS fields directly from the message template using properties that adhere to the https://messagetemplates.org/ format.

Log.Information("The time is {TraceId}", "my-trace-id");

Will directly override trace.id on the resulting ECS json document.

All supported ECS message template properties are available as constants under the LogTemplateProperties static class. For example LogTemplateProperties.TraceId will return "TraceId".

Ingest ECS log files

edit

If you are using one of our formatter libraries to log to file or stdout/stderr you can use the following options to get these logs into Elasticsearch or Elastic Cloud:

We also support writing logs directly to Elasticsearch or Elastic Cloud. See our various data shipper integrations

  1. Follow the Filebeat quick start
  2. Add the following configuration to your filebeat.yaml file.

For Filebeat 7.16+

filebeat.yaml.

filebeat.inputs:
- type: filestream 
  paths: /path/to/logs.json
  parsers:
    - ndjson:
      overwrite_keys: true 
      add_error_key: true 
      expand_keys: true 

processors: 
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

Use the filestream input to read lines from active log files.

Values from the decoded JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.) in case of conflicts.

Filebeat adds an "error.message" and "error.type: json" key in case of JSON unmarshalling errors.

Filebeat will recursively de-dot keys in the decoded JSON, and expand them into a hierarchical object structure.

Processors enhance your data. See processors to learn more.

For Filebeat < 7.16

filebeat.yaml.

filebeat.inputs:
- type: log
  paths: /path/to/logs.json
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true

processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~

For more information, see the Filebeat reference.