Application logs

edit

Application logs provide valuable insight into events that have occurred within your services and applications.

There are two aspects to consider when monitoring application logs:

  • log format: are your logs plaintext or Elastic Common Schema (ECS) logs (structured JSON)
  • log ingestion: the tool you use to send and write logs to Elasticsearch

The format of your logs (structured or plaintext) influences your log ingestion strategy. For simplicity, this documentation uses Filebeat for ingestion, but any log ingestion tool can be used with an equivalent configuration.

Application logs can be correlated to APM services with log correlation.

Using an APM agent provides extra features:

Plaintext logs vs. ECS logs

edit

Logs are typically produced as either plaintext or structured. Plaintext logs contain only text and have no special formatting, for example:

2019-08-06T12:09:12.375Z INFO:spring-petclinic: Tomcat started on port(s): 8080 (http) with context path, org.springframework.boot.web.embedded.tomcat.TomcatWebServer
2019-08-06T12:09:12.379Z INFO:spring-petclinic: Started PetClinicApplication in 7.095 seconds (JVM running for 9.082), org.springframework.samples.petclinic.PetClinicApplication
2019-08-06T14:08:40.199Z DEBUG:spring-petclinic: init find form, org.springframework.samples.petclinic.owner.OwnerController

Structured logs, on the other hand, follow a predefined, repeatable pattern or structure. This structure is applied at write time—​preventing the need for parsing at ingest time. The Elastic Common Schema (ECS) defines a common set of fields to use when structuring logs. This structure allows logs to be easily ingested, and provides the ability to correlate, search, and aggregate on individual fields within your logs.

For example, the same example shown above might look like this when structured with ECS-compatable JSON:

{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"}
{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"}
{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"}

There are two ways to make your application write ECS-formatted logs: - use an ECS logging library - use an APM agent and log reformatting

Log reformatting

edit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

This feature is supported for the following APM agents: * Ruby * Python * Java

Log correlation

edit

Log correlation works at two levels:

  • at service level: annotation with service.name, service.version and service.environment allow to link logs with APM services
  • at trace level: annotation with trace.id and transaction.id allow to link logs with traces

Elastic APM integrates with the most popular logging frameworks in each programming language to accomplish this automatically.

Correlating your application logs with trace events allows you to:

  • view the context of a log and the parameters a user provided
  • view all logs belonging to a particular trace
  • easily move between logs and traces when debugging application issues in Kibana

Learn more about log correlation in the APM Guide: log correlation, or in any of the agent-specific ingestion guides:

How to ingest application logs

edit

There are four main ways to ingest application logs into the Elastic Stack. Your use case will help determine the method that’s right for you.

Here, Filebeat is used for reference, but any ingestion tool can be used with an equivalent configuration.

Plaintext logs

edit

Use Filebeat to parse and ingest raw, plain-text application logs.

Requirements

  • (Optional) Elastic APM agent for your programming language (for log correlation)
  • Raw, plain-text application logs stored on the file system
  • Filebeat configured to monitor and capture application logs

Pros

  • All programming languages/frameworks are supported
  • Existing application logs can be ingested
  • Does not require modification of the application or its configuration, unless log correlation is required

Cons

  • Must parse application logs to be useful—meaning writing and maintaining Grok patterns and spending CPU cycles on parsing
  • Parsing is tied to the application log format, meaning it can differ per application and needs to be maintained over time
  • Log correlation requires modifying the application log format and inject IDs in log messages

Learn more: Plaintext logs.

ECS logging

edit

Elastic Common Schema (ECS) loggers format your logs into ECS-compatible JSON, removing the need to manually parse logs.

Requirements

  • (Optional) Elastic APM agent for your programming language (for log correlation)
  • The Elastic ECS logger for your language or framework
  • Filebeat configured to monitor and capture application logs

Pros

  • Popular logging frameworks supported
  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation
  • Resilient in case of outages

Cons

  • Not all frameworks are supported
  • Requires modification of the application and its log configuration

Learn more: ECS logging

Log reformatting

edit

Elastic APM agents can automatically reformat application logs to Elastic Common Schema (ECS) format without needing to add an ECS logger dependency or modify the application.

Requirements

  • The Elastic APM agent for your programming language
  • Filebeat configured to monitor and capture application logs

Pros

All the benefits of using ECS logging, without having to modify the application or its configuration:

  • Simplicity: no manual parsing with Filebeat, and a configuration can be reused across applications
  • Decently human-readable JSON structure
  • APM log correlation

Cons

  • Requires an Elastic APM agent
  • Not all APM agents support this feature

Supported APM agents/languages

Learn more: APM agent ECS reformatting

APM agent log sending

edit

Elastic APM agents can automatically capture and send logs directly to APM Server—​enabling you to easily ingest log events without needing a separate log shipper like Filebeat.

Requirements

The Elastic APM agent for your programming language.

Pros

  • Simple to set up as it only relies on the APM agent
  • No modification of the application required
  • No need to deploy Filebeat
  • No need to store log files in the file system.

Cons

  • Experimental feature
  • Limited APM agent support
  • Not resilient to outages: Log messages can be dropped when buffered in the agent or in APM Server

Supported APM agents/languages

Learn more: APM agent log sending