- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- Proofpoint module
- RabbitMQ module
- Radware module
- Redis module
- Salesforce module
- Santa module
- Snort module
- Snyk module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Threat Intel module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Lumberjack fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- Proofpoint Email Security fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Salesforce fields
- Google Santa fields
- Snort/Sourcefire fields
- Snyk fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- threatintel fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Configure logging
editConfigure logging
editThe logging
section of the filebeat.yml
config file contains options
for configuring the logging output.
The logging system can write logs to the syslog or rotate log files. If logging
is not explicitly configured the file output is used.
logging.level: info logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0640
In addition to setting logging options in the config file, you can modify the logging output configuration from the command line. See Command reference.
When Filebeat is running on a Linux system with systemd, it uses
by default the -e
command line option, that makes it write all the logging output
to stderr so it can be captured by journald. Other outputs are disabled. See
Filebeat and systemd to know more and learn how to change this.
Configuration options
editYou can specify the following options in the logging
section of the
filebeat.yml
config file:
logging.to_stderr
editWhen true, writes all logging output to standard error output. This is
equivalent to using the -e
command line option.
logging.to_syslog
editWhen true, writes all logging output to the syslog.
This option is not supported on Windows.
logging.to_eventlog
editWhen true, writes all logging output to the Windows Event Log.
logging.to_files
editWhen true, writes all logging output to files. The log files are automatically rotated when the log file size limit is reached.
Filebeat only creates a log file if there is logging output. For
example, if you set the log level
to error
and there are no
errors, there will be no log file in the directory specified for logs.
logging.level
editMinimum log level. One of debug
, info
, warning
, or error
. The default
log level is info
.
-
debug
-
Logs debug messages, including a detailed printout of all events
flushed. Also logs informational messages, warnings, errors, and
critical errors. When the log level is
debug
, you can specify a list ofselectors
to display debug messages for specific components. If no selectors are specified, the*
selector is used to display debug messages for all components. -
info
- Logs informational messages, including the number of events that are published. Also logs any warnings, errors, or critical errors.
-
warning
- Logs warnings, errors, and critical errors.
-
error
- Logs errors and critical errors.
logging.selectors
editThe list of debugging-only selector tags used by different Filebeat components.
Use *
to enable debug output for all components. Use publisher
to display
debug messages related to event publishing.
The list of available selectors may change between releases, so avoid creating tests that depend on specific selectors.
To see which selectors are available, run Filebeat in debug mode
(set logging.level: debug
in the configuration). The selector name appears
after the log level and is enclosed in brackets.
To configure multiple selectors, use the following YAML list syntax:
logging.selectors: [ harvester, input ]
To override selectors at the command line, use the -d
global flag (-d
also
sets the debug log level). For more information, see Command reference.
logging.metrics.enabled
editBy default, Filebeat periodically logs its internal metrics that have changed in the last period. For each metric that changed, the delta from the value at the beginning of the period is logged. Also, the total values for all non-zero internal metrics are logged on shutdown. Set this to false to disable this behavior. The default is true.
Here is an example log line:
2017-12-17T19:17:42.667-0500 INFO [metrics] log/log.go:110 Non-zero metrics in the last 30s: beat.info.uptime.ms=30004 beat.memstats.gc_next=5046416
Note that we currently offer no backwards compatible guarantees for the internal metrics and for this reason they are also not documented.
logging.metrics.period
editThe period after which to log the internal metrics. The default is 30s.
logging.metrics.namespaces
editA list of metrics namespaces to report in the logs. Defaults to [stats]
.
stats
contains general Beat metrics. dataset
may be present in some
Beats and contains module or input metrics.
logging.files.path
editThe directory that log files are written to. The default is the logs path. See the Directory layout section for details.
logging.files.name
editThe name of the file that logs are written to. The default is filebeat.
logging.files.rotateeverybytes
editThe maximum size of a log file. If the limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB).
logging.files.keepfiles
editThe number of most recent rotated log files to keep on disk. Older files are
deleted during log rotation. The default value is 7. The keepfiles
options has
to be in the range of 2 to 1024 files.
logging.files.permissions
editThe permissions mask to apply when rotating log files. The default value is
0600. The permissions
option must be a valid Unix-style file permissions mask
expressed in octal notation. In Go, numbers in octal notation must start with
0.
The most permissive mask allowed is 0640. If a higher permissions mask is specified via this setting, it will be subject to an umask of 0027.
This option is not supported on Windows.
Examples:
- 0640: give read and write access to the file owner, and read access to members of the group associated with the file.
- 0600: give read and write access to the file owner, and no access to all others.
logging.files.interval
editEnable log file rotation on time intervals in addition to size-based rotation. Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h are boundary-aligned with minutes, hours, days, weeks, months, and years as reported by the local system clock. All other intervals are calculated from the unix epoch. Defaults to disabled.
logging.files.rotateonstartup
editIf the log file already exists on startup, immediately rotate it and start writing to a new file instead of appending to the existing one. Defaults to true.
logging.files.redirect_stderr
[preview]
This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
editWhen true, diagnostic messages printed to Filebeat’s standard error output
will also be logged to the log file. This can be helpful in situations were
Filebeat terminates unexpectedly because an error has been detected by
Go’s runtime but diagnostic information is not present in the log file.
This feature is only available when logging to files (logging.to_files
is true).
Disabled by default.
Logging format
editThe logging format is generally the same for each logging output. The one exception is with the syslog output where the timestamp is not included in the message because syslog adds its own timestamp.
Each log message consists of the following parts:
- Timestamp in ISO8601 format
- Level
- Logger name contained in brackets (Optional)
- File name and line number of the caller
- Message
- Structured data encoded in JSON (Optional)
Below are some samples:
2017-12-17T18:54:16.241-0500 INFO logp/core_test.go:13 unnamed global logger
2017-12-17T18:54:16.242-0500 INFO [example] logp/core_test.go:16 some message
2017-12-17T18:54:16.242-0500 INFO [example] logp/core_test.go:19 some message {"x": 1}
On this page
- Configuration options
logging.to_stderr
logging.to_syslog
logging.to_eventlog
logging.to_files
logging.level
logging.selectors
logging.metrics.enabled
logging.metrics.period
logging.metrics.namespaces
logging.files.path
logging.files.name
logging.files.rotateeverybytes
logging.files.keepfiles
logging.files.permissions
logging.files.interval
logging.files.rotateonstartup
logging.files.redirect_stderr
[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.- Logging format