- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- rate_limit
- registered_domain
- rename
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- Proofpoint module
- RabbitMQ module
- Radware module
- Redis module
- Salesforce module
- Santa module
- Snort module
- Snyk module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Threat Intel module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- Proofpoint Email Security fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Salesforce fields
- Google Santa fields
- Snort/Sourcefire fields
- Snyk fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- threatintel fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Configure general settings
editConfigure general settings
editYou can specify settings in the filebeat.yml
config file to control the
general behavior of Filebeat. This includes:
- Global options that control things like publisher behavior and the location of some files.
- General options that are supported by all Elastic Beats.
Global Filebeat configuration options
editThese options are in the filebeat
namespace.
registry.path
editThe root path of the registry. If a relative path is used, it is considered
relative to the data path. See the Directory layout section for details.
The default is ${path.data}/registry
.
filebeat.registry.path: registry
The registry is only updated when new events are flushed and not on a predefined period. That means in case there are some states where the TTL expired, these are only removed when new events are processed.
registry.file_permissions
editThe permissions mask to apply on registry data file. The default value is 0600. The permissions option must be a valid Unix-style file permissions mask expressed in octal notation. In Go, numbers in octal notation must start with 0.
The most permissive mask allowed is 0640. If a higher permissions mask is specified via this setting, it will be subject to an umask of 0027.
This option is not supported on Windows.
Examples:
- 0640: give read and write access to the file owner, and read access to members of the group associated with the file.
- 0600: give read and write access to the file owner, and no access to all others.
filebeat.registry.file_permissions: 0600
registry.flush
editThe timeout value that controls when registry entries are written to disk
(flushed). When an unwritten update exceeds this value, it triggers a write to
disk. When registry.flush
is set to 0s, the registry is written to disk after
each batch of events has been published successfully. The default value is 1s.
The registry is always updated when Filebeat shuts down normally. After an
abnormal shutdown, the registry will not be up-to-date if the registry.flush
value is >0s. Filebeat will send published events again (depending on values in
the last updated registry file).
Filtering out a huge number of logs can cause many registry updates, slowing
down processing. Setting registry.flush
to a value >0s reduces write operations,
helping Filebeat process more events.
registry.migrate_file
editPrior to Filebeat 7.0 the registry is stored in a single file. When you upgrade
to 7.0, Filebeat will automatically migrate the old Filebeat 6.x registry file
to use the new directory format. Filebeat looks for the file in the location
specified by filebeat.registry.path
. If you changed the path while upgrading,
set filebeat.registry.migrate_file
to point to the old registry file.
filebeat.registry.path: ${path.data}/registry filebeat.registry.migrate_file: /path/to/old/registry_file
The registry will be migrated to the new location only if a registry using the directory format does not already exist.
config_dir
edit[6.0.0] Deprecated in 6.0.0. Use Input config instead.
The full path to the directory that contains additional input configuration files.
Each configuration file must end with .yml
. Each config file must also specify the full Filebeat
config hierarchy even though only the inputs
part of each file is processed. All global
options, such as registry_file
, are ignored.
The config_dir
option MUST point to a directory other than the directory where the main Filebeat config file resides.
If the specified path is not absolute, it is considered relative to the configuration path. See the Directory layout section for details.
filebeat.config_dir: path/to/configs
shutdown_timeout
editHow long Filebeat waits on shutdown for the publisher to finish sending events before Filebeat shuts down.
By default, this option is disabled, and Filebeat does not wait for the publisher to finish sending events before shutting down. This means that any events sent to the output, but not acknowledged before Filebeat shuts down, are sent again when you restart Filebeat. For more details about how this works, see How does Filebeat ensure at-least-once delivery?.
You can configure the shutdown_timeout
option to specify the maximum amount
of time that Filebeat waits for the publisher to finish sending events before
shutting down. If all events are acknowledged before shutdown_timeout
is
reached, Filebeat will shut down.
There is no recommended setting for this option because determining the correct
value for shutdown_timeout
depends heavily on the environment in which
Filebeat is running and the current state of the output.
Example configuration:
filebeat.shutdown_timeout: 5s
General configuration options
editThese options are supported by all Elastic Beats. Because they are common options, they are not namespaced.
Here is an example configuration:
name: "my-shipper" tags: ["service-X", "web-tier"]
name
editThe name of the Beat. If this option is empty, the hostname
of the server is
used. The name is included as the agent.name
field in each published transaction. You can
use the name to group all transactions sent by a single Beat.
Example:
name: "my-shipper"
tags
editA list of tags that the Beat includes in the tags
field of each published
transaction. Tags make it easy to group servers by different logical properties.
For example, if you have a cluster of web servers, you can add the "webservers"
tag to the Beat on each server, and then use filters and queries in the Kibana
web interface to get visualisations for the whole group of servers.
Example:
tags: ["my-service", "hardware", "test"]
fields
editOptional fields that you can specify to add additional information to the
output. Fields can be scalar values, arrays, dictionaries, or any nested
combination of these. By default, the fields that you specify here will be
grouped under a fields
sub-dictionary in the output document. To store the
custom fields as top-level fields, set the fields_under_root
option to true.
Example:
fields: {project: "myproject", instance-id: "574734885120952459"}
fields_under_root
editIf this option is set to true, the custom fields are
stored as top-level fields in the output document instead of being grouped under
a fields
sub-dictionary. If the custom field names conflict with other field
names, then the custom fields overwrite the other fields.
Example:
fields_under_root: true fields: instance_id: i-10a64379 region: us-east-1
processors
editA list of processors to apply to the data generated by the beat.
See Processors for information about specifying processors in your config.
max_procs
editSets the maximum number of CPUs that can be executing simultaneously. The default is the number of logical CPUs available in the system.
timestamp.precision
editConfigure the precision of all timestamps. By default it is set to millisecond. Available options: millisecond, microsecond, nanosecond
On this page