- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- rate_limit
- registered_domain
- rename
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- Proofpoint module
- RabbitMQ module
- Radware module
- Redis module
- Santa module
- Snort module
- Snyk module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Threat Intel module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- Proofpoint Email Security fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Google Santa fields
- Snort/Sourcefire fields
- Snyk fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- threatintel fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Script Processor
editScript Processor
editThe script
processor executes Javascript code to process an event. The processor
uses a pure Go implementation of ECMAScript 5.1 and has no external
dependencies. This can be useful in situations where one of the other processors
doesn’t provide the functionality you need to filter events.
The processor can be configured by embedding Javascript in your configuration file or by pointing the processor at external file(s).
processors: - script: lang: javascript source: > function process(event) { event.Tag("js"); }
This loads filter.js
from disk.
processors: - script: lang: javascript file: ${path.config}/filter.js
Parameters can be passed to the script by adding params
to the config.
This allows for a script to be made reusable. When using params
the
code must define a register(params)
function to receive the parameters.
processors: - script: lang: javascript tag: my_filter params: threshold: 15 source: > var params = {threshold: 42}; function register(scriptParams) { params = scriptParams; } function process(event) { if (event.Get("severity") < params.threshold) { event.Cancel(); } }
If the script defines a test()
function it will be invoked when the processor
is loaded. Any exceptions thrown will cause the processor to fail to load. This
can be used to make assertions about the behavior of the script.
function process(event) { if (event.Get("event.code") === 1102) { event.Put("event.action", "cleared"); } return event; } function test() { var event = process(new Event({event: {code: 1102}})); if (event.Get("event.action") !== "cleared") { throw "expected event.action === cleared"; } }
Configuration options
editThe script
processor has the following configuration settings:
-
lang
-
This field is required and its value must be
javascript
. -
tag
-
This is an optional identifier that is added to log messages. If defined
it enables metrics logging for this instance of the processor. The metrics
include the number of exceptions and a histogram of the execution times for
the
process
function. -
source
- Inline Javascript source code.
-
file
-
Path to a script file to load. Relative paths are interpreted as
relative to the
path.config
directory. Globs are expanded. -
files
-
List of script files to load. The scripts are concatenated together.
Relative paths are interpreted as relative to the
path.config
directory. And globs are expanded. -
params
-
A dictionary of parameters that are passed to the
register
of the script. -
tag_on_exception
-
Tag to add to events in case the Javascript code causes an
exception while processing an event. Defaults to
_js_exception
. -
timeout
-
This sets an execution timeout for the
process
function. When theprocess
function takes longer than thetimeout
period the function is interrupted. You can set this option to prevent a script from running for too long (like preventing an infinitewhile
loop). By default there is no timeout. -
max_cached_sessions
-
This sets the maximum number of Javascript VM sessions
that will be cached to avoid reallocation. The default is
4
.
Event API
editThe Event
object passed to the process
method has the following API.
Method | Description |
---|---|
|
Get a value from the event (either a scalar or an object). If the key does not
exist Example: |
|
Put a value into the event. If the key was already set then the previous value is returned. It throws an exception if the key cannot be set because one of the intermediate values is not an object. Example: |
|
Rename a key in the event. The target key must not exist. It returns true if the source key was successfully renamed to the target key. Example: |
|
Delete a field from the event. It returns true on success. Example: |
|
Flag the event as cancelled which causes the processor to drop event. Example: |
|
Append a tag to the Example: |
|
Example: |
On this page