- Fleet and Elastic Agent Guide: other versions:
- Fleet and Elastic Agent overview
- Beats and Elastic Agent capabilities
- Quick starts
- Migrate from Beats to Elastic Agent
- Deployment models
- Install Elastic Agents
- Install Fleet-managed Elastic Agents
- Install standalone Elastic Agents
- Install Elastic Agents in a containerized environment
- Run Elastic Agent in a container
- Run Elastic Agent on Kubernetes managed by Fleet
- Install Elastic Agent on Kubernetes using Helm
- Example: Install standalone Elastic Agent on Kubernetes using Helm
- Example: Install Fleet-managed Elastic Agent on Kubernetes using Helm
- Advanced Elastic Agent configuration managed by Fleet
- Configuring Kubernetes metadata enrichment on Elastic Agent
- Run Elastic Agent on GKE managed by Fleet
- Run Elastic Agent on Amazon EKS managed by Fleet
- Run Elastic Agent on Azure AKS managed by Fleet
- Run Elastic Agent Standalone on Kubernetes
- Scaling Elastic Agent on Kubernetes
- Using a custom ingest pipeline with the Kubernetes Integration
- Environment variables
- Run Elastic Agent as an OTel Collector
- Run Elastic Agent without administrative privileges
- Install Elastic Agent from an MSI package
- Installation layout
- Air-gapped environments
- Using a proxy server with Elastic Agent and Fleet
- Uninstall Elastic Agents from edge hosts
- Start and stop Elastic Agents on edge hosts
- Elastic Agent configuration encryption
- Secure connections
- Manage Elastic Agents in Fleet
- Configure standalone Elastic Agents
- Create a standalone Elastic Agent policy
- Structure of a config file
- Inputs
- Providers
- Outputs
- SSL/TLS
- Logging
- Feature flags
- Agent download
- Config file examples
- Grant standalone Elastic Agents access to Elasticsearch
- Example: Use standalone Elastic Agent with Elastic Cloud Serverless to monitor nginx
- Example: Use standalone Elastic Agent with Elasticsearch Service to monitor nginx
- Debug standalone Elastic Agents
- Kubernetes autodiscovery with Elastic Agent
- Monitoring
- Reference YAML
- Manage integrations
- Package signatures
- Add an integration to an Elastic Agent policy
- View integration policies
- Edit or delete an integration policy
- Install and uninstall integration assets
- View integration assets
- Set integration-level outputs
- Upgrade an integration
- Managed integrations content
- Best practices for integrations assets
- Data streams
- Define processors
- Processor syntax
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Command reference
- Troubleshoot
- Release notes
Define processors
editDefine processors
editElastic Agent processors are lightweight processing components that you can use to parse, filter, transform, and enrich data at the source. For example, you can use processors to:
- reduce the number of exported fields
- enhance events with additional metadata
- perform additional processing and decoding
- sanitize data
Each processor receives an event, applies a defined action to the event, and returns the event. If you define a list of processors, they are executed in the order they are defined.
event -> processor 1 -> event1 -> processor 2 -> event2 ...
Elastic Agent processors execute before ingest pipelines, which means that your processor configurations cannot refer to fields that are created by ingest pipelines or Logstash. For more limitations, refer to What are some limitations of using processors?
Where are processors valid?
editThe processors described in this section are valid:
-
Under integration settings in the Integrations UI in Kibana. For example, when configuring an Nginx integration, you can define processors for a specific dataset under Advanced options. The processor in this example adds geo metadata to the Nginx access logs collected by Elastic Agent:
Some integrations do not currently support processors.
-
Under input configuration settings for standalone Elastic Agents. For example:
inputs: - type: logfile use_output: default data_stream: namespace: default streams: - data_stream: dataset: nginx.access type: logs ignore_older: 72h paths: - /var/log/nginx/access.log* tags: - nginx-access exclude_files: - .gz$ processors: - add_host_metadata: cache.ttl: 5m geo: name: nyc-dc1-rack1 location: '40.7128, -74.0060' continent_name: North America country_iso_code: US region_name: New York region_iso_code: NY city_name: New York - add_locale: null
You can define processors that apply to a specific input defined in the configuration. Applying a processor to all the inputs on a global basis is currently not supported.
What are some limitations of using processors?
editProcessors have the following limitations.
- Cannot enrich events with data from Elasticsearch or other custom data sources.
- Cannot process data after it’s been converted to the Elastic Common Schema (ECS) because the conversion is performed by Elasticsearch ingest pipelines. This means that your processor configuration cannot refer to fields that are created by ingest pipelines or Logstash because those fields are created after the processor runs, not before.
- May break integration ingest pipelines in Elasticsearch if the user-defined processing removes or alters fields expected by ingest pipelines.
-
If you create new fields via processors, you are responsible for setting up
field mappings in the
*-@custom
component template and making sure the new mappings are aligned with ECS.
What other options are available for processing data?
editThe Elastic Stack provides several options for processing data collected by Elastic Agent. The option you choose depends on what you need to do:
If you need to… | Do this… |
---|---|
Sanitize or enrich raw data at the source |
Use an Elastic Agent processor |
Convert data to ECS, normalize field data, or enrich incoming data |
Use ingest pipelines |
Define or alter the schema at query time |
Use runtime fields |
Do something else with your data |
Use Logstash plugins |
How are Elastic Agent processors different from Logstash plugins or ingest pipelines?
editLogstash plugins and ingest pipelines both require you to send data to another system for processing. Processors, on the other hand, allow you to apply processing logic at the source. This means that you can filter out data you don’t want to send across the connection, and you can spread some of the processing load across host systems running on edge nodes.
On this page