- Fleet and Elastic Agent Guide: other versions:
- Fleet and Elastic Agent overview
- Beats and Elastic Agent capabilities
- Quick starts
- Migrate from Beats to Elastic Agent
- Deployment models
- Install Elastic Agents
- Install Fleet-managed Elastic Agents
- Install standalone Elastic Agents
- Install Elastic Agents in a containerized environment
- Run Elastic Agent in a container
- Run Elastic Agent on Kubernetes managed by Fleet
- Install Elastic Agent on Kubernetes using Helm
- Example: Install standalone Elastic Agent on Kubernetes using Helm
- Example: Install Fleet-managed Elastic Agent on Kubernetes using Helm
- Advanced Elastic Agent configuration managed by Fleet
- Configuring Kubernetes metadata enrichment on Elastic Agent
- Run Elastic Agent on GKE managed by Fleet
- Run Elastic Agent on Amazon EKS managed by Fleet
- Run Elastic Agent on Azure AKS managed by Fleet
- Run Elastic Agent Standalone on Kubernetes
- Scaling Elastic Agent on Kubernetes
- Using a custom ingest pipeline with the Kubernetes Integration
- Environment variables
- Run Elastic Agent as an OTel Collector
- Run Elastic Agent without administrative privileges
- Install Elastic Agent from an MSI package
- Installation layout
- Air-gapped environments
- Using a proxy server with Elastic Agent and Fleet
- Uninstall Elastic Agents from edge hosts
- Start and stop Elastic Agents on edge hosts
- Elastic Agent configuration encryption
- Secure connections
- Manage Elastic Agents in Fleet
- Configure standalone Elastic Agents
- Create a standalone Elastic Agent policy
- Structure of a config file
- Inputs
- Providers
- Outputs
- SSL/TLS
- Logging
- Feature flags
- Agent download
- Config file examples
- Grant standalone Elastic Agents access to Elasticsearch
- Example: Use standalone Elastic Agent with Elastic Cloud Serverless to monitor nginx
- Example: Use standalone Elastic Agent with Elasticsearch Service to monitor nginx
- Debug standalone Elastic Agents
- Kubernetes autodiscovery with Elastic Agent
- Monitoring
- Reference YAML
- Manage integrations
- Package signatures
- Add an integration to an Elastic Agent policy
- View integration policies
- Edit or delete an integration policy
- Install and uninstall integration assets
- View integration assets
- Set integration-level outputs
- Upgrade an integration
- Managed integrations content
- Best practices for integrations assets
- Data streams
- Define processors
- Processor syntax
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Command reference
- Troubleshoot
- Release notes
Tutorial: Customize data retention policies
editTutorial: Customize data retention policies
editThis tutorial explains how to apply a custom ILM policy to an integration’s data stream.
Scenario: You have Elastic Agents collecting system metrics with the System integration in two environments—one with the namespace development
, and one with production
.
Goal: Customize the ILM policy for the system.network
data stream in the production
namespace.
Specifically, apply the built-in 90-days-default
ILM policy so that data is deleted after 90 days.
If you cloned an index template to customize the data retention policy on an Elasticsearch version prior to 8.13, you must update the index template in the clone to use the ecs@mappings
component template on Elasticsearch version 8.13 or later. See Update index template cloned before Elasticsearch 8.13 for the step-by-step instructions.
Step 1: View data streams
editThe Data Streams view in Kibana shows you the data streams, index templates, and ILM policies associated with a given integration.
- Navigate to Stack Management > Index Management > Data Streams.
-
Search for
system
to see all data streams associated with the System integration. -
Select the
metrics-system.network-{namespace}
data stream to view its associated index template and ILM policy. As you can see, the data stream follows the Data stream naming scheme and starts with its type,metrics-
.
Step 2: Create a component template
editFor your changes to continue to be applied in future versions,
you must put all custom index settings into a component template.
The component template must follow the data stream naming scheme,
and end with @custom
:
<type>-<dataset>-<namespace>@custom
For example, to create custom index settings for the system.network
data stream with a namespace of production
,
the component template name would be:
metrics-system.network-production@custom
- Navigate to Stack Management > Index Management > Component Templates
- Click Create component template.
-
Use the template above to set the name—in this case,
metrics-system.network-production@custom
. Click Next. -
Under Index settings, set the ILM policy name under the
lifecycle.name
key:{ "lifecycle": { "name": "90-days-default" } }
-
Continue to Review and ensure your request looks similar to the image below. If it does, click Create component template.
Step 3: Clone and modify the existing index template
editNow that you’ve created a component template, you need to create an index template to apply the changes to the correct data stream. The easiest way to do this is to duplicate and modify the integration’s existing index template.
When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading.
- Navigate to Stack Management > Index Management > Index Templates.
-
Find the index template you want to clone. The index template will have the
<type>
and<dataset>
in its name, but not the<namespace>
. In this case, it’smetrics-system.network
. - Select Actions > Clone.
-
Set the name of the new index template to
metrics-system.network-production
. -
Change the index pattern to include a namespace—in this case,
metrics-system.network-production*
. This ensures the previously created component template is only applied to theproduction
namespace. -
Set the priority to
250
. This ensures that the new index template takes precedence over other index templates that match the index pattern. -
Under Component templates, search for and add the component template created in the previous step.
To ensure your namespace-specific settings are applied over other custom settings,
the new template should be added below the existing
@custom
template. - Create the index template.

Step 4: Roll over the data stream (optional)
editTo confirm that the data stream is now using the new index template and ILM policy, you can either repeat step one, or navigate to Dev Tools and run the following:
The result should include the following:
{ "data_streams" : [ { ... "template" : "metrics-system.network-production", "ilm_policy" : "90-days-default", ... } ] }
The name of the custom index template created in step three |
|
The name of the ILM policy applied to the new component template in step two |
New ILM policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:
POST /metrics-system.network-production/_rollover/
Update index template cloned before Elasticsearch 8.13
editIf you cloned an index template to customize the data retention policy on an Elasticsearch version prior to 8.13, you must update the index cloned index template to add the ecs@mappings
component template on Elasticsearch version 8.13 or later.
To update the cloned index template:
- Navigate to Stack Management > Index Management > Index Templates.
-
Find the index template you cloned. The index template will have the
<type>
and<dataset>
in its name. - Select Manage > Edit.
- Select (2) Component templates
-
In the Search component templates field, search for
ecs@mappings
. -
Click on the + (plus) icon to add the
ecs@mappings
component template. -
Move the
ecs@mappings
component template right below the@package
component template. - Save the index template.
Roll over the data stream to apply the changes.
On this page