Tutorial: Customize data retention policies

This tutorial explains how to apply a custom ILM policy to an integration’s data stream.

Scenario: You have Elastic Agents collecting system metrics with the System integration in two environments—one with the namespace development, and one with production.

Goal: Customize the ILM policy for the system.network data stream in the production namespace. Specifically, apply the built-in 90-days-default ILM policy so that data is deleted after 90 days.

If you cloned an index template to customize the data retention policy on an Elasticsearch version prior to 8.13, you must update the index template in the clone to use the ecs@mappings component template on Elasticsearch version 8.13 or later. See Update index template cloned before Elasticsearch 8.13 for the step-by-step instructions.

Step 1: View data streams

edit

The Data Streams view in Kibana shows you the data streams, index templates, and ILM policies associated with a given integration.

Navigate to Stack Management > Index Management > Data Streams.
Search for system to see all data streams associated with the System integration.
Select the metrics-system.network-{namespace} data stream to view its associated index template and ILM policy. As you can see, the data stream follows the Data stream naming scheme and starts with its type, metrics-.

Step 2: Create a component template

edit

For your changes to continue to be applied in future versions, you must put all custom index settings into a component template. The component template must follow the data stream naming scheme, and end with @custom:

<type>-<dataset>-<namespace>@custom

For example, to create custom index settings for the system.network data stream with a namespace of production, the component template name would be:

metrics-system.network-production@custom

Navigate to Stack Management > Index Management > Component Templates
Click Create component template.
Use the template above to set the name—in this case, metrics-system.network-production@custom. Click Next.
Under Index settings, set the ILM policy name under the lifecycle.name key:
```
{
  "lifecycle": {
    "name": "90-days-default"
  }
}
```
Continue to Review and ensure your request looks similar to the image below. If it does, click Create component template.

Step 3: Clone and modify the existing index template

edit

Now that you’ve created a component template, you need to create an index template to apply the changes to the correct data stream. The easiest way to do this is to duplicate and modify the integration’s existing index template.

When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading.

Navigate to Stack Management > Index Management > Index Templates.
Find the index template you want to clone. The index template will have the <type> and <dataset> in its name, but not the <namespace>. In this case, it’s metrics-system.network.
Select Actions > Clone.
Set the name of the new index template to metrics-system.network-production.
Change the index pattern to include a namespace—in this case, metrics-system.network-production*. This ensures the previously created component template is only applied to the production namespace.
Set the priority to 250. This ensures that the new index template takes precedence over other index templates that match the index pattern.
Under Component templates, search for and add the component template created in the previous step. To ensure your namespace-specific settings are applied over other custom settings, the new template should be added below the existing @custom template.
Create the index template.

Step 4: Roll over the data stream (optional)

edit

To confirm that the data stream is now using the new index template and ILM policy, you can either repeat step one, or navigate to Dev Tools and run the following:

GET /_data_stream/metrics-system.network-production

The name of the data stream we’ve been hacking on

The result should include the following:

{
  "data_streams" : [
    {
      ...
      "template" : "metrics-system.network-production", 
      "ilm_policy" : "90-days-default", 
      ...
    }
  ]
}

	The name of the custom index template created in step three
	The name of the ILM policy applied to the new component template in step two

New ILM policies only take effect when new indices are created, so you either must wait for a rollover to occur (usually after 30 days or when the index size reaches 50 GB), or force a rollover using the Elasticsearch rollover API:

POST /metrics-system.network-production/_rollover/

Update index template cloned before Elasticsearch 8.13

edit

If you cloned an index template to customize the data retention policy on an Elasticsearch version prior to 8.13, you must update the index cloned index template to add the ecs@mappings component template on Elasticsearch version 8.13 or later.

To update the cloned index template:

Navigate to Stack Management > Index Management > Index Templates.
Find the index template you cloned. The index template will have the <type> and <dataset> in its name.
Select Manage > Edit.
Select (2) Component templates
In the Search component templates field, search for ecs@mappings.
Click on the + (plus) icon to add the ecs@mappings component template.
Move the ecs@mappings component template right below the @package component template.
Save the index template.

Roll over the data stream to apply the changes.

« Data streams Tutorial: Transform data with custom ingest pipelines »