NOTE: You are looking at documentation for an older release. For the latest information, see the current release documentation.

« Change the output codec Load balance the output hosts »

› ›

Set up index lifecycle management

edit

Set up index lifecycle management

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

You can use the index lifecycle management feature in Elasticsearch to manage your Filebeat indices as they age. For example, instead of having Filebeat create daily indices where index size can vary based on the number of Beats and number of events sent, you can use an index lifecycle policy that automates a rollover to a new index when the existing index reaches a specified size or age.

Filebeat provides a default policy that you can load when you set up Filebeat. The default policy is applied to any new indices created by Filebeat. You can edit the policy to modify the lifecycle of both new and existing indices.

To use index lifecycle management on Filebeat indices:

Enable index lifecycle management by setting ilm.enabled: true in the Elasticsearch output configuration. For example:
```
output.elasticsearch:
  hosts: ["localhost:9200"]
  ilm.enabled: true
```
This configuration overwrites your index settings and adjusts the Filebeat template to use the correct settings for index lifecycle management.

If you’ve previously loaded the index template for this version into Elasticsearch, you must overwrite the template by setting setup.template.overwrite: true.

The rollover alias is set to filebeat-\{beat.version\} by default. You can change the prefix used in the alias by setting ilm.rollover_alias, but you can’t remove {beat.version} from the rollover alias name. The default pattern used for the rollover index is %{now/d}-000001. You can change the pattern by setting ilm.pattern. For example:
```
output.elasticsearch:
  hosts: ["localhost"]
  ilm.enabled: true
  ilm.rollover_alias: "filebeat"
  ilm.pattern: "{now/d}-000001" 
```
Date math is supported here. For more information, see Using date math with the rollover API.

If you modify the rollover_alias or pattern settings after loading the index template, you must overwrite the template to apply the changes.
Load the default policy into Elasticsearch. You can either use the setup command to load the policy without modifying it, or modify the policy and load it manually.

To use the setup command, run:
```
filebeat setup --ilm-policy
```
After loading the default policy, you can edit it in the Index lifecycle policies UI in Kibana. For more information about working with the UI, see Index lifecyle policies.

To modify the default policy before loading it, run filebeat export ilm-policy to print the policy to stdout. Modify the policy then use the Create lifecycle policy API to load it into Elasticsearch.

Advanced ILM settings

edit

We recommend that you avoid modifying these settings unless you know what you’re doing.

The default index lifecycle management settings work best for common use cases that work with the automated alias setup described earlier. It is possible to use a multiple write alias with dynamic index patterns, but this requires manual set up. This section describes the configuration options you need to change.

Let’s assume you have the index pattern customname-%{event.module} where event.module can have the values system and apache. First you must set up a rollover index for customname-system and customname-apache. For details on how to do this, see Rollover Index.

Next, set the index pattern in the Elasticsearch output. For example:

output.elasticesarch.index: customname-%{event.module}

For this example to work, every event must contain event.module.

If you change the index name, you must also set the template name, template pattern, rollover alias, and lifecycle name. The best way to set these is through an Elasticsearch template. It’s possible to disable the template loading in Filebeat and specify these settings in your own template. Or you can use the following config options in Filebeat:

setup.template.name: "customname"
setup.template.pattern: "customname-*"
setup.template.settings.index.lifecycle.rollover_alias: "customname"
setup.template.settings.index.lifecycle.name: "beats-default-policy"

If you set the options manually as shown in this example, do not set ilm.enabled, or the settings specified in the configuration file will be overwritten.

This configuration results in a managed index named something like customname-2024-11-08-000001 and the following index settings:

"aliases" : {
  "customname" : {
    "is_write_index" : true
  }
},
...
  "index" : {
    "lifecycle" : {
      "name" : "beats-default-policy",
      "rollover_alias" : "customname"
    },

« Change the output codec Load balance the output hosts »