IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Adding a policy to an index Managing Fields »
Elastic Docs ›Kibana Guide [7.4] ›Management ›Index lifecycle policies

Tutorial: Use ILM to manage Filebeat time-based indices

edit

Tutorial: Use ILM to manage Filebeat time-based indices

edit

With index lifecycle management (ILM), you can create policies that perform actions automatically on indices as they age and grow. ILM policies help you to manage performance, resilience, and retention of your data during its lifecycle. This tutorial shows you how to use Kibana’s Index Lifecycle Policies to modify and create ILM policies. You can learn more about all of the actions, benefits, and lifecycle phases in the ILM overview.

Scenario

edit

You’re tasked with sending syslog files to an Elasticsearch cluster. This log data has the following data retention guidelines:

  • Keep logs on hot data nodes for 30 days
  • Roll over to a new index if the size reaches 50GB

  • After 30 days:

    • Move the logs to warm data nodes
    • Set replica shards to 1
    • Force merge multiple index segments to free up the space used by deleted documents
  • Delete logs after 90 days

Prerequisites

edit

To complete this tutorial, you’ll need:

  • An Elasticsearch cluster with hot and warm nodes configured for shard allocation awareness. If you’re using Elasticsearch Service, choose the hot-warm architecture deployment template.

    For a self-managed cluster, add node attributes as described for shard allocation filtering to label data nodes as hot or warm. This step is required to migrate shards between nodes configured with specific hardware for the hot or warm phases.

    For example, you can set this in your elasticsearch.yml for each data node:

    node.attr.data: "warm"
  • A server with Filebeat installed and configured to send logs to the elasticsearch output as described in Getting Started with Filebeat.

View the Filebeat ILM policy

edit

Filebeat includes a default ILM policy that enables rollover. ILM is enabled automatically if you’re using the default filebeat.yml and index template.

To view the default policy in Kibana, go to Management > Index Lifecycle Policies, search for filebeat, and choose the filebeat-version policy.

This policy initiates the rollover action when the index size reaches 50GB or becomes 30 days old.

Default policy

Modify the policy

edit

The default policy is enough to prevent the creation of many tiny daily indices. You can modify the policy to meet more complex requirements.

  1. Activate the warm phase.

  2. Set either of the following options to control when the index moves to the warm phase:

    • Provide a value for Timing for warm phase. Setting this to 15 keeps the indices on hot nodes for a range of 15-45 days, depending on when the initial rollover occurred.
    • Enable Move to warm phase on rollover. The index might move to the warm phase more quickly than intended if it reaches the Maximum index size before the the Maximum age.
  3. In the Select a node attribute to control shard allocation dropdown, select data:warm(2) to migrate shards to warm data nodes.
  4. Change Number of replicas to 1.

  5. Enable Force merge data and set Number of segments to 1.

    When rollover is enabled in the hot phase, action timing in the other phases is based on the rollover date.

    Modify to add warm phase

  6. Activate the delete phase and set Timing for delete phase to 90 days.

    Add a delete phase

Create a custom policy

edit

If meeting a specific retention time period is most important, you can create a custom policy. For this option, you will use Filebeat daily indices without rollover.

  1. Create a custom policy in Kibana, go to Management > Index Lifecycle Policies > Create Policy.

  2. Activate the warm phase and configure it as follows:

    Setting Value

    Timing for warm phase

    30 days from index creation

    Node attribute

    data:warm

    Number of replicas

    1

    Force merge data

    enable

    Number of segments

    1
    Modify the custom policy to add a warm phase

  3. Activate the delete phase and set the timing.

    Setting

    Value

    Timing for delete phase

    90
    Delete phase

  4. Configure the index to use the new policy in Kibana > Management > Index Lifecycle Policies

    1. Find your ILM policy.
    2. Click the Actions link next to your policy name.
    3. Choose Add policy to index template.
    4. Select your Filebeat index template name from the Index template list. For example, filebeat-7.5.x.

    5. Click Add Policy to save the changes.

      If you initially used the default Filebeat ILM policy, you will see a notice that the template already has a policy associated with it. Confirm that you want to overwrite that configuration.

      When you change the policy associated with the index template, the active index will continue to use the policy it was associated with at index creation unless you manually update it. The next new index will use the updated policy. For more reasons that your ILM policy changes might be delayed, see Update Lifecycle Policy.

« Adding a policy to an index Managing Fields »