Elasticsearch highlights

This list summarizes the most important enhancements in Elasticsearch 8.4. For the complete list, go to Elasticsearch release highlights.

This speeds up a few aggregations when they don’t have child aggregations. That’s super common, for example, the histogram at the top of Kibana’s discover tab is a date_histogram without any child aggregations. That particular aggregation is sped up by about 85% in our rally tests, dropping from 250ms to 30ms.

#81322

The rollover API and ILM actions now support minimum conditions for rollover.

Minimum conditions prevent rollover from occurring until they are met. That is, an index will rollover once one or more max conditions are satisfied and all min conditions are satisfied.

As an example, the following ILM policy would roll an index over if it is at least 7 days old or at least 100 gigabytes, but only as long as the index is not empty.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "7d",
            "max_size": "100gb",
            "min_docs": 1
          }
        }
      }
    }
  }
}

#83345

Infinite and adaptive retries – available in 8.4 – makes it possible for transforms to recover after a failure without any user intervention. Retries can be configured per transform. The transform retries become less frequent progressively. The interval between retries doubles after reaching a one-hour threshold. This is because the possibility that retries solve the problem is less likely after each failed retry.

In the Transforms page in Stack Management in Kibana, the number of retries can be configured when creating a new transform or editing an existing one.

#87361

The support for composite aggregations in datafeeds is now generally available.

Data frame analytics is even faster in 8.4. The new function automatically stops the process of hyperparameter optimization early in case the accuracy gain for a different set of hyperparameter values would be insignificant. The early stopping of the optimization process results in a shorter runtime for the data frame analytics job.

#88589

This change adds a knn option to the _search API to support ANN search. It’s powered by the same Lucene ANN capabilities as the old _knn_search endpoint. The knn option can be combined with other search features like queries and aggregations.

#88694

When Elasticsearch ingests documents, it creates several data structures that enable querying, aggregating, and retrieving data. One of those data structures is _source, which is used for reindexing, updating documents, and in Kibana’s Discover. Storing _source enables fetching the original documents exactly as they were indexed. However, fetching the exact original documents isn’t required by many use cases, and storing _source uses a lot of space.

This release introduces a new feature in technical preview called synthetic _source. For specific data types, synthetic _source can significantly reduce the index size by rebuilding _source from doc values. Doc values are data structures that are typically stored already because they’re needed for aggregations. While synthetic _source doesn’t recreate the exact structure of the original documents, it’s good enough for features like reindexing data.

#85649

The Elasticsearch 8.4.2 release introduced a bug that may fail queries using the cross_fields type at search time with an error message containing totalTermFreq must be at least docFreq. If you run queries that use the cross_fields type, we strongly recommend upgrading to 8.4.3 to avoid hitting this bug.

#90314