Slow Log

Shard level slow search log allows to log slow search (query and fetch phases) into a dedicated log file.

Thresholds can be set for both the query phase of the execution, and fetch phase, here is a sample:

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.fetch.warn: 1s
index.search.slowlog.threshold.fetch.info: 800ms
index.search.slowlog.threshold.fetch.debug: 500ms
index.search.slowlog.threshold.fetch.trace: 200ms

All of the above settings are dynamic and can be set for each index using the update indices settings API. For example:

resp = client.indices.put_settings(
    index="my-index-000001",
    settings={
        "index.search.slowlog.threshold.query.warn": "10s",
        "index.search.slowlog.threshold.query.info": "5s",
        "index.search.slowlog.threshold.query.debug": "2s",
        "index.search.slowlog.threshold.query.trace": "500ms",
        "index.search.slowlog.threshold.fetch.warn": "1s",
        "index.search.slowlog.threshold.fetch.info": "800ms",
        "index.search.slowlog.threshold.fetch.debug": "500ms",
        "index.search.slowlog.threshold.fetch.trace": "200ms"
    },
)
print(resp)

response = client.indices.put_settings(
  index: 'my-index-000001',
  body: {
    'index.search.slowlog.threshold.query.warn' => '10s',
    'index.search.slowlog.threshold.query.info' => '5s',
    'index.search.slowlog.threshold.query.debug' => '2s',
    'index.search.slowlog.threshold.query.trace' => '500ms',
    'index.search.slowlog.threshold.fetch.warn' => '1s',
    'index.search.slowlog.threshold.fetch.info' => '800ms',
    'index.search.slowlog.threshold.fetch.debug' => '500ms',
    'index.search.slowlog.threshold.fetch.trace' => '200ms'
  }
)
puts response

const response = await client.indices.putSettings({
  index: "my-index-000001",
  settings: {
    "index.search.slowlog.threshold.query.warn": "10s",
    "index.search.slowlog.threshold.query.info": "5s",
    "index.search.slowlog.threshold.query.debug": "2s",
    "index.search.slowlog.threshold.query.trace": "500ms",
    "index.search.slowlog.threshold.fetch.warn": "1s",
    "index.search.slowlog.threshold.fetch.info": "800ms",
    "index.search.slowlog.threshold.fetch.debug": "500ms",
    "index.search.slowlog.threshold.fetch.trace": "200ms",
  },
});
console.log(response);

PUT /my-index-000001/_settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
  "index.search.slowlog.threshold.query.debug": "2s",
  "index.search.slowlog.threshold.query.trace": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "800ms",
  "index.search.slowlog.threshold.fetch.debug": "500ms",
  "index.search.slowlog.threshold.fetch.trace": "200ms"
}

By default thresholds are disabled (set to -1).

The logging is done on the shard level scope, meaning the execution of a search request within a specific shard. It does not encompass the whole search request, which can be broadcast to several shards in order to execute. Some of the benefits of shard level logging is the association of the actual execution on the specific machine, compared with request level.

The search slow log file is configured in the log4j2.properties file.

It is often useful to identify what triggered a slow running query. To include information about the user that triggered a slow search, use the index.search.slowlog.include.user setting.

resp = client.indices.put_settings(
    index="my-index-000001",
    settings={
        "index.search.slowlog.include.user": True
    },
)
print(resp)

response = client.indices.put_settings(
  index: 'my-index-000001',
  body: {
    'index.search.slowlog.include.user' => true
  }
)
puts response

const response = await client.indices.putSettings({
  index: "my-index-000001",
  settings: {
    "index.search.slowlog.include.user": true,
  },
});
console.log(response);

PUT /my-index-000001/_settings
{
  "index.search.slowlog.include.user": true
}

This will result in user information being included in the slow log.

{
  "@timestamp": "2024-02-21T12:42:37.255Z",
  "log.level": "WARN",
  "auth.type": "REALM",
  "elasticsearch.slowlog.id": "tomcat-123",
  "elasticsearch.slowlog.message": "[index6][0]",
  "elasticsearch.slowlog.search_type": "QUERY_THEN_FETCH",
  "elasticsearch.slowlog.source": "{\"query\":{\"match_all\":{\"boost\":1.0}}}",
  "elasticsearch.slowlog.stats": "[]",
  "elasticsearch.slowlog.took": "747.3micros",
  "elasticsearch.slowlog.took_millis": 0,
  "elasticsearch.slowlog.total_hits": "1 hits",
  "elasticsearch.slowlog.total_shards": 1,
  "user.name": "elastic",
  "user.realm": "reserved",
  "ecs.version": "1.2.0",
  "service.name": "ES_ECS",
  "event.dataset": "elasticsearch.index_search_slowlog",
  "process.thread.name": "elasticsearch[runTask-0][search][T#5]",
  "log.logger": "index.search.slowlog.query",
  "elasticsearch.cluster.uuid": "Ui23kfF1SHKJwu_hI1iPPQ",
  "elasticsearch.node.id": "JK-jn-XpQ3OsDUsq5ZtfGg",
  "elasticsearch.node.name": "node-0",
  "elasticsearch.cluster.name": "distribution_run"
}

If a call was initiated with an X-Opaque-ID header, then the ID is included in Search Slow logs in the elasticsearch.slowlog.id field. See X-Opaque-Id HTTP header for details and best practices.

The indexing slow log, similar in functionality to the search slow log. The log file name ends with _index_indexing_slowlog.json. Log and the thresholds are configured in the same way as the search slowlog. Index slowlog sample:

index.indexing.slowlog.threshold.index.warn: 10s
index.indexing.slowlog.threshold.index.info: 5s
index.indexing.slowlog.threshold.index.debug: 2s
index.indexing.slowlog.threshold.index.trace: 500ms
index.indexing.slowlog.source: 1000

All of the above settings are dynamic and can be set for each index using the update indices settings API. For example:

resp = client.indices.put_settings(
    index="my-index-000001",
    settings={
        "index.indexing.slowlog.threshold.index.warn": "10s",
        "index.indexing.slowlog.threshold.index.info": "5s",
        "index.indexing.slowlog.threshold.index.debug": "2s",
        "index.indexing.slowlog.threshold.index.trace": "500ms",
        "index.indexing.slowlog.source": "1000"
    },
)
print(resp)

response = client.indices.put_settings(
  index: 'my-index-000001',
  body: {
    'index.indexing.slowlog.threshold.index.warn' => '10s',
    'index.indexing.slowlog.threshold.index.info' => '5s',
    'index.indexing.slowlog.threshold.index.debug' => '2s',
    'index.indexing.slowlog.threshold.index.trace' => '500ms',
    'index.indexing.slowlog.source' => '1000'
  }
)
puts response

const response = await client.indices.putSettings({
  index: "my-index-000001",
  settings: {
    "index.indexing.slowlog.threshold.index.warn": "10s",
    "index.indexing.slowlog.threshold.index.info": "5s",
    "index.indexing.slowlog.threshold.index.debug": "2s",
    "index.indexing.slowlog.threshold.index.trace": "500ms",
    "index.indexing.slowlog.source": "1000",
  },
});
console.log(response);

PUT /my-index-000001/_settings
{
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.threshold.index.info": "5s",
  "index.indexing.slowlog.threshold.index.debug": "2s",
  "index.indexing.slowlog.threshold.index.trace": "500ms",
  "index.indexing.slowlog.source": "1000"
}

To include information about the user that triggered a slow indexing event, use the index.indexing.slowlog.include.user setting.

resp = client.indices.put_settings(
    index="my-index-000001",
    settings={
        "index.indexing.slowlog.include.user": True
    },
)
print(resp)

response = client.indices.put_settings(
  index: 'my-index-000001',
  body: {
    'index.indexing.slowlog.include.user' => true
  }
)
puts response

const response = await client.indices.putSettings({
  index: "my-index-000001",
  settings: {
    "index.indexing.slowlog.include.user": true,
  },
});
console.log(response);

PUT /my-index-000001/_settings
{
  "index.indexing.slowlog.include.user": true
}

By default Elasticsearch will log the first 1000 characters of the _source in the slowlog. You can change that with index.indexing.slowlog.source. Setting it to false or 0 will skip logging the source entirely, while setting it to true will log the entire source regardless of size. The original _source is reformatted by default to make sure that it fits on a single log line. If preserving the original document format is important, you can turn off reformatting by setting index.indexing.slowlog.reformat to false, which will cause the source to be logged "as is" and can potentially span multiple log lines.

The index slow log file is configured in the log4j2.properties file.

You can mimic the search or indexing slow log level by setting appropriate threshold making "more verbose" loggers to be switched off. If for instance we want to simulate index.indexing.slowlog.level: INFO then all we need to do is to set index.indexing.slowlog.threshold.index.debug and index.indexing.slowlog.threshold.index.trace to -1.