analyzer

edit

The values of analyzed string fields are passed through an analyzer to convert the string into a stream of tokens or terms. For instance, the string "The quick Brown Foxes." may, depending on which analyzer is used, be analyzed to the tokens: quick, brown, fox. These are the actual terms that are indexed for the field, which makes it possible to search efficiently for individual words within big blobs of text.

This analysis process needs to happen not just at index time, but also at query time: the query string needs to be passed through the same (or a similar) analyzer so that the terms that it tries to find are in the same format as those that exist in the index.

Elasticsearch ships with a number of pre-defined analyzers, which can be used without further configuration. It also ships with many character filters, tokenizers, and Token Filters which can be combined to configure custom analyzers per index.

Analyzers can be specified per-query, per-field or per-index. At index time, Elasticsearch will look for an analyzer in this order:

  • The analyzer defined in the field mapping.
  • An analyzer named default in the index settings.
  • The standard analyzer.

At query time, there are a few more layers:

  • The analyzer defined in a full-text query.
  • The search_analyzer defined in the field mapping.
  • The analyzer defined in the field mapping.
  • An analyzer named default_search in the index settings.
  • An analyzer named default in the index settings.
  • The standard analyzer.

The easiest way to specify an analyzer for a particular field is to define it in the field mapping, as follows:

PUT my_index
{
  "mappings": {
    "my_type": {
      "properties": {
        "text": { 
          "type": "string",
          "fields": {
            "english": { 
              "type":     "string",
              "analyzer": "english"
            }
          }
        }
      }
    }
  }
}

GET my_index/_analyze?field=text 
{
  "text": "The quick Brown Foxes."
}

GET my_index/_analyze?field=text.english 
{
  "text": "The quick Brown Foxes."
}

The text field uses the default standard analyzer`.

The text.english multi-field uses the english analyzer, which removes stop words and applies stemming.

This returns the tokens: [ the, quick, brown, foxes ].

This returns the tokens: [ quick, brown, fox ].