Date field type

edit

JSON doesn’t have a date data type, so dates in Elasticsearch can either be:

  • strings containing formatted dates, e.g. "2015-01-01" or "2015/01/01 12:10:30".
  • a number representing milliseconds-since-the-epoch.
  • a number representing seconds-since-the-epoch (configuration).

Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.

Use the date_nanos field type if a nanosecond resolution is expected.

Queries on dates are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.

Dates will always be rendered as strings, even if they were initially supplied as a long in the JSON document.

Date formats can be customised, but if no format is specified then it uses the default:

    "strict_date_optional_time||epoch_millis"

This means that it will accept dates with optional timestamps, which conform to the formats supported by strict_date_optional_time or milliseconds-since-the-epoch.

For instance:

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        date: {
          type: 'date'
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 1,
  body: {
    date: '2015-01-01'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 2,
  body: {
    date: '2015-01-01T12:10:30Z'
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 3,
  body: {
    date: 1_420_070_400_001
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    sort: {
      date: 'asc'
    }
  }
)
puts response
{
	res, err := es.Indices.Create(
		"my-index-000001",
		es.Indices.Create.WithBody(strings.NewReader(`{
	  "mappings": {
	    "properties": {
	      "date": {
	        "type": "date"
	      }
	    }
	  }
	}`)),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-01-01"
	} `),
		es.Index.WithDocumentID("1"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": "2015-01-01T12:10:30Z"
	} `),
		es.Index.WithDocumentID("2"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Index(
		"my-index-000001",
		strings.NewReader(`{
	  "date": 1420070400001
	} `),
		es.Index.WithDocumentID("3"),
		es.Index.WithPretty(),
	)
	fmt.Println(res, err)
}

{
	res, err := es.Search(
		es.Search.WithIndex("my-index-000001"),
		es.Search.WithBody(strings.NewReader(`{
	  "sort": {
	    "date": "asc"
	  }
	}`)),
		es.Search.WithPretty(),
	)
	fmt.Println(res, err)
}
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date" 
      }
    }
  }
}

PUT my-index-000001/_doc/1
{ "date": "2015-01-01" } 

PUT my-index-000001/_doc/2
{ "date": "2015-01-01T12:10:30Z" } 

PUT my-index-000001/_doc/3
{ "date": 1420070400001 } 

GET my-index-000001/_search
{
  "sort": { "date": "asc"} 
}

The date field uses the default format.

This document uses a plain date.

This document includes a time.

This document uses milliseconds-since-the-epoch.

Note that the sort values that are returned are all in milliseconds-since-the-epoch.

Dates will accept numbers with a decimal point like {"date": 1618249875.123456} but there are some cases (#70085) where we’ll lose precision on those dates so they should be avoided.

Multiple date formats

edit

Multiple formats can be specified by separating them with || as a separator. Each format will be tried in turn until a matching format is found. The first format will be used to convert the milliseconds-since-the-epoch value back into a string.

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        date: {
          type: 'date',
          format: 'yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis'
        }
      }
    }
  }
)
puts response
res, err := es.Indices.Create(
	"my-index-000001",
	es.Indices.Create.WithBody(strings.NewReader(`{
	  "mappings": {
	    "properties": {
	      "date": {
	        "type": "date",
	        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
	      }
	    }
	  }
	}`)),
)
fmt.Println(res, err)
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
      }
    }
  }
}

Parameters for date fields

edit

The following parameters are accepted by date fields:

doc_values

Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false.

format

The date format(s) that can be parsed. Defaults to strict_date_optional_time||epoch_millis.

locale

The locale to use when parsing dates since months do not have the same names and/or abbreviations in all languages. The default is the ROOT locale,

ignore_malformed

If true, malformed numbers are ignored. If false (default), malformed numbers throw an exception and reject the whole document. Note that this cannot be set if the script parameter is used.

index

Should the field be quickly searchable? Accepts true (default) and false. Date fields that only have doc_values enabled can also be queried, albeit slower.

null_value

Accepts a date value in one of the configured format's as the field which is substituted for any explicit null values. Defaults to null, which means the field is treated as missing. Note that this cannot be set of the script parameter is used.

on_script_error

Defines what to do if the script defined by the script parameter throws an error at indexing time. Accepts fail (default), which will cause the entire document to be rejected, and continue, which will register the field in the document’s _ignored metadata field and continue indexing. This parameter can only be set if the script field is also set.

script

If this parameter is set, then the field will index values generated by this script, rather than reading the values directly from the source. If a value is set for this field on the input document, then the document will be rejected with an error. Scripts are in the same format as their runtime equivalent, and should emit long-valued timestamps.

store

Whether the field value should be stored and retrievable separately from the _source field. Accepts true or false (default).

meta

Metadata about the field.

Epoch seconds

edit

If you need to send dates as seconds-since-the-epoch then make sure the format lists epoch_second:

response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        date: {
          type: 'date',
          format: 'strict_date_optional_time||epoch_second'
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 'example',
  refresh: true,
  body: {
    date: 1_618_321_898
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    fields: [
      {
        field: 'date'
      }
    ],
    _source: false
  }
)
puts response
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "date": {
        "type":   "date",
        "format": "strict_date_optional_time||epoch_second"
      }
    }
  }
}

PUT my-index-000001/_doc/example?refresh
{ "date": 1618321898 }

POST my-index-000001/_search
{
  "fields": [ {"field": "date"}],
  "_source": false
}

Which will reply with a date like:

{
  "hits": {
    "hits": [
      {
        "_id": "example",
        "_index": "my-index-000001",
        "_score": 1.0,
        "fields": {
          "date": ["2021-04-13T13:51:38.000Z"]
        }
      }
    ]
  }
}

Synthetic _source

edit

Synthetic _source is Generally Available only for TSDB indices (indices that have index.mode set to time_series). For other indices synthetic _source is in technical preview. Features in technical preview may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

date fields support synthetic _source in their default configuration. Synthetic _source cannot be used together with copy_to, ignore_malformed set to true or with doc_values disabled.

Synthetic source always sorts date fields. For example:

response = client.indices.create(
  index: 'idx',
  body: {
    mappings: {
      _source: {
        mode: 'synthetic'
      },
      properties: {
        date: {
          type: 'date'
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'idx',
  id: 1,
  body: {
    date: [
      '2015-01-01T12:10:30Z',
      '2014-01-01T12:10:30Z'
    ]
  }
)
puts response
PUT idx
{
  "mappings": {
    "_source": { "mode": "synthetic" },
    "properties": {
      "date": { "type": "date" }
    }
  }
}
PUT idx/_doc/1
{
  "date": ["2015-01-01T12:10:30Z", "2014-01-01T12:10:30Z"]
}

Will become:

{
  "date": ["2014-01-01T12:10:30.000Z", "2015-01-01T12:10:30.000Z"]
}