Get component templates Added in 5.1.0

GET /_cat/component_templates/{name}

Get information about component templates in a cluster. Component templates are building blocks for constructing index templates that specify index mappings, settings, and aliases.

IMPORTANT: CAT APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get component template API.

Path parameters

  • name string Required

    The name of the component template. It accepts wildcard expressions. If it is omitted, all component templates are returned.

Query parameters

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • The period to wait for a connection to the master node.

Responses

GET /_cat/component_templates/{name}
curl \
 --request GET http://api.example.com/_cat/component_templates/{name}
Response examples (200)
[
  {
    "name": "string",
    "version": "string",
    "alias_count": "string",
    "mapping_count": "string",
    "settings_count": "string",
    "metadata_count": "string",
    "included_in": "string"
  }
]
































































































































Get index template information Added in 5.2.0

GET /_cat/templates/{name}

Get information about the index templates in a cluster. You can use index templates to apply index settings and field mappings to new indices at creation. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get index template API.

Path parameters

  • name string Required

    The name of the template to return. Accepts wildcard expressions. If omitted, all templates are returned.

Query parameters

  • local boolean

    If true, the request computes the list of selected nodes from the local cluster state. If false the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node.

  • Period to wait for a connection to the master node.

Responses

GET /_cat/templates/{name}
curl \
 --request GET http://api.example.com/_cat/templates/{name}
Response examples (200)
[
  {
    "name": "string",
    "index_patterns": "string",
    "order": "string",
    "version": "string",
    "composed_of": "string"
  }
]





























































Reroute the cluster Added in 5.0.0

POST /_cluster/reroute

Manually change the allocation of individual shards in the cluster. For example, a shard can be moved from one node to another explicitly, an allocation can be canceled, and an unassigned shard can be explicitly allocated to a specific node.

It is important to note that after processing any reroute commands Elasticsearch will perform rebalancing as normal (respecting the values of settings such as cluster.routing.rebalance.enable) in order to remain in a balanced state. For example, if the requested allocation includes moving a shard from node1 to node2 then this may cause a shard to be moved from node2 back to node1 to even things out.

The cluster can be set to disable allocations using the cluster.routing.allocation.enable setting. If allocations are disabled then the only allocations that will be performed are explicit ones given using the reroute command, and consequent allocations due to rebalancing.

The cluster will attempt to allocate a shard a maximum of index.allocation.max_retries times in a row (defaults to 5), before giving up and leaving the shard unallocated. This scenario can be caused by structural problems such as having an analyzer which refers to a stopwords file which doesn’t exist on all nodes.

Once the problem has been corrected, allocation can be manually retried by calling the reroute API with the ?retry_failed URI query parameter, which will attempt a single retry round for these shards.

Query parameters

  • dry_run boolean

    If true, then the request simulates the operation. It will calculate the result of applying the commands to the current cluster state and return the resulting cluster state after the commands (and rebalancing) have been applied; it will not actually perform the requested changes.

  • explain boolean

    If true, then the response contains an explanation of why the commands can or cannot run.

  • metric string | array[string]

    Limits the information returned to the specified metrics.

  • If true, then retries allocation of shards that are blocked due to too many subsequent allocation failures.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

application/json

Body

  • commands array[object]

    Defines the commands to perform.

    Hide commands attributes Show commands attributes object
    • cancel object

      Additional properties are allowed.

      Hide cancel attributes Show cancel attributes object
    • move object

      Additional properties are allowed.

      Hide move attributes Show move attributes object
    • Additional properties are allowed.

      Hide allocate_replica attributes Show allocate_replica attributes object
    • Additional properties are allowed.

      Hide allocate_stale_primary attributes Show allocate_stale_primary attributes object
      • index string Required
      • shard number Required
      • node string Required
      • accept_data_loss boolean Required

        If a node which has a copy of the data rejoins the cluster later on, that data will be deleted. To ensure that these implications are well-understood, this command requires the flag accept_data_loss to be explicitly set to true

    • Additional properties are allowed.

      Hide allocate_empty_primary attributes Show allocate_empty_primary attributes object
      • index string Required
      • shard number Required
      • node string Required
      • accept_data_loss boolean Required

        If a node which has a copy of the data rejoins the cluster later on, that data will be deleted. To ensure that these implications are well-understood, this command requires the flag accept_data_loss to be explicitly set to true

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • acknowledged boolean Required
    • explanations array[object]
      Hide explanations attributes Show explanations attributes object
    • state object

      There aren't any guarantees on the output/structure of the raw cluster state. Here you will find the internal representation of the cluster, which can differ from the external representation.

      Additional properties are allowed.

POST /_cluster/reroute
curl \
 --request POST http://api.example.com/_cluster/reroute \
 --header "Content-Type: application/json" \
 --data '"{\n  \"commands\": [\n    {\n      \"move\": {\n        \"index\": \"test\", \"shard\": 0,\n        \"from_node\": \"node1\", \"to_node\": \"node2\"\n      }\n    },\n    {\n      \"allocate_replica\": {\n        \"index\": \"test\", \"shard\": 1,\n        \"node\": \"node3\"\n      }\n    }\n  ]\n}"'
Request example
Run `POST /_cluster/reroute?metric=none` to changes the allocation of shards in a cluster.
{
  "commands": [
    {
      "move": {
        "index": "test", "shard": 0,
        "from_node": "node1", "to_node": "node2"
      }
    },
    {
      "allocate_replica": {
        "index": "test", "shard": 1,
        "node": "node3"
      }
    }
  ]
}
Response examples (200)
{
  "acknowledged": true,
  "explanations": [
    {
      "command": "string",
      "decisions": [
        {
          "decider": "string",
          "decision": "string",
          "explanation": "string"
        }
      ],
      "parameters": {
        "allow_primary": true,
        "index": "string",
        "node": "string",
        "shard": 42.0,
        "from_node": "string",
        "to_node": "string"
      }
    }
  ],
  "state": {}
}




















































































Get node statistics

GET /_nodes/{node_id}/stats/{metric}/{index_metric}

Get statistics for nodes in a cluster. By default, all stats are returned. You can limit the returned information by using metrics.

Path parameters

  • node_id string | array[string] Required

    Comma-separated list of node IDs or names used to limit returned information.

  • metric string | array[string] Required

    Limit the information returned to the specified metrics

  • index_metric string | array[string] Required

    Limit the information returned for indices metric to the specific index metrics. It can be used only if indices (or all) metric is specified.

Query parameters

  • completion_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.

  • fielddata_fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in fielddata statistics.

  • fields string | array[string]

    Comma-separated list or wildcard expressions of fields to include in the statistics.

  • groups boolean

    Comma-separated list of search groups to include in the search statistics.

  • If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).

  • level string

    Indicates whether statistics are aggregated at the cluster, index, or shard level.

    Values are cluster, indices, or shards.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

  • types array[string]

    A comma-separated list of document types for the indexing index metric.

  • If true, the response includes information from segments that are not loaded into memory.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • _nodes object

      Additional properties are allowed.

      Hide _nodes attributes Show _nodes attributes object
      • failures array[object]
        Hide failures attributes Show failures attributes object
        • type string Required

          The type of error

        • reason string

          A human-readable explanation of the error, in English.

        • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

        • Additional properties are allowed.

        • root_cause array[object]

          Additional properties are allowed.

        • suppressed array[object]

          Additional properties are allowed.

      • total number Required

        Total number of nodes selected by the request.

      • successful number Required

        Number of nodes that responded successfully to the request.

      • failed number Required

        Number of nodes that rejected the request or failed to respond. If this value is not 0, a reason for the rejection or failure is included in the response.

    • nodes object Required
      Hide nodes attribute Show nodes attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • Statistics about adaptive replica selection.

          Hide adaptive_selection attribute Show adaptive_selection attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attributes Show * attributes object
            • The exponentially weighted moving average queue size of search requests on the keyed node.

            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • The exponentially weighted moving average response time, in nanoseconds, of search requests on the keyed node.

            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • The exponentially weighted moving average service time, in nanoseconds, of search requests on the keyed node.

            • The number of outstanding search requests to the keyed node from the node these stats are for.

            • rank string

              The rank of this node; used for shard selection when routing search requests.

        • breakers object

          Statistics about the field data circuit breaker.

          Hide breakers attribute Show breakers attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attributes Show * attributes object
            • Estimated memory used for the operation.

            • Estimated memory used, in bytes, for the operation.

            • Memory limit for the circuit breaker.

            • Memory limit, in bytes, for the circuit breaker.

            • overhead number

              A constant that all estimates for the circuit breaker are multiplied with to calculate a final estimate.

            • tripped number

              Total number of times the circuit breaker has been triggered and prevented an out of memory error.

        • fs object

          Additional properties are allowed.

          Hide fs attributes Show fs attributes object
          • data array[object]

            List of all file stores.

            Additional properties are allowed.

          • Last time the file stores statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

          • total object

            Additional properties are allowed.

            Hide total attributes Show total attributes object
            • Total disk space available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • Total number of bytes available to this Java virtual machine on all file stores. Depending on OS or process level restrictions, this might appear less than free_in_bytes. This is the actual amount of free disk space the Elasticsearch node can utilise.

            • free string

              Total unallocated disk space in all file stores.

            • Total number of unallocated bytes in all file stores.

            • total string

              Total size of all file stores.

            • Total size of all file stores in bytes.

          • io_stats object

            Additional properties are allowed.

            Hide io_stats attributes Show io_stats attributes object
            • devices array[object]

              Array of disk metrics for each device that is backing an Elasticsearch data path. These disk metrics are probed periodically and averages between the last probe and the current probe are computed.

            • total object

              Additional properties are allowed.

        • host string
        • http object

          Additional properties are allowed.

          Hide http attributes Show http attributes object
          • Current number of open HTTP connections for the node.

          • Total number of HTTP connections opened for the node.

          • clients array[object]

            Information on current and recently-closed HTTP client connections. Clients that have been closed longer than the http.client_stats.closed_channels.max_age setting will not be represented here.

            Additional properties are allowed.

          • routes object Required Added in 8.12.0

            Detailed HTTP stats broken down by route

            Hide routes attribute Show routes attribute object
            • * object Additional properties

              Additional properties are allowed.

        • ingest object

          Additional properties are allowed.

          Hide ingest attributes Show ingest attributes object
          • Contains statistics about ingest pipelines for the node.

            Hide pipelines attribute Show pipelines attribute object
            • * object Additional properties

              Additional properties are allowed.

          • total object

            Additional properties are allowed.

            Hide total attributes Show total attributes object
            • count number Required

              Total number of documents ingested during the lifetime of this node.

            • current number Required

              Total number of documents currently being ingested.

            • failed number Required

              Total number of failed ingest operations during the lifetime of this node.

        • ip string | array[string]

          IP address and port for the node.

        • jvm object

          Additional properties are allowed.

          Hide jvm attributes Show jvm attributes object
          • Contains statistics about JVM buffer pools for the node.

            Hide buffer_pools attribute Show buffer_pools attribute object
            • * object Additional properties

              Additional properties are allowed.

          • classes object

            Additional properties are allowed.

            Hide classes attributes Show classes attributes object
          • gc object

            Additional properties are allowed.

            Hide gc attribute Show gc attribute object
            • Contains statistics about JVM garbage collectors for the node.

          • mem object

            Additional properties are allowed.

            Hide mem attributes Show mem attributes object
          • threads object

            Additional properties are allowed.

            Hide threads attributes Show threads attributes object
            • count number

              Number of active threads in use by JVM.

            • Highest number of threads used by JVM.

          • Last time JVM statistics were refreshed.

          • uptime string

            Human-readable JVM uptime. Only returned if the human query parameter is true.

          • JVM uptime in milliseconds.

        • name string
        • os object

          Additional properties are allowed.

          Hide os attributes Show os attributes object
          • cpu object

            Additional properties are allowed.

            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • mem object

            Additional properties are allowed.

          • swap object

            Additional properties are allowed.

            Hide swap attributes Show swap attributes object
          • cgroup object

            Additional properties are allowed.

            Hide cgroup attributes Show cgroup attributes object
            • cpuacct object

              Additional properties are allowed.

            • cpu object

              Additional properties are allowed.

            • memory object

              Additional properties are allowed.

        • process object

          Additional properties are allowed.

          Hide process attributes Show process attributes object
          • cpu object

            Additional properties are allowed.

            Hide cpu attributes Show cpu attributes object
            • percent number
            • sys string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • total string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • user string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • mem object

            Additional properties are allowed.

            Hide mem attributes Show mem attributes object
          • Number of opened file descriptors associated with the current or -1 if not supported.

          • Maximum number of file descriptors allowed on the system, or -1 if not supported.

          • Last time the statistics were refreshed. Recorded in milliseconds since the Unix Epoch.

        • roles array[string]
          • @doc_id node-roles

          Values are master, data, data_cold, data_content, data_frozen, data_hot, data_warm, client, ingest, ml, voting_only, transform, remote_cluster_client, or coordinating_only.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • Total number of times the script cache has evicted old data.

          • Total number of inline script compilations performed by the node.

          • Contains this recent history of script compilations.

            Hide compilations_history attribute Show compilations_history attribute object
            • * number Additional properties
          • Total number of times the script compilation circuit breaker has limited inline script compilations.

          • contexts array[object]

            Additional properties are allowed.

        • Statistics about each thread pool, including current size, queue and rejected tasks.

          Hide thread_pool attribute Show thread_pool attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attributes Show * attributes object
            • active number

              Number of active threads in the thread pool.

            • Number of tasks completed by the thread pool executor.

            • largest number

              Highest number of active threads in the thread pool.

            • queue number

              Number of tasks in queue for the thread pool.

            • rejected number

              Number of tasks rejected by the thread pool executor.

            • threads number

              Number of threads in the thread pool.

        • Additional properties are allowed.

          Hide transport attributes Show transport attributes object
          • The distribution of the time spent handling each inbound message on a transport thread, represented as a histogram.

            Additional properties are allowed.

          • The distribution of the time spent sending each outbound transport message on a transport thread, represented as a histogram.

            Additional properties are allowed.

          • rx_count number

            Total number of RX (receive) packets received by the node during internal cluster communication.

          • rx_size string

            Size of RX packets received by the node during internal cluster communication.

          • Size, in bytes, of RX packets received by the node during internal cluster communication.

          • Current number of inbound TCP connections used for internal communication between nodes.

          • tx_count number

            Total number of TX (transmit) packets sent by the node during internal cluster communication.

          • tx_size string

            Size of TX packets sent by the node during internal cluster communication.

          • Size, in bytes, of TX packets sent by the node during internal cluster communication.

          • The cumulative number of outbound transport connections that this node has opened since it started. Each transport connection may comprise multiple TCP connections but is only counted once in this statistic. Transport connections are typically long-lived so this statistic should remain constant in a stable cluster.

        • Contains a list of attributes for the node.

          Hide attributes attribute Show attributes attribute object
          • * string Additional properties
        • Additional properties are allowed.

          Hide discovery attributes Show discovery attributes object
          • Additional properties are allowed.

            Hide cluster_state_queue attributes Show cluster_state_queue attributes object
            • total number

              Total number of cluster states in queue.

            • pending number

              Number of pending cluster states in queue.

            • Number of committed cluster states in queue.

          • Additional properties are allowed.

            Hide published_cluster_states attributes Show published_cluster_states attributes object
          • Contains low-level statistics about how long various activities took during cluster state updates while the node was the elected master. Omitted if the node is not master-eligible. Every field whose name ends in _time within this object is also represented as a raw number of milliseconds in a field whose name ends in _time_millis. The human-readable fields with a _time suffix are only returned if requested with the ?human=true query parameter.

            Hide cluster_state_update attribute Show cluster_state_update attribute object
            • * object Additional properties

              Additional properties are allowed.

          • Additional properties are allowed.

            Hide serialized_cluster_states attributes Show serialized_cluster_states attributes object
            • Additional properties are allowed.

            • diffs object

              Additional properties are allowed.

          • Additional properties are allowed.

            Hide cluster_applier_stats attribute Show cluster_applier_stats attribute object
        • Additional properties are allowed.

          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object

            Additional properties are allowed.

            Hide memory attributes Show memory attributes object
            • Configured memory limit, in bytes, for the indexing requests. Replica requests have an automatic limit that is 1.5x this value.

            • current object

              Additional properties are allowed.

            • total object

              Additional properties are allowed.

        • indices object

          Additional properties are allowed.

          Hide indices attributes Show indices attributes object
GET /_nodes/{node_id}/stats/{metric}/{index_metric}
curl \
 --request GET http://api.example.com/_nodes/{node_id}/stats/{metric}/{index_metric}
Response examples (200)
{
  "_nodes": {
    "failures": [
      {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      }
    ],
    "total": 42.0,
    "successful": 42.0,
    "failed": 42.0
  },
  "cluster_name": "string",
  "nodes": {
    "additionalProperty1": {
      "adaptive_selection": {
        "additionalProperty1": {
          "avg_queue_size": 42.0,
          "avg_response_time": "string",
          "avg_response_time_ns": 42.0,
          "avg_service_time": "string",
          "avg_service_time_ns": 42.0,
          "outgoing_searches": 42.0,
          "rank": "string"
        },
        "additionalProperty2": {
          "avg_queue_size": 42.0,
          "avg_response_time": "string",
          "avg_response_time_ns": 42.0,
          "avg_service_time": "string",
          "avg_service_time_ns": 42.0,
          "outgoing_searches": 42.0,
          "rank": "string"
        }
      },
      "breakers": {
        "additionalProperty1": {
          "estimated_size": "string",
          "estimated_size_in_bytes": 42.0,
          "limit_size": "string",
          "limit_size_in_bytes": 42.0,
          "overhead": 42.0,
          "tripped": 42.0
        },
        "additionalProperty2": {
          "estimated_size": "string",
          "estimated_size_in_bytes": 42.0,
          "limit_size": "string",
          "limit_size_in_bytes": 42.0,
          "overhead": 42.0,
          "tripped": 42.0
        }
      },
      "fs": {
        "data": [
          {}
        ],
        "timestamp": 42.0,
        "total": {
          "available": "string",
          "available_in_bytes": 42.0,
          "free": "string",
          "free_in_bytes": 42.0,
          "total": "string",
          "total_in_bytes": 42.0
        },
        "io_stats": {
          "devices": [
            {}
          ],
          "total": {}
        }
      },
      "host": "string",
      "http": {
        "current_open": 42.0,
        "total_opened": 42.0,
        "clients": [
          {}
        ],
        "routes": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      },
      "ingest": {
        "pipelines": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "total": {
          "count": 42.0,
          "current": 42.0,
          "failed": 42.0
        }
      },
      "ip": "string",
      "jvm": {
        "buffer_pools": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "classes": {
          "current_loaded_count": 42.0,
          "total_loaded_count": 42.0,
          "total_unloaded_count": 42.0
        },
        "gc": {
          "collectors": {}
        },
        "mem": {
          "heap_used_in_bytes": 42.0,
          "heap_used_percent": 42.0,
          "heap_committed_in_bytes": 42.0,
          "heap_max_in_bytes": 42.0,
          "non_heap_used_in_bytes": 42.0,
          "non_heap_committed_in_bytes": 42.0,
          "pools": {}
        },
        "threads": {
          "count": 42.0,
          "peak_count": 42.0
        },
        "timestamp": 42.0,
        "uptime": "string",
        "uptime_in_millis": 42.0
      },
      "name": "string",
      "os": {
        "cpu": {
          "percent": 42.0,
          "sys": "string",
          "total": "string",
          "user": "string",
          "load_average": {}
        },
        "": {},
        "swap": {
          "adjusted_total_in_bytes": 42.0,
          "resident": "string",
          "resident_in_bytes": 42.0,
          "share": "string",
          "share_in_bytes": 42.0,
          "total_virtual": "string",
          "total_virtual_in_bytes": 42.0,
          "total_in_bytes": 42.0,
          "free_in_bytes": 42.0,
          "used_in_bytes": 42.0
        },
        "cgroup": {
          "cpuacct": {},
          "cpu": {},
          "memory": {}
        },
        "timestamp": 42.0
      },
      "process": {
        "cpu": {
          "percent": 42.0,
          "sys": "string",
          "total": "string",
          "user": "string",
          "load_average": {}
        },
        "mem": {
          "adjusted_total_in_bytes": 42.0,
          "resident": "string",
          "resident_in_bytes": 42.0,
          "share": "string",
          "share_in_bytes": 42.0,
          "total_virtual": "string",
          "total_virtual_in_bytes": 42.0,
          "total_in_bytes": 42.0,
          "free_in_bytes": 42.0,
          "used_in_bytes": 42.0
        },
        "open_file_descriptors": 42.0,
        "max_file_descriptors": 42.0,
        "timestamp": 42.0
      },
      "roles": [
        "master"
      ],
      "script": {
        "cache_evictions": 42.0,
        "compilations": 42.0,
        "compilations_history": {
          "additionalProperty1": 42.0,
          "additionalProperty2": 42.0
        },
        "compilation_limit_triggered": 42.0,
        "contexts": [
          {}
        ]
      },
      "script_cache": {},
      "thread_pool": {
        "additionalProperty1": {
          "active": 42.0,
          "completed": 42.0,
          "largest": 42.0,
          "queue": 42.0,
          "rejected": 42.0,
          "threads": 42.0
        },
        "additionalProperty2": {
          "active": 42.0,
          "completed": 42.0,
          "largest": 42.0,
          "queue": 42.0,
          "rejected": 42.0,
          "threads": 42.0
        }
      },
      "timestamp": 42.0,
      "transport": {
        "inbound_handling_time_histogram": [
          {}
        ],
        "outbound_handling_time_histogram": [
          {}
        ],
        "rx_count": 42.0,
        "rx_size": "string",
        "rx_size_in_bytes": 42.0,
        "server_open": 42.0,
        "tx_count": 42.0,
        "tx_size": "string",
        "tx_size_in_bytes": 42.0,
        "total_outbound_connections": 42.0
      },
      "transport_address": "string",
      "attributes": {
        "additionalProperty1": "string",
        "additionalProperty2": "string"
      },
      "discovery": {
        "cluster_state_queue": {
          "total": 42.0,
          "pending": 42.0,
          "committed": 42.0
        },
        "published_cluster_states": {
          "full_states": 42.0,
          "incompatible_diffs": 42.0,
          "compatible_diffs": 42.0
        },
        "cluster_state_update": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "serialized_cluster_states": {
          "full_states": {},
          "diffs": {}
        },
        "cluster_applier_stats": {
          "recordings": [
            {}
          ]
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit_in_bytes": 42.0,
          "current": {},
          "total": {}
        }
      },
      "indices": {
        "commit": {
          "generation": 42.0,
          "id": "string",
          "num_docs": 42.0,
          "user_data": {}
        },
        "completion": {
          "size_in_bytes": 42.0,
          "fields": {}
        },
        "docs": {
          "count": 42.0,
          "deleted": 42.0
        },
        "fielddata": {
          "evictions": 42.0,
          "memory_size_in_bytes": 42.0,
          "fields": {}
        },
        "flush": {
          "periodic": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "get": {
          "current": 42.0,
          "exists_time": "string",
          "exists_total": 42.0,
          "missing_time": "string",
          "missing_total": 42.0,
          "time": "string",
          "total": 42.0
        },
        "indexing": {
          "index_current": 42.0,
          "delete_current": 42.0,
          "delete_time": "string",
          "delete_total": 42.0,
          "is_throttled": true,
          "noop_update_total": 42.0,
          "throttle_time": "string",
          "index_time": "string",
          "index_total": 42.0,
          "index_failed": 42.0,
          "types": {},
          "write_load": 42.0
        },
        "mappings": {
          "total_count": 42.0,
          "total_estimated_overhead_in_bytes": 42.0
        },
        "merges": {
          "current": 42.0,
          "current_docs": 42.0,
          "current_size": "string",
          "current_size_in_bytes": 42.0,
          "total": 42.0,
          "total_auto_throttle": "string",
          "total_auto_throttle_in_bytes": 42.0,
          "total_docs": 42.0,
          "total_size": "string",
          "total_size_in_bytes": 42.0,
          "total_stopped_time": "string",
          "total_throttled_time": "string",
          "total_time": "string"
        },
        "shard_path": {
          "data_path": "string",
          "is_custom_data_path": true,
          "state_path": "string"
        },
        "query_cache": {
          "cache_count": 42.0,
          "cache_size": 42.0,
          "evictions": 42.0,
          "hit_count": 42.0,
          "memory_size_in_bytes": 42.0,
          "miss_count": 42.0,
          "total_count": 42.0
        },
        "recovery": {
          "current_as_source": 42.0,
          "current_as_target": 42.0,
          "throttle_time": "string"
        },
        "refresh": {
          "external_total": 42.0,
          "listeners": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "request_cache": {
          "evictions": 42.0,
          "hit_count": 42.0,
          "memory_size": "string",
          "memory_size_in_bytes": 42.0,
          "miss_count": 42.0
        },
        "retention_leases": {
          "primary_term": 42.0,
          "version": 42.0,
          "leases": [
            {}
          ]
        },
        "routing": {
          "node": "string",
          "primary": true,
          "state": "UNASSIGNED"
        },
        "search": {
          "fetch_current": 42.0,
          "fetch_time": "string",
          "fetch_total": 42.0,
          "open_contexts": 42.0,
          "query_current": 42.0,
          "query_time": "string",
          "query_total": 42.0,
          "scroll_current": 42.0,
          "scroll_time": "string",
          "scroll_total": 42.0,
          "suggest_current": 42.0,
          "suggest_time": "string",
          "suggest_total": 42.0,
          "groups": {}
        },
        "segments": {
          "count": 42.0,
          "doc_values_memory_in_bytes": 42.0,
          "file_sizes": {},
          "fixed_bit_set_memory_in_bytes": 42.0,
          "index_writer_max_memory_in_bytes": 42.0,
          "index_writer_memory_in_bytes": 42.0,
          "max_unsafe_auto_id_timestamp": 42.0,
          "memory_in_bytes": 42.0,
          "norms_memory_in_bytes": 42.0,
          "points_memory_in_bytes": 42.0,
          "stored_fields_memory_in_bytes": 42.0,
          "terms_memory_in_bytes": 42.0,
          "term_vectors_memory_in_bytes": 42.0,
          "version_map_memory_in_bytes": 42.0
        },
        "seq_no": {
          "global_checkpoint": 42.0,
          "local_checkpoint": 42.0,
          "max_seq_no": 42.0
        },
        "store": {
          "size_in_bytes": 42.0,
          "reserved_in_bytes": 42.0,
          "total_data_set_size_in_bytes": 42.0
        },
        "translog": {
          "earliest_last_modified_age": 42.0,
          "operations": 42.0,
          "size": "string",
          "size_in_bytes": 42.0,
          "uncommitted_operations": 42.0,
          "uncommitted_size": "string",
          "uncommitted_size_in_bytes": 42.0
        },
        "warmer": {
          "current": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "bulk": {
          "total_operations": 42.0,
          "total_time": "string",
          "total_size_in_bytes": 42.0,
          "avg_time": "string",
          "avg_size_in_bytes": 42.0
        },
        "shards": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "shard_stats": {
          "total_count": 42.0
        },
        "additionalProperty1": {
          "primaries": {},
          "shards": {},
          "total": {},
          "uuid": "string",
          "health": "green",
          "status": "open"
        },
        "additionalProperty2": {
          "primaries": {},
          "shards": {},
          "total": {},
          "uuid": "string",
          "health": "green",
          "status": "open"
        }
      }
    },
    "additionalProperty2": {
      "adaptive_selection": {
        "additionalProperty1": {
          "avg_queue_size": 42.0,
          "avg_response_time": "string",
          "avg_response_time_ns": 42.0,
          "avg_service_time": "string",
          "avg_service_time_ns": 42.0,
          "outgoing_searches": 42.0,
          "rank": "string"
        },
        "additionalProperty2": {
          "avg_queue_size": 42.0,
          "avg_response_time": "string",
          "avg_response_time_ns": 42.0,
          "avg_service_time": "string",
          "avg_service_time_ns": 42.0,
          "outgoing_searches": 42.0,
          "rank": "string"
        }
      },
      "breakers": {
        "additionalProperty1": {
          "estimated_size": "string",
          "estimated_size_in_bytes": 42.0,
          "limit_size": "string",
          "limit_size_in_bytes": 42.0,
          "overhead": 42.0,
          "tripped": 42.0
        },
        "additionalProperty2": {
          "estimated_size": "string",
          "estimated_size_in_bytes": 42.0,
          "limit_size": "string",
          "limit_size_in_bytes": 42.0,
          "overhead": 42.0,
          "tripped": 42.0
        }
      },
      "fs": {
        "data": [
          {}
        ],
        "timestamp": 42.0,
        "total": {
          "available": "string",
          "available_in_bytes": 42.0,
          "free": "string",
          "free_in_bytes": 42.0,
          "total": "string",
          "total_in_bytes": 42.0
        },
        "io_stats": {
          "devices": [
            {}
          ],
          "total": {}
        }
      },
      "host": "string",
      "http": {
        "current_open": 42.0,
        "total_opened": 42.0,
        "clients": [
          {}
        ],
        "routes": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      },
      "ingest": {
        "pipelines": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "total": {
          "count": 42.0,
          "current": 42.0,
          "failed": 42.0
        }
      },
      "ip": "string",
      "jvm": {
        "buffer_pools": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "classes": {
          "current_loaded_count": 42.0,
          "total_loaded_count": 42.0,
          "total_unloaded_count": 42.0
        },
        "gc": {
          "collectors": {}
        },
        "mem": {
          "heap_used_in_bytes": 42.0,
          "heap_used_percent": 42.0,
          "heap_committed_in_bytes": 42.0,
          "heap_max_in_bytes": 42.0,
          "non_heap_used_in_bytes": 42.0,
          "non_heap_committed_in_bytes": 42.0,
          "pools": {}
        },
        "threads": {
          "count": 42.0,
          "peak_count": 42.0
        },
        "timestamp": 42.0,
        "uptime": "string",
        "uptime_in_millis": 42.0
      },
      "name": "string",
      "os": {
        "cpu": {
          "percent": 42.0,
          "sys": "string",
          "total": "string",
          "user": "string",
          "load_average": {}
        },
        "": {},
        "swap": {
          "adjusted_total_in_bytes": 42.0,
          "resident": "string",
          "resident_in_bytes": 42.0,
          "share": "string",
          "share_in_bytes": 42.0,
          "total_virtual": "string",
          "total_virtual_in_bytes": 42.0,
          "total_in_bytes": 42.0,
          "free_in_bytes": 42.0,
          "used_in_bytes": 42.0
        },
        "cgroup": {
          "cpuacct": {},
          "cpu": {},
          "memory": {}
        },
        "timestamp": 42.0
      },
      "process": {
        "cpu": {
          "percent": 42.0,
          "sys": "string",
          "total": "string",
          "user": "string",
          "load_average": {}
        },
        "mem": {
          "adjusted_total_in_bytes": 42.0,
          "resident": "string",
          "resident_in_bytes": 42.0,
          "share": "string",
          "share_in_bytes": 42.0,
          "total_virtual": "string",
          "total_virtual_in_bytes": 42.0,
          "total_in_bytes": 42.0,
          "free_in_bytes": 42.0,
          "used_in_bytes": 42.0
        },
        "open_file_descriptors": 42.0,
        "max_file_descriptors": 42.0,
        "timestamp": 42.0
      },
      "roles": [
        "master"
      ],
      "script": {
        "cache_evictions": 42.0,
        "compilations": 42.0,
        "compilations_history": {
          "additionalProperty1": 42.0,
          "additionalProperty2": 42.0
        },
        "compilation_limit_triggered": 42.0,
        "contexts": [
          {}
        ]
      },
      "script_cache": {},
      "thread_pool": {
        "additionalProperty1": {
          "active": 42.0,
          "completed": 42.0,
          "largest": 42.0,
          "queue": 42.0,
          "rejected": 42.0,
          "threads": 42.0
        },
        "additionalProperty2": {
          "active": 42.0,
          "completed": 42.0,
          "largest": 42.0,
          "queue": 42.0,
          "rejected": 42.0,
          "threads": 42.0
        }
      },
      "timestamp": 42.0,
      "transport": {
        "inbound_handling_time_histogram": [
          {}
        ],
        "outbound_handling_time_histogram": [
          {}
        ],
        "rx_count": 42.0,
        "rx_size": "string",
        "rx_size_in_bytes": 42.0,
        "server_open": 42.0,
        "tx_count": 42.0,
        "tx_size": "string",
        "tx_size_in_bytes": 42.0,
        "total_outbound_connections": 42.0
      },
      "transport_address": "string",
      "attributes": {
        "additionalProperty1": "string",
        "additionalProperty2": "string"
      },
      "discovery": {
        "cluster_state_queue": {
          "total": 42.0,
          "pending": 42.0,
          "committed": 42.0
        },
        "published_cluster_states": {
          "full_states": 42.0,
          "incompatible_diffs": 42.0,
          "compatible_diffs": 42.0
        },
        "cluster_state_update": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "serialized_cluster_states": {
          "full_states": {},
          "diffs": {}
        },
        "cluster_applier_stats": {
          "recordings": [
            {}
          ]
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit_in_bytes": 42.0,
          "current": {},
          "total": {}
        }
      },
      "indices": {
        "commit": {
          "generation": 42.0,
          "id": "string",
          "num_docs": 42.0,
          "user_data": {}
        },
        "completion": {
          "size_in_bytes": 42.0,
          "fields": {}
        },
        "docs": {
          "count": 42.0,
          "deleted": 42.0
        },
        "fielddata": {
          "evictions": 42.0,
          "memory_size_in_bytes": 42.0,
          "fields": {}
        },
        "flush": {
          "periodic": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "get": {
          "current": 42.0,
          "exists_time": "string",
          "exists_total": 42.0,
          "missing_time": "string",
          "missing_total": 42.0,
          "time": "string",
          "total": 42.0
        },
        "indexing": {
          "index_current": 42.0,
          "delete_current": 42.0,
          "delete_time": "string",
          "delete_total": 42.0,
          "is_throttled": true,
          "noop_update_total": 42.0,
          "throttle_time": "string",
          "index_time": "string",
          "index_total": 42.0,
          "index_failed": 42.0,
          "types": {},
          "write_load": 42.0
        },
        "mappings": {
          "total_count": 42.0,
          "total_estimated_overhead_in_bytes": 42.0
        },
        "merges": {
          "current": 42.0,
          "current_docs": 42.0,
          "current_size": "string",
          "current_size_in_bytes": 42.0,
          "total": 42.0,
          "total_auto_throttle": "string",
          "total_auto_throttle_in_bytes": 42.0,
          "total_docs": 42.0,
          "total_size": "string",
          "total_size_in_bytes": 42.0,
          "total_stopped_time": "string",
          "total_throttled_time": "string",
          "total_time": "string"
        },
        "shard_path": {
          "data_path": "string",
          "is_custom_data_path": true,
          "state_path": "string"
        },
        "query_cache": {
          "cache_count": 42.0,
          "cache_size": 42.0,
          "evictions": 42.0,
          "hit_count": 42.0,
          "memory_size_in_bytes": 42.0,
          "miss_count": 42.0,
          "total_count": 42.0
        },
        "recovery": {
          "current_as_source": 42.0,
          "current_as_target": 42.0,
          "throttle_time": "string"
        },
        "refresh": {
          "external_total": 42.0,
          "listeners": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "request_cache": {
          "evictions": 42.0,
          "hit_count": 42.0,
          "memory_size": "string",
          "memory_size_in_bytes": 42.0,
          "miss_count": 42.0
        },
        "retention_leases": {
          "primary_term": 42.0,
          "version": 42.0,
          "leases": [
            {}
          ]
        },
        "routing": {
          "node": "string",
          "primary": true,
          "state": "UNASSIGNED"
        },
        "search": {
          "fetch_current": 42.0,
          "fetch_time": "string",
          "fetch_total": 42.0,
          "open_contexts": 42.0,
          "query_current": 42.0,
          "query_time": "string",
          "query_total": 42.0,
          "scroll_current": 42.0,
          "scroll_time": "string",
          "scroll_total": 42.0,
          "suggest_current": 42.0,
          "suggest_time": "string",
          "suggest_total": 42.0,
          "groups": {}
        },
        "segments": {
          "count": 42.0,
          "doc_values_memory_in_bytes": 42.0,
          "file_sizes": {},
          "fixed_bit_set_memory_in_bytes": 42.0,
          "index_writer_max_memory_in_bytes": 42.0,
          "index_writer_memory_in_bytes": 42.0,
          "max_unsafe_auto_id_timestamp": 42.0,
          "memory_in_bytes": 42.0,
          "norms_memory_in_bytes": 42.0,
          "points_memory_in_bytes": 42.0,
          "stored_fields_memory_in_bytes": 42.0,
          "terms_memory_in_bytes": 42.0,
          "term_vectors_memory_in_bytes": 42.0,
          "version_map_memory_in_bytes": 42.0
        },
        "seq_no": {
          "global_checkpoint": 42.0,
          "local_checkpoint": 42.0,
          "max_seq_no": 42.0
        },
        "store": {
          "size_in_bytes": 42.0,
          "reserved_in_bytes": 42.0,
          "total_data_set_size_in_bytes": 42.0
        },
        "translog": {
          "earliest_last_modified_age": 42.0,
          "operations": 42.0,
          "size": "string",
          "size_in_bytes": 42.0,
          "uncommitted_operations": 42.0,
          "uncommitted_size": "string",
          "uncommitted_size_in_bytes": 42.0
        },
        "warmer": {
          "current": 42.0,
          "total": 42.0,
          "total_time": "string"
        },
        "bulk": {
          "total_operations": 42.0,
          "total_time": "string",
          "total_size_in_bytes": 42.0,
          "avg_time": "string",
          "avg_size_in_bytes": 42.0
        },
        "shards": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "shard_stats": {
          "total_count": 42.0
        },
        "additionalProperty1": {
          "primaries": {},
          "shards": {},
          "total": {},
          "uuid": "string",
          "health": "green",
          "status": "open"
        },
        "additionalProperty2": {
          "primaries": {},
          "shards": {},
          "total": {},
          "uuid": "string",
          "health": "green",
          "status": "open"
        }
      }
    }
  }
}










































































































Update the connector error field Technical preview

PUT /_connector/{connector_id}/_error

Set the error field for the connector. If the error provided in the request body is non-null, the connector’s status is updated to error. Otherwise, if the error is reset to null, the connector status is updated to connected.

Path parameters

  • connector_id string Required

    The unique identifier of the connector to be updated

application/json

Body Required

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • result string Required

      Values are created, updated, deleted, not_found, or noop.

PUT /_connector/{connector_id}/_error
curl \
 --request PUT http://api.example.com/_connector/{connector_id}/_error \
 --header "Content-Type: application/json" \
 --data '"{\n    \"error\": \"Houston, we have a problem!\"\n}"'
Request example
{
    "error": "Houston, we have a problem!"
}
Response examples (200)
{
  "result": "updated"
}















































































































































































































































































Get term vector information

POST /{index}/_termvectors

Get information and statistics about terms in the fields of a particular document.

You can retrieve term vectors for documents stored in the index or for artificial documents passed in the body of the request. You can specify the fields you are interested in through the fields parameter or by adding the fields to the request body. For example:

GET /my-index-000001/_termvectors/1?fields=message

Fields can be specified using wildcards, similar to the multi match query.

Term vectors are real-time by default, not near real-time. This can be changed by setting realtime parameter to false.

You can request three types of values: term information, term statistics, and field statistics. By default, all term information and field statistics are returned for all fields but term statistics are excluded.

Term information

  • term frequency in the field (always returned)
  • term positions (positions: true)
  • start and end offsets (offsets: true)
  • term payloads (payloads: true), as base64 encoded bytes

If the requested information wasn't stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.


Start and end offsets assume UTF-16 encoding is being used. If you want to use these offsets in order to get the original text that produced this token, you should make sure that the string you are taking a sub-string of is also encoded using UTF-16.

Behaviour

The term and field statistics are not accurate. Deleted documents are not taken into account. The information is only retrieved for the shard the requested document resides in. The term and field statistics are therefore only useful as relative measures whereas the absolute numbers have no meaning in this context. By default, when requesting term vectors of artificial documents, a shard to get the statistics from is randomly selected. Use routing only to hit a particular shard.

Path parameters

  • index string Required

    The name of the index that contains the document.

Query parameters

  • fields string | array[string]

    A comma-separated list or wildcard expressions of fields to include in the statistics. It is used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters.

  • If true, the response includes:

    • The document count (how many documents contain this field).
    • The sum of document frequencies (the sum of document frequencies for all terms in this field).
    • The sum of total term frequencies (the sum of total term frequencies of each term in this field).
  • offsets boolean

    If true, the response includes term offsets.

  • payloads boolean

    If true, the response includes term payloads.

  • positions boolean

    If true, the response includes term positions.

  • The node or shard the operation should be performed on. It is random by default.

  • realtime boolean

    If true, the request is real-time as opposed to near-real-time.

  • routing string

    A custom value that is used to route operations to a specific shard.

  • If true, the response includes:

    • The total term frequency (how often a term occurs in all documents).
    • The document frequency (the number of documents containing the current term).

    By default these values are not returned since term statistics can have a serious performance impact.

  • version number

    If true, returns the document version as part of a hit.

  • The version type.

    Values are internal, external, external_gte, or force.

application/json

Body

  • doc object

    An artificial document (a document not present in the index) for which you want to retrieve term vectors.

    Additional properties are allowed.

  • filter object

    Additional properties are allowed.

    Hide filter attributes Show filter attributes object
    • Ignore words which occur in more than this many docs. Defaults to unbounded.

    • The maximum number of terms that must be returned per field.

    • Ignore words with more than this frequency in the source doc. It defaults to unbounded.

    • The maximum word length above which words will be ignored. Defaults to unbounded.

    • Ignore terms which do not occur in at least this many docs.

    • Ignore words with less than this frequency in the source doc.

    • The minimum word length below which words will be ignored.

  • Override the default per-field analyzer. This is useful in order to generate term vectors in any fashion, especially when using artificial documents. When providing an analyzer for a field that already stores term vectors, the term vectors will be regenerated.

    Hide per_field_analyzer attribute Show per_field_analyzer attribute object
    • * string Additional properties

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • found boolean Required
    • _id string
    • _index string Required
    • Hide term_vectors attribute Show term_vectors attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
    • took number Required
    • _version number Required
POST /{index}/_termvectors
curl \
 --request POST http://api.example.com/{index}/_termvectors \
 --header "Content-Type: application/json" \
 --data '"{\n  \"fields\" : [\"text\"],\n  \"offsets\" : true,\n  \"payloads\" : true,\n  \"positions\" : true,\n  \"term_statistics\" : true,\n  \"field_statistics\" : true\n}"'
Run `GET /my-index-000001/_termvectors/1` to return all information and statistics for field `text` in document 1.
{
  "fields" : ["text"],
  "offsets" : true,
  "payloads" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}
Run `GET /my-index-000001/_termvectors/1` to set per-field analyzers. A different analyzer than the one at the field may be provided by using the `per_field_analyzer` parameter.
{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "test test test"
  },
  "fields": ["fullname"],
  "per_field_analyzer" : {
    "fullname": "keyword"
  }
}
Run `GET /imdb/_termvectors` to filter the terms returned based on their tf-idf scores. It returns the three most "interesting" keywords from the artificial document having the given "plot" field value. Notice that the keyword "Tony" or any stop words are not part of the response, as their tf-idf must be too low.
{
  "doc": {
    "plot": "When wealthy industrialist Tony Stark is forced to build an armored suit after a life-threatening incident, he ultimately decides to use its technology to fight against evil."
  },
  "term_statistics": true,
  "field_statistics": true,
  "positions": false,
  "offsets": false,
  "filter": {
    "max_num_terms": 3,
    "min_term_freq": 1,
    "min_doc_freq": 1
  }
}
Run `GET /my-index-000001/_termvectors/1`. Term vectors which are not explicitly stored in the index are automatically computed on the fly. This request returns all information and statistics for the fields in document 1, even though the terms haven't been explicitly stored in the index. Note that for the field text, the terms are not regenerated.
{
  "fields" : ["text", "some_field_without_term_vectors"],
  "offsets" : true,
  "positions" : true,
  "term_statistics" : true,
  "field_statistics" : true
}
Run `GET /my-index-000001/_termvectors`. Term vectors can be generated for artificial documents, that is for documents not present in the index. If dynamic mapping is turned on (default), the document fields not in the original mapping will be dynamically created.
{
  "doc" : {
    "fullname" : "John Doe",
    "text" : "test test test"
  }
}
Response examples (200)
A successful response from `GET /my-index-000001/_termvectors/1`.
{
  "_index": "my-index-000001",
  "_id": "1",
  "_version": 1,
  "found": true,
  "took": 6,
  "term_vectors": {
    "text": {
      "field_statistics": {
        "sum_doc_freq": 4,
        "doc_count": 2,
        "sum_ttf": 6
      },
      "terms": {
        "test": {
          "doc_freq": 2,
          "ttf": 4,
          "term_freq": 3,
          "tokens": [
            {
              "position": 0,
              "start_offset": 0,
              "end_offset": 4,
              "payload": "d29yZA=="
            },
            {
              "position": 1,
              "start_offset": 5,
              "end_offset": 9,
              "payload": "d29yZA=="
            },
            {
              "position": 2,
              "start_offset": 10,
              "end_offset": 14,
              "payload": "d29yZA=="
            }
          ]
        }
      }
    }
  }
}
A successful response from `GET /my-index-000001/_termvectors` with `per_field_analyzer` in the request body.
{
  "_index": "my-index-000001",
  "_version": 0,
  "found": true,
  "took": 6,
  "term_vectors": {
    "fullname": {
      "field_statistics": {
          "sum_doc_freq": 2,
          "doc_count": 4,
          "sum_ttf": 4
      },
      "terms": {
          "John Doe": {
            "term_freq": 1,
            "tokens": [
                {
                  "position": 0,
                  "start_offset": 0,
                  "end_offset": 8
                }
            ]
          }
      }
    }
  }
}
A successful response from `GET /my-index-000001/_termvectors` with a `filter` in the request body.
{
  "_index": "imdb",
  "_version": 0,
  "found": true,
  "term_vectors": {
      "plot": {
        "field_statistics": {
            "sum_doc_freq": 3384269,
            "doc_count": 176214,
            "sum_ttf": 3753460
        },
        "terms": {
            "armored": {
              "doc_freq": 27,
              "ttf": 27,
              "term_freq": 1,
              "score": 9.74725
            },
            "industrialist": {
              "doc_freq": 88,
              "ttf": 88,
              "term_freq": 1,
              "score": 8.590818
            },
            "stark": {
              "doc_freq": 44,
              "ttf": 47,
              "term_freq": 1,
              "score": 9.272792
            }
        }
      }
  }
}































































































































































































































































































































































Get index templates Added in 7.9.0

GET /_index_template

Get information about one or more index templates.

Query parameters

  • local boolean

    If true, the request retrieves information from the local node only. Defaults to false, which means information is retrieved from the master node.

  • If true, returns settings in flat format.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • If true, returns all relevant default configurations for the index template.

Responses

GET /_index_template
curl \
 --request GET http://api.example.com/_index_template
Response examples (200)
{
  "index_templates": [
    {
      "name": "string",
      "index_template": {
        "index_patterns": "string",
        "composed_of": [
          "string"
        ],
        "template": {
          "aliases": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "mappings": {
            "all_field": {},
            "date_detection": true,
            "dynamic": "strict",
            "dynamic_date_formats": [
              "string"
            ],
            "dynamic_templates": [
              {}
            ],
            "_field_names": {},
            "index_field": {},
            "_meta": {},
            "numeric_detection": true,
            "properties": {},
            "_routing": {},
            "_size": {},
            "_source": {},
            "runtime": {},
            "enabled": true,
            "subobjects": true,
            "_data_stream_timestamp": {}
          },
          "settings": {
            "index": {},
            "mode": "string",
            "soft_deletes": {},
            "sort": {},
            "number_of_routing_shards": 42.0,
            "check_on_startup": "true",
            "codec": "string",
            "load_fixed_bitset_filters_eagerly": true,
            "auto_expand_replicas": "string",
            "merge": {},
            "search": {},
            "refresh_interval": "string",
            "max_result_window": 42.0,
            "max_inner_result_window": 42.0,
            "max_rescore_window": 42.0,
            "max_docvalue_fields_search": 42.0,
            "max_script_fields": 42.0,
            "max_ngram_diff": 42.0,
            "max_shingle_diff": 42.0,
            "blocks": {},
            "max_refresh_listeners": 42.0,
            "analyze": {},
            "highlight": {},
            "max_terms_count": 42.0,
            "max_regex_length": 42.0,
            "routing": {},
            "gc_deletes": "string",
            "default_pipeline": "string",
            "final_pipeline": "string",
            "lifecycle": {},
            "provided_name": "string",
            "uuid": "string",
            "version": {},
            "max_slices_per_scroll": 42.0,
            "translog": {},
            "query_string": {},
            "top_metrics_max_size": 42.0,
            "analysis": {},
            "settings": {},
            "time_series": {},
            "queries": {},
            "similarity": {},
            "mapping": {},
            "indexing.slowlog": {},
            "indexing_pressure": {},
            "store": {}
          },
          "": {}
        },
        "version": 42.0,
        "priority": 42.0,
        "_meta": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "allow_auto_create": true,
        "data_stream": {
          "hidden": true,
          "allow_custom_routing": true
        },
        "deprecated": true,
        "ignore_missing_component_templates": "string"
      }
    }
  ]
}
























Get index settings

GET /{index}/_settings

Get setting information for one or more indices. For data streams, it returns setting information for the stream's backing indices.

Path parameters

  • index string | array[string] Required

    Comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (*). To target all data streams and indices, omit this parameter or use * or _all.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden.

  • If true, returns settings in flat format.

  • If false, the request returns an error if it targets a missing or closed index.

  • If true, return all default settings in the response.

  • local boolean

    If true, the request retrieves information from the local node only. If false, information is retrieved from the master node.

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • * object

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • aliases object
        Hide aliases attribute Show aliases attribute object
      • mappings object

        Additional properties are allowed.

        Hide mappings attributes Show mappings attributes object
      • settings object

        Additional properties are allowed.

        Hide settings attributes Show settings attributes object
        • index object

          Additional properties are allowed.

        • mode string
        • Additional properties are allowed.

          Hide soft_deletes attributes Show soft_deletes attributes object
          • enabled boolean

            Indicates whether soft deletes are enabled on the index.

          • Additional properties are allowed.

            Hide retention_lease attribute Show retention_lease attribute object
            • period string Required

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
        • Values are true, false, or checksum.

        • codec string
        • routing_partition_size number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • merge object

          Additional properties are allowed.

          Hide merge attribute Show merge attribute object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • blocks object

          Additional properties are allowed.

          Hide blocks attributes Show blocks attributes object
        • analyze object

          Additional properties are allowed.

          Hide analyze attribute Show analyze attribute object
        • Additional properties are allowed.

          Hide highlight attribute Show highlight attribute object
        • routing object

          Additional properties are allowed.

          Hide routing attributes Show routing attributes object
          • Additional properties are allowed.

            Hide allocation attributes Show allocation attributes object
            • enable string

              Values are all, primaries, new_primaries, or none.

            • include object

              Additional properties are allowed.

              Hide include attributes Show include attributes object
            • Additional properties are allowed.

              Hide initial_recovery attribute Show initial_recovery attribute object
            • disk object

              Additional properties are allowed.

              Hide disk attribute Show disk attribute object
          • Additional properties are allowed.

            Hide rebalance attribute Show rebalance attribute object
            • enable string Required

              Values are all, primaries, replicas, or none.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide lifecycle attributes Show lifecycle attributes object
          • name string
          • indexing_complete boolean | string

            Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

            Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

          • If specified, this is the timestamp used to calculate the index age for its phase transitions. Use this setting if you create a new index that contains old data and want to use the original creation date to calculate the index age. Specified as a Unix epoch value in milliseconds.

          • Set to true to parse the origination date from the index name. This origination date is used to calculate the index age for its phase transitions. The index name must match the pattern .*-{date_format}-\d+, where the date_format is yyyy.MM.dd and the trailing digits are optional. An index that was rolled over would normally match the full format, for example logs-2016.10.31-000002). If the index name doesn’t match the pattern, index creation fails.

          • step object

            Additional properties are allowed.

            Hide step attribute Show step attribute object
            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • The index alias to update when the index rolls over. Specify when using a policy that contains a rollover action. When the index rolls over, the alias is updated to reflect that the index is no longer the write index. For more information about rolling indices, see Rollover.

          • prefer_ilm boolean | string

            Preference for the system that manages a data stream backing index (preferring ILM when both ILM and DLM are applicable for an index).

        • creation_date number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • creation_date_string string | number

          A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

        • uuid string
        • version object

          Additional properties are allowed.

          Hide version attributes Show version attributes object
        • translog object

          Additional properties are allowed.

          Hide translog attributes Show translog attributes object
        • Additional properties are allowed.

          Hide query_string attribute Show query_string attribute object
        • analysis object

          Additional properties are allowed.

          Hide analysis attributes Show analysis attributes object
        • settings object

          Additional properties are allowed.

        • Additional properties are allowed.

          Hide time_series attributes Show time_series attributes object
        • queries object

          Additional properties are allowed.

          Hide queries attribute Show queries attribute object
          • cache object

            Additional properties are allowed.

            Hide cache attribute Show cache attribute object
        • Configure custom similarity settings to customize how search results are scored.

        • mapping object

          Additional properties are allowed.

          Hide mapping attributes Show mapping attributes object
          • coerce boolean
          • Additional properties are allowed.

            Hide total_fields attributes Show total_fields attributes object
            • limit number | string

              The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.

            • ignore_dynamic_beyond_limit boolean | string

              This setting determines what happens when a dynamically mapped field would exceed the total fields limit. When set to false (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message Limit of total fields [X] has been exceeded. When set to true, the index request will not fail. Instead, fields that would exceed the limit are not added to the mapping, similar to dynamic: false. The fields that were not added to the mapping will be added to the _ignored field.

          • depth object

            Additional properties are allowed.

            Hide depth attribute Show depth attribute object
            • limit number

              The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc.

          • Additional properties are allowed.

            Hide nested_fields attribute Show nested_fields attribute object
            • limit number

              The maximum number of distinct nested mappings in an index. The nested type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique nested types per index.

          • Additional properties are allowed.

            Hide nested_objects attribute Show nested_objects attribute object
            • limit number

              The maximum number of nested JSON objects that a single document can contain across all nested types. This limit helps to prevent out of memory errors when a document contains too many nested objects.

          • Additional properties are allowed.

            Hide field_name_length attribute Show field_name_length attribute object
            • limit number

              Setting for the maximum length of a field name. This setting isn’t really something that addresses mappings explosion but might still be useful if you want to limit the field length. It usually shouldn’t be necessary to set this setting. The default is okay unless a user starts to add a huge number of fields with really long names. Default is Long.MAX_VALUE (no limit).

          • Additional properties are allowed.

            Hide dimension_fields attribute Show dimension_fields attribute object
            • limit number

              [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

          • source object

            Additional properties are allowed.

            Hide source attribute Show source attribute object
            • mode string Required

              Values are DISABLED, STORED, or SYNTHETIC.

        • Additional properties are allowed.

          Hide indexing.slowlog attributes Show indexing.slowlog attributes object
          • level string
          • source number
          • reformat boolean
          • Additional properties are allowed.

            Hide threshold attribute Show threshold attribute object
            • index object

              Additional properties are allowed.

              Hide index attributes Show index attributes object
              • warn string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • info string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • debug string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • trace string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object Required

            Additional properties are allowed.

            Hide memory attribute Show memory attribute object
            • limit number

              Number of outstanding bytes that may be consumed by indexing requests. When this limit is reached or exceeded, the node will reject new coordinating and primary operations. When replica operations consume 1.5x this limit, the node will reject new replica operations. Defaults to 10% of the heap.

        • store object

          Additional properties are allowed.

          Hide store attributes Show store attributes object
          • type string Required

          • allow_mmap boolean

            You can restrict the use of the mmapfs and the related hybridfs store type via the setting node.store.allow_mmap. This is a boolean setting indicating whether or not memory-mapping is allowed. The default is to allow it. This setting is useful, for example, if you are in an environment where you can not control the ability to create a lot of memory maps so you need disable the ability to use memory-mapping.

      • defaults object

        Additional properties are allowed.

        Hide defaults attributes Show defaults attributes object
        • index object

          Additional properties are allowed.

        • mode string
        • Additional properties are allowed.

          Hide soft_deletes attributes Show soft_deletes attributes object
          • enabled boolean

            Indicates whether soft deletes are enabled on the index.

          • Additional properties are allowed.

            Hide retention_lease attribute Show retention_lease attribute object
            • period string Required

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
        • Values are true, false, or checksum.

        • codec string
        • routing_partition_size number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • merge object

          Additional properties are allowed.

          Hide merge attribute Show merge attribute object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • blocks object

          Additional properties are allowed.

          Hide blocks attributes Show blocks attributes object
        • analyze object

          Additional properties are allowed.

          Hide analyze attribute Show analyze attribute object
        • Additional properties are allowed.

          Hide highlight attribute Show highlight attribute object
        • routing object

          Additional properties are allowed.

          Hide routing attributes Show routing attributes object
          • Additional properties are allowed.

            Hide allocation attributes Show allocation attributes object
            • enable string

              Values are all, primaries, new_primaries, or none.

            • include object

              Additional properties are allowed.

              Hide include attributes Show include attributes object
            • Additional properties are allowed.

              Hide initial_recovery attribute Show initial_recovery attribute object
            • disk object

              Additional properties are allowed.

              Hide disk attribute Show disk attribute object
          • Additional properties are allowed.

            Hide rebalance attribute Show rebalance attribute object
            • enable string Required

              Values are all, primaries, replicas, or none.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide lifecycle attributes Show lifecycle attributes object
          • name string
          • indexing_complete boolean | string

            Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

            Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

          • If specified, this is the timestamp used to calculate the index age for its phase transitions. Use this setting if you create a new index that contains old data and want to use the original creation date to calculate the index age. Specified as a Unix epoch value in milliseconds.

          • Set to true to parse the origination date from the index name. This origination date is used to calculate the index age for its phase transitions. The index name must match the pattern .*-{date_format}-\d+, where the date_format is yyyy.MM.dd and the trailing digits are optional. An index that was rolled over would normally match the full format, for example logs-2016.10.31-000002). If the index name doesn’t match the pattern, index creation fails.

          • step object

            Additional properties are allowed.

            Hide step attribute Show step attribute object
            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • The index alias to update when the index rolls over. Specify when using a policy that contains a rollover action. When the index rolls over, the alias is updated to reflect that the index is no longer the write index. For more information about rolling indices, see Rollover.

          • prefer_ilm boolean | string

            Preference for the system that manages a data stream backing index (preferring ILM when both ILM and DLM are applicable for an index).

        • creation_date number | string

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • creation_date_string string | number

          A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

        • uuid string
        • version object

          Additional properties are allowed.

          Hide version attributes Show version attributes object
        • translog object

          Additional properties are allowed.

          Hide translog attributes Show translog attributes object
        • Additional properties are allowed.

          Hide query_string attribute Show query_string attribute object
        • analysis object

          Additional properties are allowed.

          Hide analysis attributes Show analysis attributes object
        • settings object

          Additional properties are allowed.

        • Additional properties are allowed.

          Hide time_series attributes Show time_series attributes object
        • queries object

          Additional properties are allowed.

          Hide queries attribute Show queries attribute object
          • cache object

            Additional properties are allowed.

            Hide cache attribute Show cache attribute object
        • Configure custom similarity settings to customize how search results are scored.

        • mapping object

          Additional properties are allowed.

          Hide mapping attributes Show mapping attributes object
          • coerce boolean
          • Additional properties are allowed.

            Hide total_fields attributes Show total_fields attributes object
            • limit number | string

              The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The limit is in place to prevent mappings and searches from becoming too large. Higher values can lead to performance degradations and memory issues, especially in clusters with a high load or few resources.

            • ignore_dynamic_beyond_limit boolean | string

              This setting determines what happens when a dynamically mapped field would exceed the total fields limit. When set to false (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message Limit of total fields [X] has been exceeded. When set to true, the index request will not fail. Instead, fields that would exceed the limit are not added to the mapping, similar to dynamic: false. The fields that were not added to the mapping will be added to the _ignored field.

          • depth object

            Additional properties are allowed.

            Hide depth attribute Show depth attribute object
            • limit number

              The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc.

          • Additional properties are allowed.

            Hide nested_fields attribute Show nested_fields attribute object
            • limit number

              The maximum number of distinct nested mappings in an index. The nested type should only be used in special cases, when arrays of objects need to be queried independently of each other. To safeguard against poorly designed mappings, this setting limits the number of unique nested types per index.

          • Additional properties are allowed.

            Hide nested_objects attribute Show nested_objects attribute object
            • limit number

              The maximum number of nested JSON objects that a single document can contain across all nested types. This limit helps to prevent out of memory errors when a document contains too many nested objects.

          • Additional properties are allowed.

            Hide field_name_length attribute Show field_name_length attribute object
            • limit number

              Setting for the maximum length of a field name. This setting isn’t really something that addresses mappings explosion but might still be useful if you want to limit the field length. It usually shouldn’t be necessary to set this setting. The default is okay unless a user starts to add a huge number of fields with really long names. Default is Long.MAX_VALUE (no limit).

          • Additional properties are allowed.

            Hide dimension_fields attribute Show dimension_fields attribute object
            • limit number

              [preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

          • source object

            Additional properties are allowed.

            Hide source attribute Show source attribute object
            • mode string Required

              Values are DISABLED, STORED, or SYNTHETIC.

        • Additional properties are allowed.

          Hide indexing.slowlog attributes Show indexing.slowlog attributes object
          • level string
          • source number
          • reformat boolean
          • Additional properties are allowed.

            Hide threshold attribute Show threshold attribute object
            • index object

              Additional properties are allowed.

              Hide index attributes Show index attributes object
              • warn string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • info string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • debug string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

              • trace string

                A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide indexing_pressure attribute Show indexing_pressure attribute object
          • memory object Required

            Additional properties are allowed.

            Hide memory attribute Show memory attribute object
            • limit number

              Number of outstanding bytes that may be consumed by indexing requests. When this limit is reached or exceeded, the node will reject new coordinating and primary operations. When replica operations consume 1.5x this limit, the node will reject new replica operations. Defaults to 10% of the heap.

        • store object

          Additional properties are allowed.

          Hide store attributes Show store attributes object
          • type string Required

          • allow_mmap boolean

            You can restrict the use of the mmapfs and the related hybridfs store type via the setting node.store.allow_mmap. This is a boolean setting indicating whether or not memory-mapping is allowed. The default is to allow it. This setting is useful, for example, if you are in an environment where you can not control the ability to create a lot of memory maps so you need disable the ability to use memory-mapping.

      • Additional properties are allowed.

        Hide lifecycle attributes Show lifecycle attributes object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • Additional properties are allowed.

          Hide downsampling attribute Show downsampling attribute object
          • rounds array[object] Required

            The list of downsampling rounds to execute as part of this downsampling configuration

            Hide rounds attributes Show rounds attributes object
            • after string Required

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • config object Required

              Additional properties are allowed.

        • enabled boolean

          If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle that's disabled (enabled: false) will have no effect on the data stream.

GET /{index}/_settings
curl \
 --request GET http://api.example.com/{index}/_settings
Response examples (200)
{
  "*": {
    "aliases": {
      "additionalProperty1": {
        "filter": {},
        "index_routing": "string",
        "is_hidden": true,
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string"
      },
      "additionalProperty2": {
        "filter": {},
        "index_routing": "string",
        "is_hidden": true,
        "is_write_index": true,
        "routing": "string",
        "search_routing": "string"
      }
    },
    "mappings": {
      "all_field": {
        "analyzer": "string",
        "enabled": true,
        "omit_norms": true,
        "search_analyzer": "string",
        "similarity": "string",
        "store": true,
        "store_term_vector_offsets": true,
        "store_term_vector_payloads": true,
        "store_term_vector_positions": true,
        "store_term_vectors": true
      },
      "date_detection": true,
      "dynamic": "strict",
      "dynamic_date_formats": [
        "string"
      ],
      "dynamic_templates": [
        {}
      ],
      "_field_names": {
        "enabled": true
      },
      "index_field": {
        "enabled": true
      },
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      },
      "numeric_detection": true,
      "properties": {},
      "_routing": {
        "required": true
      },
      "_size": {
        "enabled": true
      },
      "_source": {
        "compress": true,
        "compress_threshold": "string",
        "enabled": true,
        "excludes": [
          "string"
        ],
        "includes": [
          "string"
        ],
        "mode": "disabled"
      },
      "runtime": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "fetch_fields": [
            {}
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {},
            "options": {}
          },
          "type": "boolean"
        }
      },
      "enabled": true,
      "subobjects": true,
      "_data_stream_timestamp": {
        "enabled": true
      }
    },
    "settings": {
      "index": {},
      "mode": "string",
      "routing_path": "string",
      "soft_deletes": {
        "enabled": true,
        "retention_lease": {
          "period": "string"
        }
      },
      "sort": {
        "field": "string",
        "order": "asc",
        "mode": "min",
        "missing": "_last"
      },
      "number_of_shards": 42.0,
      "number_of_replicas": 42.0,
      "number_of_routing_shards": 42.0,
      "check_on_startup": "true",
      "codec": "string",
      "": "string",
      "load_fixed_bitset_filters_eagerly": true,
      "hidden": true,
      "auto_expand_replicas": "string",
      "merge": {
        "scheduler": {
          "": 42.0
        }
      },
      "search": {
        "idle": {
          "after": "string"
        },
        "slowlog": {
          "level": "string",
          "source": 42.0,
          "reformat": true,
          "threshold": {
            "query": {},
            "fetch": {}
          }
        }
      },
      "refresh_interval": "string",
      "max_result_window": 42.0,
      "max_inner_result_window": 42.0,
      "max_rescore_window": 42.0,
      "max_docvalue_fields_search": 42.0,
      "max_script_fields": 42.0,
      "max_ngram_diff": 42.0,
      "max_shingle_diff": 42.0,
      "blocks": {
        "": true
      },
      "max_refresh_listeners": 42.0,
      "analyze": {
        "": 42.0
      },
      "highlight": {
        "max_analyzed_offset": 42.0
      },
      "max_terms_count": 42.0,
      "max_regex_length": 42.0,
      "routing": {
        "allocation": {
          "enable": "all",
          "include": {
            "_tier_preference": "string",
            "_id": "string"
          },
          "initial_recovery": {
            "_id": "string"
          },
          "disk": {}
        },
        "rebalance": {
          "enable": "all"
        }
      },
      "gc_deletes": "string",
      "default_pipeline": "string",
      "final_pipeline": "string",
      "lifecycle": {
        "name": "string",
        "": true,
        "origination_date": 42.0,
        "parse_origination_date": true,
        "step": {
          "wait_time_threshold": "string"
        },
        "rollover_alias": "string",
        "prefer_ilm": true
      },
      "provided_name": "string",
      "uuid": "string",
      "version": {
        "created": "string",
        "created_string": "string"
      },
      "verified_before_close": true,
      "format": "string",
      "max_slices_per_scroll": 42.0,
      "translog": {
        "sync_interval": "string",
        "durability": "request",
        "": 42.0,
        "retention": {
          "": 42.0,
          "age": "string"
        }
      },
      "query_string": {
        "": true
      },
      "priority": 42.0,
      "top_metrics_max_size": 42.0,
      "analysis": {
        "analyzer": {},
        "char_filter": {},
        "filter": {},
        "normalizer": {},
        "tokenizer": {}
      },
      "settings": {},
      "time_series": {
        "": "string"
      },
      "queries": {
        "cache": {
          "enabled": true
        }
      },
      "similarity": {},
      "mapping": {
        "coerce": true,
        "total_fields": {
          "limit": 42.0,
          "ignore_dynamic_beyond_limit": true
        },
        "depth": {
          "limit": 42.0
        },
        "nested_fields": {
          "limit": 42.0
        },
        "nested_objects": {
          "limit": 42.0
        },
        "field_name_length": {
          "limit": 42.0
        },
        "dimension_fields": {
          "limit": 42.0
        },
        "source": {
          "mode": "DISABLED"
        },
        "ignore_malformed": true
      },
      "indexing.slowlog": {
        "level": "string",
        "source": 42.0,
        "reformat": true,
        "threshold": {
          "index": {
            "warn": "string",
            "info": "string",
            "debug": "string",
            "trace": "string"
          }
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit": 42.0
        }
      },
      "store": {
        "": "fs",
        "allow_mmap": true
      }
    },
    "defaults": {
      "index": {},
      "mode": "string",
      "routing_path": "string",
      "soft_deletes": {
        "enabled": true,
        "retention_lease": {
          "period": "string"
        }
      },
      "sort": {
        "field": "string",
        "order": "asc",
        "mode": "min",
        "missing": "_last"
      },
      "number_of_shards": 42.0,
      "number_of_replicas": 42.0,
      "number_of_routing_shards": 42.0,
      "check_on_startup": "true",
      "codec": "string",
      "": "string",
      "load_fixed_bitset_filters_eagerly": true,
      "hidden": true,
      "auto_expand_replicas": "string",
      "merge": {
        "scheduler": {
          "": 42.0
        }
      },
      "search": {
        "idle": {
          "after": "string"
        },
        "slowlog": {
          "level": "string",
          "source": 42.0,
          "reformat": true,
          "threshold": {
            "query": {},
            "fetch": {}
          }
        }
      },
      "refresh_interval": "string",
      "max_result_window": 42.0,
      "max_inner_result_window": 42.0,
      "max_rescore_window": 42.0,
      "max_docvalue_fields_search": 42.0,
      "max_script_fields": 42.0,
      "max_ngram_diff": 42.0,
      "max_shingle_diff": 42.0,
      "blocks": {
        "": true
      },
      "max_refresh_listeners": 42.0,
      "analyze": {
        "": 42.0
      },
      "highlight": {
        "max_analyzed_offset": 42.0
      },
      "max_terms_count": 42.0,
      "max_regex_length": 42.0,
      "routing": {
        "allocation": {
          "enable": "all",
          "include": {
            "_tier_preference": "string",
            "_id": "string"
          },
          "initial_recovery": {
            "_id": "string"
          },
          "disk": {}
        },
        "rebalance": {
          "enable": "all"
        }
      },
      "gc_deletes": "string",
      "default_pipeline": "string",
      "final_pipeline": "string",
      "lifecycle": {
        "name": "string",
        "": true,
        "origination_date": 42.0,
        "parse_origination_date": true,
        "step": {
          "wait_time_threshold": "string"
        },
        "rollover_alias": "string",
        "prefer_ilm": true
      },
      "provided_name": "string",
      "uuid": "string",
      "version": {
        "created": "string",
        "created_string": "string"
      },
      "verified_before_close": true,
      "format": "string",
      "max_slices_per_scroll": 42.0,
      "translog": {
        "sync_interval": "string",
        "durability": "request",
        "": 42.0,
        "retention": {
          "": 42.0,
          "age": "string"
        }
      },
      "query_string": {
        "": true
      },
      "priority": 42.0,
      "top_metrics_max_size": 42.0,
      "analysis": {
        "analyzer": {},
        "char_filter": {},
        "filter": {},
        "normalizer": {},
        "tokenizer": {}
      },
      "settings": {},
      "time_series": {
        "": "string"
      },
      "queries": {
        "cache": {
          "enabled": true
        }
      },
      "similarity": {},
      "mapping": {
        "coerce": true,
        "total_fields": {
          "limit": 42.0,
          "ignore_dynamic_beyond_limit": true
        },
        "depth": {
          "limit": 42.0
        },
        "nested_fields": {
          "limit": 42.0
        },
        "nested_objects": {
          "limit": 42.0
        },
        "field_name_length": {
          "limit": 42.0
        },
        "dimension_fields": {
          "limit": 42.0
        },
        "source": {
          "mode": "DISABLED"
        },
        "ignore_malformed": true
      },
      "indexing.slowlog": {
        "level": "string",
        "source": 42.0,
        "reformat": true,
        "threshold": {
          "index": {
            "warn": "string",
            "info": "string",
            "debug": "string",
            "trace": "string"
          }
        }
      },
      "indexing_pressure": {
        "memory": {
          "limit": 42.0
        }
      },
      "store": {
        "": "fs",
        "allow_mmap": true
      }
    },
    "data_stream": "string",
    "lifecycle": {
      "data_retention": "string",
      "downsampling": {
        "rounds": [
          {
            "after": "string",
            "config": {}
          }
        ]
      },
      "enabled": true
    }
  }
}























































































































































































































































































































































Simulate data ingestion Technical preview

POST /_ingest/_simulate

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Query parameters

  • pipeline string

    The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index.

application/json

Body Required

  • docs array[object] Required

    Sample documents to test in the pipeline.

    Hide docs attributes Show docs attributes object
    • _id string
    • _index string
    • _source object Required

      JSON body for the document.

      Additional properties are allowed.

  • A map of component template names to substitute component template definition objects.

    Hide component_template_substitutions attribute Show component_template_substitutions attribute object
  • A map of index template names to substitute index template definition objects.

    Hide index_template_subtitutions attribute Show index_template_subtitutions attribute object
    • * object

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • index_patterns string | array[string] Required
      • composed_of array[string] Required

        An ordered list of component template names. Component templates are merged in the order specified, meaning that the last component template specified has the highest precedence.

      • template object

        Additional properties are allowed.

        Hide template attributes Show template attributes object
      • version number
      • priority number

        Priority to determine index template precedence when a new data stream or index is created. The index template with the highest priority is chosen. If no priority is specified the template is treated as though it is of priority 0 (lowest priority). This number is not automatically generated by Elasticsearch.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide data_stream attributes Show data_stream attributes object
      • deprecated boolean

        Marks this index template as deprecated. When creating or updating a non-deprecated index template that uses deprecated components, Elasticsearch will emit a deprecation warning.

  • Additional properties are allowed.

    Hide mapping_addition attributes Show mapping_addition attributes object
  • Pipelines to test. If you don’t specify the pipeline request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

    Hide pipeline_substitutions attribute Show pipeline_substitutions attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • Description of the ingest pipeline.

      • on_failure array[object]

        Processors to run immediately after a processor failure.

        Hide on_failure attributes Show on_failure attributes object
        • append object

          Additional properties are allowed.

          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Additional properties are allowed.

          Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object

          Additional properties are allowed.

          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object

          Additional properties are allowed.

          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object

          Additional properties are allowed.

          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object

          Additional properties are allowed.

          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

            Additional properties are allowed.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object

          Additional properties are allowed.

          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Additional properties are allowed.

          Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object

          Additional properties are allowed.

          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Additional properties are allowed.

          Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object

          Additional properties are allowed.

          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object

          Additional properties are allowed.

          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object

          Additional properties are allowed.

          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Additional properties are allowed.

          Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object

          Additional properties are allowed.

          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object

          Additional properties are allowed.

          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object

          Additional properties are allowed.

          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object

          Additional properties are allowed.

          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object

          Additional properties are allowed.

          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • Additional properties are allowed.

        • join object

          Additional properties are allowed.

          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object

          Additional properties are allowed.

          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object

          Additional properties are allowed.

          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Additional properties are allowed.

          Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object

          Additional properties are allowed.

          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object

          Additional properties are allowed.

          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Additional properties are allowed.

          Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object

          Additional properties are allowed.

          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object

          Additional properties are allowed.

          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object

          Additional properties are allowed.

          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang string

            Script language.

          • params object

            Object containing parameters for the script.

          • source string

            Inline script. If no id is specified, this parameter is required.

        • set object

          Additional properties are allowed.

          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object

          Additional properties are allowed.

          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object

          Additional properties are allowed.

          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • processors array[object]

        Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

        Hide processors attributes Show processors attributes object
        • append object

          Additional properties are allowed.

          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Additional properties are allowed.

          Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object

          Additional properties are allowed.

          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object

          Additional properties are allowed.

          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object

          Additional properties are allowed.

          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object

          Additional properties are allowed.

          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

            Additional properties are allowed.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object

          Additional properties are allowed.

          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Additional properties are allowed.

          Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object

          Additional properties are allowed.

          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Additional properties are allowed.

          Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object

          Additional properties are allowed.

          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object

          Additional properties are allowed.

          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object

          Additional properties are allowed.

          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Additional properties are allowed.

          Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object

          Additional properties are allowed.

          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object

          Additional properties are allowed.

          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object

          Additional properties are allowed.

          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object

          Additional properties are allowed.

          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object

          Additional properties are allowed.

          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • Additional properties are allowed.

        • join object

          Additional properties are allowed.

          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object

          Additional properties are allowed.

          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object

          Additional properties are allowed.

          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Additional properties are allowed.

          Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object

          Additional properties are allowed.

          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object

          Additional properties are allowed.

          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Additional properties are allowed.

          Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object

          Additional properties are allowed.

          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object

          Additional properties are allowed.

          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object

          Additional properties are allowed.

          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang string

            Script language.

          • params object

            Object containing parameters for the script.

          • source string

            Inline script. If no id is specified, this parameter is required.

        • set object

          Additional properties are allowed.

          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object

          Additional properties are allowed.

          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object

          Additional properties are allowed.

          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • version number
      • deprecated boolean

        Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

          Additional properties are allowed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attribute Show docs attribute object
      • doc object

        Additional properties are allowed.

        Hide doc attributes Show doc attributes object
        • _id string Required
        • _index string Required
        • _source object Required

          JSON body for the document.

          Hide _source attribute Show _source attribute object
          • * object Additional properties

            Additional properties are allowed.

        • _version number | string Required

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • executed_pipelines array[string] Required

          A list of the names of the pipelines executed on this document.

        • ignored_fields array[object]

          A list of the fields that would be ignored at the indexing step. For example, a field whose value is larger than the allowed limit would make it through all of the pipelines, but would not be indexed into Elasticsearch.

          Hide ignored_fields attribute Show ignored_fields attribute object
          • * string Additional properties
        • error object

          Additional properties are allowed.

          Hide error attributes Show error attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]

            Additional properties are allowed.

          • suppressed array[object]

            Additional properties are allowed.

POST /_ingest/_simulate
curl \
 --request POST http://api.example.com/_ingest/_simulate \
 --header "Content-Type: application/json" \
 --data '{"docs":[{"_id":123,"_index":"my-index","_source":{"foo":"bar"}},{"_id":456,"_index":"my-index","_source":{"foo":"rab"}}]}'
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. Since both documents are being ingested into `my-index`, both pipelines are run using the pipeline definitions that are already in the system.
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. But a substitute definition of `my-pipeline` is provided in `pipeline_substitutions`. The substitute `my-pipeline` will be used in place of the `my-pipeline` that is in the system, and then the `my-final-pipeline` that is already defined in the system will run.
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}
In this example, imagine that the index `my-index` has a strict mapping with only the `foo` keyword field defined. Say that field mapping came from a component template named `my-mappings-template`. You want to test adding a new field, `bar`. So a substitute definition of `my-mappings-template` is provided in `component_template_substitutions`. The substitute `my-mappings-template` will be used in place of the existing mapping for `my-index` and in place of the `my-mappings-template` that is in the system.
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
The pipeline, component template, and index template substitutions replace the existing pipeline details for the duration of this request.
{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  },
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ],
      "index_patterns": [
        "my-index-*"
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  }
}
A successful response when the simulation uses pipeline definitions that are already in the system.
{
  "docs": [
    {
      "_id": 123,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "bar",
        "field1": "value1",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "_id": 456,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "rab",
        "field1": "value1",
        "field2": "value2"
      },
      "_version": "-3,",
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "_id": 123,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "BAR",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "_id": 456,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "RAB",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "doc": {
        "_id": "123",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "foo": "foo"
        },
        "executed_pipelines": []
      }
    },
    {
      "doc": {
        "_id": "456",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "bar": "rab"
        },
      "executed_pipelines": []
      }
    }
  ]
}

Simulate data ingestion Technical preview

GET /_ingest/{index}/_simulate

Run ingest pipelines against a set of provided documents, optionally with substitute pipeline definitions, to simulate ingesting data into an index.

This API is meant to be used for troubleshooting or pipeline development, as it does not actually index any data into Elasticsearch.

The API runs the default and final pipeline for that index against a set of documents provided in the body of the request. If a pipeline contains a reroute processor, it follows that reroute processor to the new index, running that index's pipelines as well the same way that a non-simulated ingest would. No data is indexed into Elasticsearch. Instead, the transformed document is returned, along with the list of pipelines that have been run and the name of the index where the document would have been indexed if this were not a simulation. The transformed document is validated against the mappings that would apply to this index, and any validation error is reported in the result.

This API differs from the simulate pipeline API in that you specify a single pipeline for that API, and it runs only that one pipeline. The simulate pipeline API is more useful for developing a single pipeline, while the simulate ingest API is more useful for troubleshooting the interaction of the various pipelines that get applied when ingesting into an index.

By default, the pipeline definitions that are currently in the system are used. However, you can supply substitute pipeline definitions in the body of the request. These will be used in place of the pipeline definitions that are already in the system. This can be used to replace existing pipeline definitions or to create new ones. The pipeline substitutions are used only within this request.

Path parameters

  • index string Required

    The index to simulate ingesting into. This value can be overridden by specifying an index on each document. If you specify this parameter in the request path, it is used for any documents that do not explicitly specify an index argument.

Query parameters

  • pipeline string

    The pipeline to use as the default pipeline. This value can be used to override the default pipeline of the index.

application/json

Body Required

  • docs array[object] Required

    Sample documents to test in the pipeline.

    Hide docs attributes Show docs attributes object
    • _id string
    • _index string
    • _source object Required

      JSON body for the document.

      Additional properties are allowed.

  • A map of component template names to substitute component template definition objects.

    Hide component_template_substitutions attribute Show component_template_substitutions attribute object
  • A map of index template names to substitute index template definition objects.

    Hide index_template_subtitutions attribute Show index_template_subtitutions attribute object
    • * object

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • index_patterns string | array[string] Required
      • composed_of array[string] Required

        An ordered list of component template names. Component templates are merged in the order specified, meaning that the last component template specified has the highest precedence.

      • template object

        Additional properties are allowed.

        Hide template attributes Show template attributes object
      • version number
      • priority number

        Priority to determine index template precedence when a new data stream or index is created. The index template with the highest priority is chosen. If no priority is specified the template is treated as though it is of priority 0 (lowest priority). This number is not automatically generated by Elasticsearch.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

          Additional properties are allowed.

      • Additional properties are allowed.

        Hide data_stream attributes Show data_stream attributes object
      • deprecated boolean

        Marks this index template as deprecated. When creating or updating a non-deprecated index template that uses deprecated components, Elasticsearch will emit a deprecation warning.

  • Additional properties are allowed.

    Hide mapping_addition attributes Show mapping_addition attributes object
  • Pipelines to test. If you don’t specify the pipeline request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

    Hide pipeline_substitutions attribute Show pipeline_substitutions attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • Description of the ingest pipeline.

      • on_failure array[object]

        Processors to run immediately after a processor failure.

        Hide on_failure attributes Show on_failure attributes object
        • append object

          Additional properties are allowed.

          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Additional properties are allowed.

          Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object

          Additional properties are allowed.

          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object

          Additional properties are allowed.

          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object

          Additional properties are allowed.

          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object

          Additional properties are allowed.

          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

            Additional properties are allowed.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object

          Additional properties are allowed.

          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Additional properties are allowed.

          Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object

          Additional properties are allowed.

          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Additional properties are allowed.

          Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object

          Additional properties are allowed.

          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object

          Additional properties are allowed.

          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object

          Additional properties are allowed.

          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Additional properties are allowed.

          Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object

          Additional properties are allowed.

          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object

          Additional properties are allowed.

          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object

          Additional properties are allowed.

          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object

          Additional properties are allowed.

          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object

          Additional properties are allowed.

          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • Additional properties are allowed.

        • join object

          Additional properties are allowed.

          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object

          Additional properties are allowed.

          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object

          Additional properties are allowed.

          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Additional properties are allowed.

          Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object

          Additional properties are allowed.

          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object

          Additional properties are allowed.

          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Additional properties are allowed.

          Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object

          Additional properties are allowed.

          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object

          Additional properties are allowed.

          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object

          Additional properties are allowed.

          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang string

            Script language.

          • params object

            Object containing parameters for the script.

          • source string

            Inline script. If no id is specified, this parameter is required.

        • set object

          Additional properties are allowed.

          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object

          Additional properties are allowed.

          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object

          Additional properties are allowed.

          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • processors array[object]

        Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

        Hide processors attributes Show processors attributes object
        • append object

          Additional properties are allowed.

          Hide append attributes Show append attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If false, the processor does not append values already present in the field.

        • Additional properties are allowed.

          Hide attachment attributes Show attachment attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The number of chars being used for extraction to prevent huge fields. Use -1 for no limit.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Array of properties to select to be stored. Can be content, title, name, author, keywords, date, content_type, content_length, language.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the binary field will be removed from the document

          • Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.

        • bytes object

          Additional properties are allowed.

          Hide bytes attributes Show bytes attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • circle object

          Additional properties are allowed.

          Hide circle attributes Show circle attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • error_distance number Required

            The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • shape_type string Required

            Values are geo_shape or shape.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide community_id attributes Show community_id attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • seed number

            Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • convert object

          Additional properties are allowed.

          Hide convert attributes Show convert attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • type string Required

            Values are integer, long, double, float, boolean, ip, string, or auto.

        • csv object

          Additional properties are allowed.

          Hide csv attributes Show csv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Value used to fill empty fields. Empty fields are skipped if this is not provided. An empty field is one with no value (2 consecutive separators) or empty quotes ("").

            Additional properties are allowed.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • quote string

            Quote used in CSV, has to be single character string.

          • Separator used in CSV, has to be single character string.

          • target_fields string | array[string] Required
          • trim boolean

            Trim whitespaces in unquoted fields.

        • date object

          Additional properties are allowed.

          Hide date attributes Show date attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • formats array[string] Required

            An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • locale string

            The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • timezone string

            The timezone to use when parsing the date. Supports template snippets.

          • The format to use when writing the date to target_field. Must be a valid java time pattern.

        • Additional properties are allowed.

          Hide date_index_name attributes Show date_index_name attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • date_formats array[string]

            An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.

          • date_rounding string Required

            How to round the date when formatting the date into the index name. Valid values are: y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second). Supports template snippets.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.

          • A prefix of the index name to be prepended before the printed date. Supports template snippets.

          • locale string

            The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.

          • timezone string

            The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.

        • dissect object

          Additional properties are allowed.

          Hide dissect attributes Show dissect attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The character(s) that separate the appended fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to apply to the field.

        • Additional properties are allowed.

          Hide dot_expander attributes Show dot_expander attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • override boolean

            Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

          • path string

            The field that contains the field to expand. Only required if the field to expand is part another object field, because the field option can only understand leaf fields.

        • drop object

          Additional properties are allowed.

          Hide drop attributes Show drop attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • enrich object

          Additional properties are allowed.

          Hide enrich attributes Show enrich attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The maximum number of matched documents to include under the configured target field. The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object. In order to avoid documents getting too large, the maximum allowed value is 128.

          • override boolean

            If processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • policy_name string Required

            The name of the enrich policy to use.

          • Values are intersects, disjoint, within, or contains.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • fail object

          Additional properties are allowed.

          Hide fail attributes Show fail attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • message string Required

            The error message thrown by the processor. Supports template snippets.

        • Additional properties are allowed.

          Hide fingerprint attributes Show fingerprint attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • fields string | array[string] Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • salt string

            Salt value for the hash function.

          • method string

            Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.

          • If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.

        • foreach object

          Additional properties are allowed.

          Hide foreach attributes Show foreach attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true, the processor silently exits without changing the document if the field is null or missing.

          • processor object Required

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide ip_location attributes Show ip_location attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found IP location data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the IP location lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • geo_grid object

          Additional properties are allowed.

          Hide geo_grid attributes Show geo_grid attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            The field to interpret as a geo-tile.= The field format is determined by the tile_type.

          • tile_type string Required

            Values are geotile, geohex, or geohash.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Values are geojson or wkt.

        • geoip object

          Additional properties are allowed.

          Hide geoip attributes Show geoip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • first_only boolean

            If true, only the first found geoip data will be returned, even if the field contains an array.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • properties array[string]

            Controls what properties are added to the target_field based on the geoip lookup.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created. Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.

        • grok object

          Additional properties are allowed.

          Hide grok attributes Show grok attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.

          • patterns array[string] Required

            An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.

          • When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.

        • gsub object

          Additional properties are allowed.

          Hide gsub attributes Show gsub attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • pattern string Required

            The pattern to be replaced.

          • replacement string Required

            The string to replace the matching patterns with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide html_strip attributes Show html_strip attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document,

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide inference attributes Show inference attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • model_id string Required
          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.

          • Additional properties are allowed.

        • join object

          Additional properties are allowed.

          Hide join attributes Show join attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • separator string Required

            The separator character.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • json object

          Additional properties are allowed.

          Hide json attributes Show json attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Flag that forces the parsed JSON to be added at the top level of the document. target_field must not be set when this option is chosen.

          • Values are replace or merge.

          • When set to true, the JSON parser will not fail if the JSON contains duplicate keys. Instead, the last encountered value for any duplicate key wins.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • kv object

          Additional properties are allowed.

          Hide kv attributes Show kv attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • exclude_keys array[string]

            List of keys to exclude from document.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field_split string Required

            Regex pattern to use for splitting key-value pairs.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • include_keys array[string]

            List of keys to filter and insert into document. Defaults to including all keys.

          • prefix string

            Prefix to be added to extracted keys.

          • If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • trim_key string

            String of characters to trim from extracted keys.

          • String of characters to trim from extracted values.

          • value_split string Required

            Regex pattern to use for splitting the key from the value within a key-value pair.

        • Additional properties are allowed.

          Hide lowercase attributes Show lowercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide network_direction attributes Show network_direction attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • internal_networks array[string]

            List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • pipeline object

          Additional properties are allowed.

          Hide pipeline attributes Show pipeline attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • name string Required
          • Whether to ignore missing pipelines instead of failing.

        • redact object

          Additional properties are allowed.

          Hide redact attributes Show redact attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • patterns array[string] Required

            A list of grok expressions to match and redact named captures with

          • prefix string

            Start a redacted section with this token

          • suffix string

            End a redacted section with this token

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document

          • If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted

        • Additional properties are allowed.

          Hide registered_domain attributes Show registered_domain attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and any required fields are missing, the processor quietly exits without modifying the document.

        • remove object

          Additional properties are allowed.

          Hide remove attributes Show remove attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string | array[string] Required
          • keep string | array[string]
          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

        • rename object

          Additional properties are allowed.

          Hide rename attributes Show rename attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • target_field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • reroute object

          Additional properties are allowed.

          Hide reroute attributes Show reroute attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • A static value for the target. Can’t be set when the dataset or namespace option is set.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • id string
          • lang string

            Script language.

          • params object

            Object containing parameters for the script.

          • source string

            Inline script. If no id is specified, this parameter is required.

        • set object

          Additional properties are allowed.

          Hide set attributes Show set attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.

          • The media type for encoding value. Applies only when value is a template snippet. Must be one of application/json, text/plain, or application/x-www-form-urlencoded.

          • override boolean

            If true processor will update fields with pre-existing non-null-valued field. When set to false, such fields will not be touched.

          • value object

            The value to be set for the field. Supports template snippets. May specify only one of value or copy_from.

            Additional properties are allowed.

        • Additional properties are allowed.

          Hide set_security_user attributes Show set_security_user attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what user related properties are added to the field.

        • sort object

          Additional properties are allowed.

          Hide sort attributes Show sort attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • order string

            Values are asc or desc.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • split object

          Additional properties are allowed.

          Hide split attributes Show split attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Preserves empty trailing fields, if any.

          • separator string Required

            A regex which matches the separator, for example, , or \s+.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide terminate attributes Show terminate attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

        • trim object

          Additional properties are allowed.

          Hide trim attributes Show trim attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uppercase attributes Show uppercase attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide urldecode attributes Show urldecode attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist or is null, the processor quietly exits without modifying the document.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide uri_parts attributes Show uri_parts attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • If true, the processor copies the unparsed URI to <target_field>.original.

          • If true, the processor removes the field after parsing the URI string. If parsing fails, the processor does not remove the field.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Additional properties are allowed.

          Hide user_agent attributes Show user_agent attributes object
          • Description of the processor. Useful for describing the purpose of the processor or its configuration.

          • if object

            Additional properties are allowed.

          • Ignore failures for the processor.

          • on_failure array[object]

            Handle failures for the processor.

          • tag string

            Identifier for the processor. Useful for debugging and metrics.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • If true and field does not exist, the processor quietly exits without modifying the document.

          • The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • properties array[string]

            Controls what properties are added to target_field.

            Values are name, os, device, original, or version.

          • Extracts device type from the user agent string on a best-effort basis.

      • version number
      • deprecated boolean

        Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

          Additional properties are allowed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • docs array[object] Required
      Hide docs attribute Show docs attribute object
      • doc object

        Additional properties are allowed.

        Hide doc attributes Show doc attributes object
        • _id string Required
        • _index string Required
        • _source object Required

          JSON body for the document.

          Hide _source attribute Show _source attribute object
          • * object Additional properties

            Additional properties are allowed.

        • _version number | string Required

          Some APIs will return values such as numbers also as a string (notably epoch timestamps). This behavior is used to capture this behavior while keeping the semantics of the field type.

          Depending on the target language, code generators can keep the union or remove it and leniently parse strings to the target type.

        • executed_pipelines array[string] Required

          A list of the names of the pipelines executed on this document.

        • ignored_fields array[object]

          A list of the fields that would be ignored at the indexing step. For example, a field whose value is larger than the allowed limit would make it through all of the pipelines, but would not be indexed into Elasticsearch.

          Hide ignored_fields attribute Show ignored_fields attribute object
          • * string Additional properties
        • error object

          Additional properties are allowed.

          Hide error attributes Show error attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]

            Additional properties are allowed.

          • suppressed array[object]

            Additional properties are allowed.

GET /_ingest/{index}/_simulate
curl \
 --request GET http://api.example.com/_ingest/{index}/_simulate \
 --header "Content-Type: application/json" \
 --data '{"docs":[{"_id":123,"_index":"my-index","_source":{"foo":"bar"}},{"_id":456,"_index":"my-index","_source":{"foo":"rab"}}]}'
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. Since both documents are being ingested into `my-index`, both pipelines are run using the pipeline definitions that are already in the system.
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}
In this example the index `my-index` has a default pipeline called `my-pipeline` and a final pipeline called `my-final-pipeline`. But a substitute definition of `my-pipeline` is provided in `pipeline_substitutions`. The substitute `my-pipeline` will be used in place of the `my-pipeline` that is in the system, and then the `my-final-pipeline` that is already defined in the system will run.
{
  "docs": [
    {
      "_id": 123,
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": 456,
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "uppercase": {
            "field": "foo"
          }
        }
      ]
    }
  }
}
In this example, imagine that the index `my-index` has a strict mapping with only the `foo` keyword field defined. Say that field mapping came from a component template named `my-mappings-template`. You want to test adding a new field, `bar`. So a substitute definition of `my-mappings-template` is provided in `component_template_substitutions`. The substitute `my-mappings-template` will be used in place of the existing mapping for `my-index` and in place of the `my-mappings-template` that is in the system.
{
  "docs": [
    {
      "_index": "my-index",
      "_id": "123",
      "_source": {
        "foo": "foo"
      }
    },
    {
      "_index": "my-index",
      "_id": "456",
      "_source": {
        "bar": "rab"
      }
    }
  ],
  "component_template_substitutions": {
    "my-mappings_template": {
      "template": {
        "mappings": {
          "dynamic": "strict",
          "properties": {
            "foo": {
              "type": "keyword"
            },
            "bar": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}
The pipeline, component template, and index template substitutions replace the existing pipeline details for the duration of this request.
{
  "docs": [
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_id": "id",
      "_index": "my-index",
      "_source": {
        "foo": "rab"
      }
    }
  ],
  "mapping_addition": {
    "dynamic": "strict",
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  },
  "pipeline_substitutions": {
    "my-pipeline": {
      "processors": [
        {
          "set": {
            "field": "field3",
            "value": "value3"
          }
        }
      ]
    }
  },
  "index_template_substitutions": {
    "my-index-template": {
      "composed_of": [
        "component_template_1",
        "component_template_2"
      ],
      "index_patterns": [
        "my-index-*"
      ]
    }
  },
  "component_template_substitutions": {
    "my-component-template": {
      "template": {
        "mappings": {
          "dynamic": true,
          "properties": {
            "field3": {
              "type": "keyword"
            }
          }
        },
        "settings": {
          "index": {
            "default_pipeline": "my-pipeline"
          }
        }
      }
    }
  }
}
A successful response when the simulation uses pipeline definitions that are already in the system.
{
  "docs": [
    {
      "_id": 123,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "bar",
        "field1": "value1",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "_id": 456,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "rab",
        "field1": "value1",
        "field2": "value2"
      },
      "_version": "-3,",
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "_id": 123,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "BAR",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    },
    {
      "_id": 456,
      "doc": null,
      "_index": "my-index",
      "_source": {
        "foo": "RAB",
        "field2": "value2"
      },
      "_version": -3,
      "executed_pipelines": [
        "my-pipeline",
        "my-final-pipeline"
      ]
    }
  ]
}
A successful response when the simulation uses pipeline substitutions.
{
  "docs": [
    {
      "doc": {
        "_id": "123",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "foo": "foo"
        },
        "executed_pipelines": []
      }
    },
    {
      "doc": {
        "_id": "456",
        "_index": "my-index",
        "_version": -3,
        "_source": {
          "bar": "rab"
        },
      "executed_pipelines": []
      }
    }
  ]
}























































Get machine learning memory usage info Added in 8.2.0

GET /_ml/memory/_stats

Get information about how machine learning jobs and trained models are using memory, on each node, both within the JVM heap, and natively, outside of the JVM.

Query parameters

  • Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Responses

GET /_ml/memory/_stats
curl \
 --request GET http://api.example.com/_ml/memory/_stats
Response examples (200)
{
  "_nodes": {
    "failures": [
      {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      }
    ],
    "total": 42.0,
    "successful": 42.0,
    "failed": 42.0
  },
  "cluster_name": "string",
  "nodes": {
    "additionalProperty1": {
      "attributes": {
        "additionalProperty1": "string",
        "additionalProperty2": "string"
      },
      "jvm": {
        "": 42.0,
        "heap_max_in_bytes": 42.0,
        "java_inference_in_bytes": 42.0,
        "java_inference_max_in_bytes": 42.0
      },
      "mem": {
        "": 42.0,
        "adjusted_total_in_bytes": 42.0,
        "total_in_bytes": 42.0,
        "ml": {
          "": 42.0,
          "anomaly_detectors_in_bytes": 42.0,
          "data_frame_analytics_in_bytes": 42.0,
          "max_in_bytes": 42.0,
          "native_code_overhead_in_bytes": 42.0,
          "native_inference_in_bytes": 42.0
        }
      },
      "name": "string",
      "roles": [
        "string"
      ],
      "transport_address": "string",
      "ephemeral_id": "string"
    },
    "additionalProperty2": {
      "attributes": {
        "additionalProperty1": "string",
        "additionalProperty2": "string"
      },
      "jvm": {
        "": 42.0,
        "heap_max_in_bytes": 42.0,
        "java_inference_in_bytes": 42.0,
        "java_inference_max_in_bytes": 42.0
      },
      "mem": {
        "": 42.0,
        "adjusted_total_in_bytes": 42.0,
        "total_in_bytes": 42.0,
        "ml": {
          "": 42.0,
          "anomaly_detectors_in_bytes": 42.0,
          "data_frame_analytics_in_bytes": 42.0,
          "max_in_bytes": 42.0,
          "native_code_overhead_in_bytes": 42.0,
          "native_inference_in_bytes": 42.0
        }
      },
      "name": "string",
      "roles": [
        "string"
      ],
      "transport_address": "string",
      "ephemeral_id": "string"
    }
  }
}

















































Create a datafeed Added in 5.4.0

PUT /_ml/datafeeds/{datafeed_id}

Datafeeds retrieve data from Elasticsearch for analysis by an anomaly detection job. You can associate only one datafeed with each anomaly detection job. The datafeed contains a query that runs at a defined interval (frequency). If you are concerned about delayed data, you can add a delay (query_delay') at each interval. By default, the datafeed uses the following query:{"match_all": {"boost": 1}}`.

When Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead. You must use Kibana, this API, or the create anomaly detection jobs API to create a datafeed. Do not add a datafeed directly to the .ml-config index. Do not give users write privileges on the .ml-config index.

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the _all string or when no indices are specified.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.

  • ignore_throttled boolean Deprecated

    If true, concrete, expanded, or aliased indices are ignored when frozen.

  • If true, unavailable indices (missing or closed) are ignored.

application/json

Body Required

  • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

  • Additional properties are allowed.

    Hide chunking_config attributes Show chunking_config attributes object
    • mode string Required

      Values are auto, manual, or off.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • Additional properties are allowed.

    Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • enabled boolean Required

      Specifies whether the datafeed periodically checks for delayed data.

  • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • indices string | array[string]
  • Additional properties are allowed.

    Hide indices_options attributes Show indices_options attributes object
    • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

    • expand_wildcards string | array[string]
    • If true, missing or closed indices are not included in the response.

    • If true, concrete, expanded or aliased indices are ignored when frozen.

  • job_id string
  • If a real-time datafeed has never seen any data (including during any initial training period), it automatically stops and closes the associated job after this many real-time searches return no documents. In other words, it stops after frequency times max_empty_searches of real-time operation. If not set, a datafeed with no end time that sees no data remains started until it is explicitly stopped. By default, it is not set.

  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • Hide runtime_mappings attribute Show runtime_mappings attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • source string

          The script source.

        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

  • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

    Hide script_fields attribute Show script_fields attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • script object Required

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • source string

          The script source.

        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
  • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • headers object

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • Additional properties are allowed.

      Hide authorization attributes Show authorization attributes object
      • api_key object

        Additional properties are allowed.

        Hide api_key attributes Show api_key attributes object
        • id string Required

          The identifier for the API key.

        • name string Required

          The name of the API key.

      • roles array[string]

        If a user ID was used for the most recent update to the datafeed, its roles at the time of the update are listed in the response.

      • If a service account was used for the most recent update to the datafeed, the account name is listed in the response.

    • chunking_config object Required

      Additional properties are allowed.

      Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Additional properties are allowed.

      Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • datafeed_id string Required
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices array[string] Required
    • job_id string Required
    • Additional properties are allowed.

      Hide indices_options attributes Show indices_options attributes object
      • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • If true, missing or closed indices are not included in the response.

      • If true, concrete, expanded or aliased indices are ignored when frozen.

    • query object Required

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      Additional properties are allowed.

    • query_delay string Required

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

    • Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • script object Required

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
    • scroll_size number Required
PUT /_ml/datafeeds/{datafeed_id}
curl \
 --request PUT http://api.example.com/_ml/datafeeds/{datafeed_id} \
 --header "Content-Type: application/json" \
 --data '{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0,"headers":{}}'
Request examples
{
  "aggregations": {},
  "chunking_config": {
    "mode": "auto",
    "time_span": "string"
  },
  "delayed_data_check_config": {
    "check_window": "string",
    "enabled": true
  },
  "frequency": "string",
  "indices": "string",
  "indices_options": {
    "allow_no_indices": true,
    "expand_wildcards": "string",
    "ignore_unavailable": true,
    "ignore_throttled": true
  },
  "job_id": "string",
  "max_empty_searches": 42.0,
  "query": {},
  "query_delay": "string",
  "runtime_mappings": {
    "additionalProperty1": {
      "fields": {
        "additionalProperty1": {
          "type": "boolean"
        },
        "additionalProperty2": {
          "type": "boolean"
        }
      },
      "fetch_fields": [
        {
          "field": "string",
          "format": "string"
        }
      ],
      "format": "string",
      "input_field": "string",
      "target_field": "string",
      "target_index": "string",
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "type": "boolean"
    },
    "additionalProperty2": {
      "fields": {
        "additionalProperty1": {
          "type": "boolean"
        },
        "additionalProperty2": {
          "type": "boolean"
        }
      },
      "fetch_fields": [
        {
          "field": "string",
          "format": "string"
        }
      ],
      "format": "string",
      "input_field": "string",
      "target_field": "string",
      "target_index": "string",
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "type": "boolean"
    }
  },
  "script_fields": {
    "additionalProperty1": {
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "ignore_failure": true
    },
    "additionalProperty2": {
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "ignore_failure": true
    }
  },
  "scroll_size": 42.0,
  "headers": {}
}
Response examples (200)
{
  "aggregations": {},
  "authorization": {
    "api_key": {
      "id": "string",
      "name": "string"
    },
    "roles": [
      "string"
    ],
    "service_account": "string"
  },
  "chunking_config": {
    "mode": "auto",
    "time_span": "string"
  },
  "delayed_data_check_config": {
    "check_window": "string",
    "enabled": true
  },
  "datafeed_id": "string",
  "frequency": "string",
  "indices": [
    "string"
  ],
  "job_id": "string",
  "indices_options": {
    "allow_no_indices": true,
    "expand_wildcards": "string",
    "ignore_unavailable": true,
    "ignore_throttled": true
  },
  "max_empty_searches": 42.0,
  "query": {},
  "query_delay": "string",
  "runtime_mappings": {
    "additionalProperty1": {
      "fields": {
        "additionalProperty1": {
          "type": "boolean"
        },
        "additionalProperty2": {
          "type": "boolean"
        }
      },
      "fetch_fields": [
        {
          "field": "string",
          "format": "string"
        }
      ],
      "format": "string",
      "input_field": "string",
      "target_field": "string",
      "target_index": "string",
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "type": "boolean"
    },
    "additionalProperty2": {
      "fields": {
        "additionalProperty1": {
          "type": "boolean"
        },
        "additionalProperty2": {
          "type": "boolean"
        }
      },
      "fetch_fields": [
        {
          "field": "string",
          "format": "string"
        }
      ],
      "format": "string",
      "input_field": "string",
      "target_field": "string",
      "target_index": "string",
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "type": "boolean"
    }
  },
  "script_fields": {
    "additionalProperty1": {
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "ignore_failure": true
    },
    "additionalProperty2": {
      "script": {
        "source": "string",
        "id": "string",
        "params": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "": "painless",
        "options": {
          "additionalProperty1": "string",
          "additionalProperty2": "string"
        }
      },
      "ignore_failure": true
    }
  },
  "scroll_size": 42.0
}

Delete a datafeed Added in 5.4.0

DELETE /_ml/datafeeds/{datafeed_id}

Path parameters

  • datafeed_id string Required

    A numerical character string that uniquely identifies the datafeed. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • force boolean

    Use to forcefully delete a started datafeed; this method is quicker than stopping and deleting the datafeed.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

DELETE /_ml/datafeeds/{datafeed_id}
curl \
 --request DELETE http://api.example.com/_ml/datafeeds/{datafeed_id}
Response examples (200)
A successful response when deleting a datafeed.
{
  "acknowledged": true
}




































Create an anomaly detection job Added in 5.4.0

PUT /_ml/anomaly_detectors/{job_id}

If you include a datafeed_config, you must have read index privileges on the source index. If you include a datafeed_config but do not provide a query, the datafeed uses {"match_all": {"boost": 1}}.

Path parameters

  • job_id string Required

    The identifier for the anomaly detection job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

  • If true, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the _all string or when no indices are specified.

  • expand_wildcards string | array[string]

    Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values. Valid values are:

    • all: Match any data stream or index, including hidden ones.
    • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
    • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
    • none: Wildcard patterns are not accepted.
    • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
  • ignore_throttled boolean Deprecated

    If true, concrete, expanded or aliased indices are ignored when frozen.

  • If true, unavailable indices (missing or closed) are ignored.

application/json

Body Required

  • Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node. By default, if a machine learning node with capacity to run the job cannot immediately be found, the open anomaly detection jobs API returns an error. However, this is also subject to the cluster-wide xpack.ml.max_lazy_ml_nodes setting. If this option is set to true, the open anomaly detection jobs API does not return an error and the job waits in the opening state until sufficient machine learning node capacity is available.

  • analysis_config object Required

    Additional properties are allowed.

    Hide analysis_config attributes Show analysis_config attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • categorization_analyzer string | object

      One of:
    • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • If categorization_field_name is specified, you can also define optional filters. This property expects an array of regular expressions. The expressions are used to filter out matching sequences from the categorization field values. You can use this functionality to fine tune the categorization by excluding sequences from consideration when categories are defined. For example, you can exclude SQL statements that appear in your log files. This property cannot be used at the same time as categorization_analyzer. If you only want to define simple regular expression filters that are applied prior to tokenization, setting this property is the easiest method. If you also want to customize the tokenizer or post-tokenization filtering, use the categorization_analyzer property instead and include the filters as pattern_replace character filters. The effect is exactly the same.

    • detectors array[object] Required

      Detector configuration objects specify which data fields a job analyzes. They also specify which analytical functions are used. You can specify multiple detectors for a job. If the detectors array does not contain at least one detector, no analysis can occur and an error is returned.

      Hide detectors attributes Show detectors attributes object
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • custom_rules array[object]

        Custom rules enable you to customize the way detectors operate. For example, a rule may dictate conditions under which results should be skipped. Kibana refers to custom rules as job rules.

        Hide custom_rules attributes Show custom_rules attributes object
        • actions array[string]

          The set of actions to be triggered when the rule applies. If more than one action is specified the effects of all actions are combined.

          Values are skip_result or skip_model_update.

        • conditions array[object]

          An array of numeric conditions when the rule applies. A rule must either have a non-empty scope or at least one condition. Multiple conditions are combined together with a logical AND.

          Additional properties are allowed.

        • scope object

          A scope of series where the rule applies. A rule must either have a non-empty scope or at least one condition. By default, the scope includes all series. Scoping is allowed for any of the fields that are also specified in by_field_name, over_field_name, or partition_field_name.

          Hide scope attribute Show scope attribute object
          • * object Additional properties

            Additional properties are allowed.

      • A description of the detector.

      • A unique identifier for the detector. This identifier is based on the order of the detectors in the analysis_config, starting at zero. If you specify a value for this property, it is ignored.

      • Values are all, none, by, or over.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • function string

        The analysis function that is used. For example, count, rare, mean, min, max, or sum.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • use_null boolean

        Defines whether a new series is used as the null series when there is no value for the by or partition fields.

    • influencers array[string]

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • latency string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • This functionality is reserved for internal use. It is not supported for use in customer environments and is not subject to the support SLA of official GA features. If set to true, the analysis will automatically find correlations between metrics for a given by field value and report anomalies when those correlations cease to hold. For example, suppose CPU and memory usage on host A is usually highly correlated with the same metrics on host B. Perhaps this correlation occurs because they are running a load-balanced application. If you enable this property, anomalies will be reported when, for example, CPU usage on host A is high and the value of CPU usage on host B is low. That is to say, you’ll see an anomaly when the CPU of host A is unusual given the CPU of host B. To use the multivariate_by_fields property, you must also specify by_field_name in your detector.

    • Additional properties are allowed.

      Hide per_partition_categorization attributes Show per_partition_categorization attributes object
      • enabled boolean

        To enable this setting, you must also set the partition_field_name property to the same value in every detector that uses the keyword mlcategory. Otherwise, job creation fails.

      • This setting can be set to true only if per-partition categorization is enabled. If true, both categorization and subsequent anomaly detection stops for partitions where the categorization status changes to warn. This setting makes it viable to have a job where it is expected that categorization works well for some partitions but not others; you do not pay the cost of bad categorization forever in the partitions where it works badly.

    • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • Additional properties are allowed.

    Hide analysis_limits attributes Show analysis_limits attributes object
  • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

  • Custom metadata about the job

    Additional properties are allowed.

  • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job. Valid values range from 0 to model_snapshot_retention_days.

  • data_description object Required

    Additional properties are allowed.

    Hide data_description attributes Show data_description attributes object
    • format string

      Only JSON format is supported at this time.

    • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • The time format, which can be epoch, epoch_ms, or a custom pattern. The value epoch refers to UNIX or Epoch time (the number of seconds since 1 Jan 1970). The value epoch_ms indicates that time is measured in milliseconds since the epoch. The epoch and epoch_ms time formats accept either integer or real values. Custom patterns must conform to the Java DateTimeFormatter class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: yyyy-MM-dd'T'HH:mm:ssX. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.

  • Additional properties are allowed.

    Hide datafeed_config attributes Show datafeed_config attributes object
    • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

    • Additional properties are allowed.

      Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Additional properties are allowed.

      Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices string | array[string]
    • Additional properties are allowed.

      Hide indices_options attributes Show indices_options attributes object
      • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • If true, missing or closed indices are not included in the response.

      • If true, concrete, expanded or aliased indices are ignored when frozen.

    • job_id string
    • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

    • query object

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      Additional properties are allowed.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

    • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

      Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • script object Required

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
    • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • A description of the job.

  • job_id string
  • groups array[string]

    A list of job groups. A job can belong to no groups or many.

  • Additional properties are allowed.

    Hide model_plot_config attributes Show model_plot_config attributes object
    • If true, enables calculation and storage of the model change annotations for each entity that is being analyzed.

    • enabled boolean

      If true, enables calculation and storage of the model bounds for each entity that is being analyzed.

    • terms string

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

  • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. By default, snapshots ten days older than the newest snapshot are deleted.

  • Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket spans.

  • Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • allow_lazy_open boolean Required
    • analysis_config object Required

      Additional properties are allowed.

      Hide analysis_config attributes Show analysis_config attributes object
      • bucket_span string Required

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • categorization_analyzer string | object

        One of:
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • If categorization_field_name is specified, you can also define optional filters. This property expects an array of regular expressions. The expressions are used to filter out matching sequences from the categorization field values.

      • detectors array[object] Required

        An array of detector configuration objects. Detector configuration objects specify which data fields a job analyzes. They also specify which analytical functions are used. You can specify multiple detectors for a job.

        Hide detectors attributes Show detectors attributes object
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • custom_rules array[object]

          An array of custom rule objects, which enable you to customize the way detectors operate. For example, a rule may dictate to the detector conditions under which results should be skipped. Kibana refers to custom rules as job rules.

          Hide custom_rules attributes Show custom_rules attributes object
          • actions array[string]

            The set of actions to be triggered when the rule applies. If more than one action is specified the effects of all actions are combined.

            Values are skip_result or skip_model_update.

          • conditions array[object]

            An array of numeric conditions when the rule applies. A rule must either have a non-empty scope or at least one condition. Multiple conditions are combined together with a logical AND.

          • scope object

            A scope of series where the rule applies. A rule must either have a non-empty scope or at least one condition. By default, the scope includes all series. Scoping is allowed for any of the fields that are also specified in by_field_name, over_field_name, or partition_field_name.

        • A description of the detector.

        • A unique identifier for the detector. This identifier is based on the order of the detectors in the analysis_config, starting at zero.

        • Values are all, none, by, or over.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • function string Required

          The analysis function that is used. For example, count, rare, mean, min, max, and sum.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • use_null boolean

          Defines whether a new series is used as the null series when there is no value for the by or partition fields.

      • influencers array[string] Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • latency string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • This functionality is reserved for internal use. It is not supported for use in customer environments and is not subject to the support SLA of official GA features. If set to true, the analysis will automatically find correlations between metrics for a given by field value and report anomalies when those correlations cease to hold.

      • Additional properties are allowed.

        Hide per_partition_categorization attributes Show per_partition_categorization attributes object
        • enabled boolean

          To enable this setting, you must also set the partition_field_name property to the same value in every detector that uses the keyword mlcategory. Otherwise, job creation fails.

        • This setting can be set to true only if per-partition categorization is enabled. If true, both categorization and subsequent anomaly detection stops for partitions where the categorization status changes to warn. This setting makes it viable to have a job where it is expected that categorization works well for some partitions but not others; you do not pay the cost of bad categorization forever in the partitions where it works badly.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • analysis_limits object Required

      Additional properties are allowed.

      Hide analysis_limits attributes Show analysis_limits attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • create_time string | number Required

      A date and time, either as a string whose format can depend on the context (defaulting to ISO 8601), or a number of milliseconds since the Epoch. Elasticsearch accepts both as input, but will generally output a string representation.

    • Custom metadata about the job

      Additional properties are allowed.

    • data_description object Required

      Additional properties are allowed.

      Hide data_description attributes Show data_description attributes object
      • format string

        Only JSON format is supported at this time.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • The time format, which can be epoch, epoch_ms, or a custom pattern. The value epoch refers to UNIX or Epoch time (the number of seconds since 1 Jan 1970). The value epoch_ms indicates that time is measured in milliseconds since the epoch. The epoch and epoch_ms time formats accept either integer or real values. Custom patterns must conform to the Java DateTimeFormatter class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: yyyy-MM-dd'T'HH:mm:ssX. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.

    • Additional properties are allowed.

      Hide datafeed_config attributes Show datafeed_config attributes object
      • Additional properties are allowed.

        Hide authorization attributes Show authorization attributes object
        • api_key object

          Additional properties are allowed.

          Hide api_key attributes Show api_key attributes object
          • id string Required

            The identifier for the API key.

          • name string Required

            The name of the API key.

        • roles array[string]

          If a user ID was used for the most recent update to the datafeed, its roles at the time of the update are listed in the response.

        • If a service account was used for the most recent update to the datafeed, the account name is listed in the response.

      • Additional properties are allowed.

        Hide chunking_config attributes Show chunking_config attributes object
        • mode string Required

          Values are auto, manual, or off.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • datafeed_id string Required
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • indices array[string] Required
      • indexes array[string]
      • job_id string Required
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Hide script_fields attribute Show script_fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • script object Required

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
      • Additional properties are allowed.

        Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • enabled boolean Required

          Specifies whether the datafeed periodically checks for delayed data.

      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • Additional properties are allowed.

        Hide indices_options attributes Show indices_options attributes object
        • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

        • expand_wildcards string | array[string]
        • If true, missing or closed indices are not included in the response.

        • If true, concrete, expanded or aliased indices are ignored when frozen.

      • query object Required

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {"boost": 1}}.

        Additional properties are allowed.

    • groups array[string]
    • job_id string Required
    • job_type string Required
    • job_version string Required
    • Additional properties are allowed.

      Hide model_plot_config attributes Show model_plot_config attributes object
      • If true, enables calculation and storage of the model change annotations for each entity that is being analyzed.

      • enabled boolean

        If true, enables calculation and storage of the model bounds for each entity that is being analyzed.

      • terms string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • results_index_name string Required
PUT /_ml/anomaly_detectors/{job_id}
curl \
 --request PUT http://api.example.com/_ml/anomaly_detectors/{job_id} \
 --header "Content-Type: application/json" \
 --data '{"analysis_config":{"detectors":[{"function":"sum","field_name":"bytes","detector_description":"Sum of bytes"}],"bucket_span":"15m"},"analysis_limits":{"model_memory_limit":"11MB"},"datafeed_config":{"query":{"bool":{"must":[{"match_all":{}}]}},"indices":["kibana_sample_data_logs"],"datafeed_id":"datafeed-test-job1","runtime_mappings":{"hour_of_day":{"type":"long","script":{"source":"emit(doc['timestamp'].value.getHour());"}}}},"data_description":{"time_field":"timestamp","time_format":"epoch_ms"},"model_plot_config":{"enabled":true,"annotations_enabled":true},"results_index_name":"test-job1"}'
Request example
A request to create an anomaly detection job and datafeed.
{
  "analysis_config": {
    "detectors": [
      {
        "function": "sum",
        "field_name": "bytes",
        "detector_description": "Sum of bytes"
      }
    ],
    "bucket_span": "15m"
  },
  "analysis_limits": {
    "model_memory_limit": "11MB"
  },
  "datafeed_config": {
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "indices": [
      "kibana_sample_data_logs"
    ],
    "datafeed_id": "datafeed-test-job1",
    "runtime_mappings": {
      "hour_of_day": {
        "type": "long",
        "script": {
          "source": "emit(doc['timestamp'].value.getHour());"
        }
      }
    }
  },
  "data_description": {
    "time_field": "timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true,
    "annotations_enabled": true
  },
  "results_index_name": "test-job1"
}
Response examples (200)
A successful response when creating an anomaly detection job and datafeed.
{
  "job_id": "test-job1",
  "job_type": "anomaly_detector",
  "create_time": 1656087283340,
  "job_version": "8.4.0",
  "allow_lazy_open": false,
  "analysis_config": {
    "detectors": [
      {
        "function": "sum",
        "field_name": "bytes",
        "detector_index": 0,
        "detector_description": "Sum of bytes"
      }
    ],
    "bucket_span": "15m",
    "influencers": [],
    "model_prune_window": "30d"
  },
  "analysis_limits": {
    "model_memory_limit": "11mb",
    "categorization_examples_limit": 4
  },
  "datafeed_config": {
    "query": {
      "bool": {
        "must": [
          {
            "match_all": {}
          }
        ]
      }
    },
    "job_id": "test-job1",
    "indices": [
      "kibana_sample_data_logs"
    ],
    "datafeed_id": "datafeed-test-job1",
    "query_delay": "61499ms",
    "scroll_size": 1000,
    "authorization": {
      "roles": [
        "superuser"
      ]
    },
    "chunking_config": {
      "mode": "auto"
    },
    "indices_options": {
      "allow_no_indices": true,
      "expand_wildcards": [
        "open"
      ],
      "ignore_throttled": true,
      "ignore_unavailable": false
    },
    "runtime_mappings": {
      "hour_of_day": {
        "type": "long",
        "script": {
          "source": "emit(doc['timestamp'].value.getHour());"
        }
      }
    },
    "delayed_data_check_config": {
      "enabled": true
    }
  },
  "data_description": {
    "time_field": "timestamp",
    "time_format": "epoch_ms"
  },
  "model_plot_config": {
    "enabled": true,
    "annotations_enabled": true
  },
  "results_index_name": "custom-test-job1",
  "model_snapshot_retention_days": 10,
  "daily_model_snapshot_retention_after_days": 1
}




























































































































































Preview a datafeed Added in 5.4.0

POST /_ml/datafeeds/_preview

This API returns the first "page" of search results from a datafeed. You can preview an existing datafeed or provide configuration details for a datafeed and anomaly detection job in the API. The preview shows the structure of the data that will be passed to the anomaly detection engine. IMPORTANT: When Elasticsearch security features are enabled, the preview uses the credentials of the user that called the API. However, when the datafeed starts it uses the roles of the last user that created or updated the datafeed. To get a preview that accurately reflects the behavior of the datafeed, use the appropriate credentials. You can also use secondary authorization headers to supply the credentials.

Query parameters

  • start string | number

    The start time from where the datafeed preview should begin

  • end string | number

    The end time when the datafeed preview should stop

application/json

Body

  • Additional properties are allowed.

    Hide datafeed_config attributes Show datafeed_config attributes object
    • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

    • Additional properties are allowed.

      Hide chunking_config attributes Show chunking_config attributes object
      • mode string Required

        Values are auto, manual, or off.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Additional properties are allowed.

      Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • enabled boolean Required

        Specifies whether the datafeed periodically checks for delayed data.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • indices string | array[string]
    • Additional properties are allowed.

      Hide indices_options attributes Show indices_options attributes object
      • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

      • expand_wildcards string | array[string]
      • If true, missing or closed indices are not included in the response.

      • If true, concrete, expanded or aliased indices are ignored when frozen.

    • job_id string
    • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

    • query object

      An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

      Additional properties are allowed.

    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attribute Show * attribute object
            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • fetch_fields array[object]

          For type lookup

          Hide fetch_fields attributes Show fetch_fields attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • format string
        • format string

          A custom format for date type runtime fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • script object

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
        • type string Required

          Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

    • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

      Hide script_fields attribute Show script_fields attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • script object Required

          Additional properties are allowed.

          Hide script attributes Show script attributes object
          • source string

            The script source.

          • id string
          • params object

            Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

            Hide params attribute Show params attribute object
            • * object Additional properties

              Additional properties are allowed.

          • lang string

            Any of:

            Values are painless, expression, mustache, or java.

          • options object
            Hide options attribute Show options attribute object
            • * string Additional properties
    • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

  • Additional properties are allowed.

    Hide job_config attributes Show job_config attributes object
    • Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node.

    • analysis_config object Required

      Additional properties are allowed.

      Hide analysis_config attributes Show analysis_config attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • categorization_analyzer string | object

        One of:
      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • If categorization_field_name is specified, you can also define optional filters. This property expects an array of regular expressions. The expressions are used to filter out matching sequences from the categorization field values. You can use this functionality to fine tune the categorization by excluding sequences from consideration when categories are defined. For example, you can exclude SQL statements that appear in your log files. This property cannot be used at the same time as categorization_analyzer. If you only want to define simple regular expression filters that are applied prior to tokenization, setting this property is the easiest method. If you also want to customize the tokenizer or post-tokenization filtering, use the categorization_analyzer property instead and include the filters as pattern_replace character filters. The effect is exactly the same.

      • detectors array[object] Required

        Detector configuration objects specify which data fields a job analyzes. They also specify which analytical functions are used. You can specify multiple detectors for a job. If the detectors array does not contain at least one detector, no analysis can occur and an error is returned.

        Hide detectors attributes Show detectors attributes object
        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • custom_rules array[object]

          Custom rules enable you to customize the way detectors operate. For example, a rule may dictate conditions under which results should be skipped. Kibana refers to custom rules as job rules.

          Hide custom_rules attributes Show custom_rules attributes object
          • actions array[string]

            The set of actions to be triggered when the rule applies. If more than one action is specified the effects of all actions are combined.

            Values are skip_result or skip_model_update.

          • conditions array[object]

            An array of numeric conditions when the rule applies. A rule must either have a non-empty scope or at least one condition. Multiple conditions are combined together with a logical AND.

          • scope object

            A scope of series where the rule applies. A rule must either have a non-empty scope or at least one condition. By default, the scope includes all series. Scoping is allowed for any of the fields that are also specified in by_field_name, over_field_name, or partition_field_name.

        • A description of the detector.

        • A unique identifier for the detector. This identifier is based on the order of the detectors in the analysis_config, starting at zero. If you specify a value for this property, it is ignored.

        • Values are all, none, by, or over.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • function string

          The analysis function that is used. For example, count, rare, mean, min, max, or sum.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • use_null boolean

          Defines whether a new series is used as the null series when there is no value for the by or partition fields.

      • influencers array[string]

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • latency string

        A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • This functionality is reserved for internal use. It is not supported for use in customer environments and is not subject to the support SLA of official GA features. If set to true, the analysis will automatically find correlations between metrics for a given by field value and report anomalies when those correlations cease to hold. For example, suppose CPU and memory usage on host A is usually highly correlated with the same metrics on host B. Perhaps this correlation occurs because they are running a load-balanced application. If you enable this property, anomalies will be reported when, for example, CPU usage on host A is high and the value of CPU usage on host B is low. That is to say, you’ll see an anomaly when the CPU of host A is unusual given the CPU of host B. To use the multivariate_by_fields property, you must also specify by_field_name in your detector.

      • Additional properties are allowed.

        Hide per_partition_categorization attributes Show per_partition_categorization attributes object
        • enabled boolean

          To enable this setting, you must also set the partition_field_name property to the same value in every detector that uses the keyword mlcategory. Otherwise, job creation fails.

        • This setting can be set to true only if per-partition categorization is enabled. If true, both categorization and subsequent anomaly detection stops for partitions where the categorization status changes to warn. This setting makes it viable to have a job where it is expected that categorization works well for some partitions but not others; you do not pay the cost of bad categorization forever in the partitions where it works badly.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Additional properties are allowed.

      Hide analysis_limits attributes Show analysis_limits attributes object
    • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

    • Custom metadata about the job

      Additional properties are allowed.

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job.

    • data_description object Required

      Additional properties are allowed.

      Hide data_description attributes Show data_description attributes object
      • format string

        Only JSON format is supported at this time.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • The time format, which can be epoch, epoch_ms, or a custom pattern. The value epoch refers to UNIX or Epoch time (the number of seconds since 1 Jan 1970). The value epoch_ms indicates that time is measured in milliseconds since the epoch. The epoch and epoch_ms time formats accept either integer or real values. Custom patterns must conform to the Java DateTimeFormatter class. When you use date-time formatting patterns, it is recommended that you provide the full date, time and time zone. For example: yyyy-MM-dd'T'HH:mm:ssX. If the pattern that you specify is not sufficient to produce a complete timestamp, job creation fails.

    • Additional properties are allowed.

      Hide datafeed_config attributes Show datafeed_config attributes object
      • If set, the datafeed performs aggregation searches. Support for aggregations is limited and should be used only with low cardinality data.

      • Additional properties are allowed.

        Hide chunking_config attributes Show chunking_config attributes object
        • mode string Required

          Values are auto, manual, or off.

        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Additional properties are allowed.

        Hide delayed_data_check_config attributes Show delayed_data_check_config attributes object
        • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

        • enabled boolean Required

          Specifies whether the datafeed periodically checks for delayed data.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • indices string | array[string]
      • Additional properties are allowed.

        Hide indices_options attributes Show indices_options attributes object
        • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

        • expand_wildcards string | array[string]
        • If true, missing or closed indices are not included in the response.

        • If true, concrete, expanded or aliased indices are ignored when frozen.

      • job_id string
      • If a real-time datafeed has never seen any data (including during any initial training period) then it will automatically stop itself and close its associated job after this many real-time searches that return no documents. In other words, it will stop after frequency times max_empty_searches of real-time operation. If not set then a datafeed with no end time that sees no data will remain started until it is explicitly stopped.

      • query object

        An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

        Additional properties are allowed.

      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • Specifies scripts that evaluate custom expressions and returns script fields to the datafeed. The detector configuration objects in a job can contain functions that use these script fields.

        Hide script_fields attribute Show script_fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • script object Required

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
      • The size parameter that is used in Elasticsearch searches when the datafeed does not use aggregations. The maximum value is the value of index.max_result_window, which is 10,000 by default.

    • A description of the job.

    • groups array[string]

      A list of job groups. A job can belong to no groups or many.

    • job_id string
    • job_type string

      Reserved for future use, currently set to anomaly_detector.

    • Additional properties are allowed.

      Hide model_plot_config attributes Show model_plot_config attributes object
      • If true, enables calculation and storage of the model change annotations for each entity that is being analyzed.

      • enabled boolean

        If true, enables calculation and storage of the model bounds for each entity that is being analyzed.

      • terms string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. The default value is 10, which means snapshots ten days older than the newest snapshot are deleted.

    • Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket_spans.

    • Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.

Responses

  • 200 application/json

    Additional properties are allowed.

POST /_ml/datafeeds/_preview
curl \
 --request POST http://api.example.com/_ml/datafeeds/_preview \
 --header "Content-Type: application/json" \
 --data '{"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"job_config":{"allow_lazy_open":true,"analysis_config":{"bucket_span":"string","":"string","categorization_field_name":"string","categorization_filters":["string"],"detectors":[{"by_field_name":"string","custom_rules":[{"actions":["skip_result"],"conditions":[{}],"scope":{}}],"detector_description":"string","detector_index":42.0,"exclude_frequent":"all","field_name":"string","function":"string","over_field_name":"string","partition_field_name":"string","use_null":true}],"influencers":["string"],"latency":"string","model_prune_window":"string","multivariate_by_fields":true,"per_partition_categorization":{"enabled":true,"stop_on_warn":true},"summary_count_field_name":"string"},"analysis_limits":{"categorization_examples_limit":42.0,"":42.0},"background_persist_interval":"string","custom_settings":{},"daily_model_snapshot_retention_after_days":42.0,"data_description":{"format":"string","time_field":"string","time_format":"string","field_delimiter":"string"},"datafeed_config":{"aggregations":{},"chunking_config":{"mode":"auto","time_span":"string"},"datafeed_id":"string","delayed_data_check_config":{"check_window":"string","enabled":true},"frequency":"string","indices":"string","indices_options":{"allow_no_indices":true,"expand_wildcards":"string","ignore_unavailable":true,"ignore_throttled":true},"job_id":"string","max_empty_searches":42.0,"query":{},"query_delay":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"script_fields":{"additionalProperty1":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true},"additionalProperty2":{"script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"ignore_failure":true}},"scroll_size":42.0},"description":"string","groups":["string"],"job_id":"string","job_type":"string","model_plot_config":{"annotations_enabled":true,"enabled":true,"terms":"string"},"model_snapshot_retention_days":42.0,"renormalization_window_days":42.0,"results_index_name":"string","results_retention_days":42.0}}'
Request examples
{
  "datafeed_config": {
    "aggregations": {},
    "chunking_config": {
      "mode": "auto",
      "time_span": "string"
    },
    "datafeed_id": "string",
    "delayed_data_check_config": {
      "check_window": "string",
      "enabled": true
    },
    "frequency": "string",
    "indices": "string",
    "indices_options": {
      "allow_no_indices": true,
      "expand_wildcards": "string",
      "ignore_unavailable": true,
      "ignore_throttled": true
    },
    "job_id": "string",
    "max_empty_searches": 42.0,
    "query": {},
    "query_delay": "string",
    "runtime_mappings": {
      "additionalProperty1": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      },
      "additionalProperty2": {
        "fields": {
          "additionalProperty1": {
            "type": "boolean"
          },
          "additionalProperty2": {
            "type": "boolean"
          }
        },
        "fetch_fields": [
          {
            "field": "string",
            "format": "string"
          }
        ],
        "format": "string",
        "input_field": "string",
        "target_field": "string",
        "target_index": "string",
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "type": "boolean"
      }
    },
    "script_fields": {
      "additionalProperty1": {
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "ignore_failure": true
      },
      "additionalProperty2": {
        "script": {
          "source": "string",
          "id": "string",
          "params": {
            "additionalProperty1": {},
            "additionalProperty2": {}
          },
          "": "painless",
          "options": {
            "additionalProperty1": "string",
            "additionalProperty2": "string"
          }
        },
        "ignore_failure": true
      }
    },
    "scroll_size": 42.0
  },
  "job_config": {
    "allow_lazy_open": true,
    "analysis_config": {
      "bucket_span": "string",
      "": "string",
      "categorization_field_name": "string",
      "categorization_filters": [
        "string"
      ],
      "detectors": [
        {
          "by_field_name": "string",
          "custom_rules": [
            {
              "actions": [
                "skip_result"
              ],
              "conditions": [
                {}
              ],
              "scope": {}
            }
          ],
          "detector_description": "string",
          "detector_index": 42.0,
          "exclude_frequent": "all",
          "field_name": "string",
          "function": "string",
          "over_field_name": "string",
          "partition_field_name": "string",
          "use_null": true
        }
      ],
      "influencers": [
        "string"
      ],
      "latency": "string",
      "model_prune_window": "string",
      "multivariate_by_fields": true,
      "per_partition_categorization": {
        "enabled": true,
        "stop_on_warn": true
      },
      "summary_count_field_name": "string"
    },
    "analysis_limits": {
      "categorization_examples_limit": 42.0,
      "": 42.0
    },
    "background_persist_interval": "string",
    "custom_settings": {},
    "daily_model_snapshot_retention_after_days": 42.0,
    "data_description": {
      "format": "string",
      "time_field": "string",
      "time_format": "string",
      "field_delimiter": "string"
    },
    "datafeed_config": {
      "aggregations": {},
      "chunking_config": {
        "mode": "auto",
        "time_span": "string"
      },
      "datafeed_id": "string",
      "delayed_data_check_config": {
        "check_window": "string",
        "enabled": true
      },
      "frequency": "string",
      "indices": "string",
      "indices_options": {
        "allow_no_indices": true,
        "expand_wildcards": "string",
        "ignore_unavailable": true,
        "ignore_throttled": true
      },
      "job_id": "string",
      "max_empty_searches": 42.0,
      "query": {},
      "query_delay": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "script_fields": {
        "additionalProperty1": {
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "ignore_failure": true
        },
        "additionalProperty2": {
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "ignore_failure": true
        }
      },
      "scroll_size": 42.0
    },
    "description": "string",
    "groups": [
      "string"
    ],
    "job_id": "string",
    "job_type": "string",
    "model_plot_config": {
      "annotations_enabled": true,
      "enabled": true,
      "terms": "string"
    },
    "model_snapshot_retention_days": 42.0,
    "renormalization_window_days": 42.0,
    "results_index_name": "string",
    "results_retention_days": 42.0
  }
}
Response examples (200)
[
  {}
]

























































































Preview features used by data frame analytics Added in 7.13.0

GET /_ml/data_frame/analytics/_preview

Preview the extracted features used by a data frame analytics config.

application/json

Body

  • config object

    Additional properties are allowed.

    Hide config attributes Show config attributes object
    • source object Required

      Additional properties are allowed.

      Hide source attributes Show source attributes object
      • index string | array[string] Required
      • Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • fields object

            For type composite

            Hide fields attribute Show fields attribute object
            • * object Additional properties

              Additional properties are allowed.

              Hide * attribute Show * attribute object
              • type string Required

                Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

          • fetch_fields array[object]

            For type lookup

            Hide fetch_fields attributes Show fetch_fields attributes object
            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
          • format string

            A custom format for date type runtime fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • script object

            Additional properties are allowed.

            Hide script attributes Show script attributes object
            • source string

              The script source.

            • id string
            • params object

              Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

              Hide params attribute Show params attribute object
              • * object Additional properties

                Additional properties are allowed.

            • lang string

              Any of:

              Values are painless, expression, mustache, or java.

            • options object
              Hide options attribute Show options attribute object
              • * string Additional properties
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • _source object

        Additional properties are allowed.

        Hide _source attributes Show _source attributes object
        • includes array[string]

          An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

        • excludes array[string]

          An array of strings that defines the fields that will be included in the analysis.

      • query object

        The Elasticsearch query domain-specific language (DSL). This value corresponds to the query object in an Elasticsearch search POST body. All the options that are supported by Elasticsearch can be used, as this object is passed verbatim to Elasticsearch. By default, this property has the following value: {"match_all": {}}.

        Additional properties are allowed.

    • analysis object Required

      Additional properties are allowed.

      Hide analysis attributes Show analysis attributes object
      • Additional properties are allowed.

        Hide classification attributes Show classification attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • Defines the number of categories for which the predicted probabilities are reported. It must be non-negative or -1. If it is -1 or greater than the total number of categories, probabilities are reported for all categories; if you have a large number of categories, there could be a significant effect on the size of your destination index. NOTE: To use the AUC ROC evaluation method, num_top_classes must be set to -1 or a value greater than or equal to the total number of categories.

      • Additional properties are allowed.

        Hide outlier_detection attributes Show outlier_detection attributes object
        • Specifies whether the feature influence calculation is enabled.

        • The minimum outlier score that a document needs to have in order to calculate its feature influence score. Value range: 0-1.

        • method string

          The method that outlier detection uses. Available methods are lof, ldof, distance_kth_nn, distance_knn, and ensemble. The default value is ensemble, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score.

        • Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.

        • The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.

        • If true, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i).

      • Additional properties are allowed.

        Hide regression attributes Show regression attributes object
        • alpha number

          Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.

        • dependent_variable string Required

          Defines which field of the document is to be predicted. It must match one of the fields in the index being used to train. If this field is missing from a document, then that document will not be used for training, but a prediction with the trained model will be generated for it. It is also known as continuous target variable. For classification analysis, the data type of the field must be numeric (integer, short, long, byte), categorical (ip or keyword), or boolean. There must be no more than 30 different values in this field. For regression analysis, the data type of the field must be numeric.

        • Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.

        • Advanced configuration option. Specifies whether the training process should finish if it is not finding any better performing models. If disabled, the training process can take significantly longer and the chance of finding a better performing model is unremarkable.

        • eta number

          Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.

        • Advanced configuration option. Specifies the rate at which eta increases for each new tree that is added to the forest. For example, a rate of 1.05 increases eta by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2.

        • Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.

        • feature_processors array[object]

          Advanced configuration option. A collection of feature preprocessors that modify one or more included fields. The analysis uses the resulting one or more features instead of the original document field. However, these features are ephemeral; they are not stored in the destination index. Multiple feature_processors entries can refer to the same document fields. Automatic categorical feature encoding still occurs for the fields that are unprocessed by a custom processor or that have categorical values. Use this property only if you want to override the automatic feature encoding of the specified fields.

          Hide feature_processors attributes Show feature_processors attributes object
        • gamma number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • lambda number

          Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.

        • Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.

        • Advanced configuration option. Specifies the maximum number of feature importance values per document to return. By default, no feature importance calculation occurs.

        • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • Defines the seed for the random generator that is used to pick training data. By default, it is randomly generated. Set it to a specific value to use the same training data each time you start a job (assuming other related parameters such as source and analyzed_fields are the same).

        • Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This soft limit combines with the soft_tree_depth_tolerance to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.

        • Advanced configuration option. This option controls how quickly the regularized loss increases when the tree depth exceeds soft_tree_depth_limit. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.

        • The loss function used during regression. Available options are mse (mean squared error), msle (mean squared logarithmic error), huber (Pseudo-Huber loss).

        • A positive number that is used as a parameter to the loss_function.

    • Additional properties are allowed.

      Hide analyzed_fields attributes Show analyzed_fields attributes object
      • includes array[string]

        An array of strings that defines the fields that will be excluded from the analysis. You do not need to add fields with unsupported data types to excludes, these fields are excluded from the analysis automatically.

      • excludes array[string]

        An array of strings that defines the fields that will be included in the analysis.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • feature_values array[object] Required

      An array of objects that contain feature name and value pairs. The features have been processed and indicate what will be sent to the model for training.

      Hide feature_values attribute Show feature_values attribute object
      • * string Additional properties
GET /_ml/data_frame/analytics/_preview
curl \
 --request GET http://api.example.com/_ml/data_frame/analytics/_preview \
 --header "Content-Type: application/json" \
 --data '{"config":{"source":{"index":"string","runtime_mappings":{"additionalProperty1":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"},"additionalProperty2":{"fields":{"additionalProperty1":{"type":"boolean"},"additionalProperty2":{"type":"boolean"}},"fetch_fields":[{"field":"string","format":"string"}],"format":"string","input_field":"string","target_field":"string","target_index":"string","script":{"source":"string","id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}},"":"painless","options":{"additionalProperty1":"string","additionalProperty2":"string"}},"type":"boolean"}},"_source":{"includes":["string"],"excludes":["string"]},"query":{}},"analysis":{"":{"alpha":42.0,"dependent_variable":"string","downsample_factor":42.0,"early_stopping_enabled":true,"eta":42.0,"eta_growth_rate_per_tree":42.0,"feature_bag_fraction":42.0,"feature_processors":[{"frequency_encoding":{},"multi_encoding":{},"n_gram_encoding":{},"one_hot_encoding":{},"target_mean_encoding":{}}],"gamma":42.0,"lambda":42.0,"max_optimization_rounds_per_hyperparameter":42.0,"max_trees":42.0,"num_top_feature_importance_values":42.0,"prediction_field_name":"string","randomize_seed":42.0,"soft_tree_depth_limit":42.0,"soft_tree_depth_tolerance":42.0,"":"string","loss_function":"string","loss_function_parameter":42.0},"outlier_detection":{"compute_feature_influence":true,"feature_influence_threshold":42.0,"method":"string","n_neighbors":42.0,"outlier_fraction":42.0,"standardization_enabled":true}},"model_memory_limit":"string","max_num_threads":42.0,"analyzed_fields":{"includes":["string"],"excludes":["string"]}}}'
Request examples
{
  "config": {
    "source": {
      "index": "string",
      "runtime_mappings": {
        "additionalProperty1": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        },
        "additionalProperty2": {
          "fields": {
            "additionalProperty1": {
              "type": "boolean"
            },
            "additionalProperty2": {
              "type": "boolean"
            }
          },
          "fetch_fields": [
            {
              "field": "string",
              "format": "string"
            }
          ],
          "format": "string",
          "input_field": "string",
          "target_field": "string",
          "target_index": "string",
          "script": {
            "source": "string",
            "id": "string",
            "params": {
              "additionalProperty1": {},
              "additionalProperty2": {}
            },
            "": "painless",
            "options": {
              "additionalProperty1": "string",
              "additionalProperty2": "string"
            }
          },
          "type": "boolean"
        }
      },
      "_source": {
        "includes": [
          "string"
        ],
        "excludes": [
          "string"
        ]
      },
      "query": {}
    },
    "analysis": {
      "": {
        "alpha": 42.0,
        "dependent_variable": "string",
        "downsample_factor": 42.0,
        "early_stopping_enabled": true,
        "eta": 42.0,
        "eta_growth_rate_per_tree": 42.0,
        "feature_bag_fraction": 42.0,
        "feature_processors": [
          {
            "frequency_encoding": {},
            "multi_encoding": {},
            "n_gram_encoding": {},
            "one_hot_encoding": {},
            "target_mean_encoding": {}
          }
        ],
        "gamma": 42.0,
        "lambda": 42.0,
        "max_optimization_rounds_per_hyperparameter": 42.0,
        "max_trees": 42.0,
        "num_top_feature_importance_values": 42.0,
        "prediction_field_name": "string",
        "randomize_seed": 42.0,
        "soft_tree_depth_limit": 42.0,
        "soft_tree_depth_tolerance": 42.0,
        "": "string",
        "loss_function": "string",
        "loss_function_parameter": 42.0
      },
      "outlier_detection": {
        "compute_feature_influence": true,
        "feature_influence_threshold": 42.0,
        "method": "string",
        "n_neighbors": 42.0,
        "outlier_fraction": 42.0,
        "standardization_enabled": true
      }
    },
    "model_memory_limit": "string",
    "max_num_threads": 42.0,
    "analyzed_fields": {
      "includes": [
        "string"
      ],
      "excludes": [
        "string"
      ]
    }
  }
}
Response examples (200)
{
  "feature_values": [
    {
      "additionalProperty1": "string",
      "additionalProperty2": "string"
    }
  ]
}














































































































































































































Get rollup job information Deprecated Technical preview

GET /_rollup/job

Get the configuration, stats, and status of rollup jobs.

NOTE: This API returns only active (both STARTED and STOPPED) jobs. If a job was created, ran for a while, then was deleted, the API does not return any details about it. For details about a historical rollup job, the rollup capabilities API may be more useful.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • jobs array[object] Required
      Hide jobs attributes Show jobs attributes object
      • config object Required

        Additional properties are allowed.

        Hide config attributes Show config attributes object
        • cron string Required
        • groups object Required

          Additional properties are allowed.

          Hide groups attributes Show groups attributes object
          • Additional properties are allowed.

            Hide date_histogram attributes Show date_histogram attributes object
            • delay string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • field string Required

              Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • format string
            • interval string

              A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

            • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • Additional properties are allowed.

            Hide histogram attributes Show histogram attributes object
            • fields string | array[string] Required
            • interval number Required

              The interval of histogram buckets to be generated when rolling up. For example, a value of 5 creates buckets that are five units wide (0-5, 5-10, etc). Note that only one interval can be specified in the histogram group, meaning that all fields being grouped via the histogram must share the same interval.

          • terms object

            Additional properties are allowed.

            Hide terms attribute Show terms attribute object
            • fields string | array[string] Required
        • id string Required
        • index_pattern string Required
        • metrics array[object] Required
          Hide metrics attributes Show metrics attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • metrics array[string] Required

            An array of metrics to collect for the field. At least one metric must be configured.

            Values are min, max, sum, avg, or value_count.

        • page_size number Required
        • rollup_index string Required
        • timeout string Required

          A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • stats object Required

        Additional properties are allowed.

        Hide stats attributes Show stats attributes object
      • status object Required

        Additional properties are allowed.

        Hide status attributes Show status attributes object
        • Hide current_position attribute Show current_position attribute object
          • * object Additional properties

            Additional properties are allowed.

        • job_state string Required

          Values are started, indexing, stopping, stopped, or aborting.

GET /_rollup/job
curl \
 --request GET http://api.example.com/_rollup/job
Response examples (200)
{
  "jobs": [
    {
      "config": {
        "cron": "string",
        "groups": {
          "date_histogram": {
            "delay": "string",
            "field": "string",
            "format": "string",
            "interval": "string",
            "calendar_interval": "string",
            "fixed_interval": "string",
            "time_zone": "string"
          },
          "histogram": {
            "fields": "string",
            "interval": 42.0
          },
          "terms": {
            "fields": "string"
          }
        },
        "id": "string",
        "index_pattern": "string",
        "metrics": [
          {
            "field": "string",
            "metrics": [
              "min"
            ]
          }
        ],
        "page_size": 42.0,
        "rollup_index": "string",
        "timeout": "string"
      },
      "stats": {
        "documents_processed": 42.0,
        "index_failures": 42.0,
        "": 42.0,
        "index_total": 42.0,
        "pages_processed": 42.0,
        "rollups_indexed": 42.0,
        "search_failures": 42.0,
        "search_total": 42.0,
        "trigger_count": 42.0,
        "processing_total": 42.0
      },
      "status": {
        "current_position": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "job_state": "started",
        "upgraded_doc_id": true
      }
    }
  ]
}














































































































































Get the field capabilities Added in 5.4.0

GET /_field_caps

Get information about the capabilities of fields among multiple indices.

For data streams, the API returns field capabilities among the stream’s backing indices. It returns runtime fields like any other field. For example, a runtime field with a type of keyword is returned the same as any other field that belongs to the keyword family.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • expand_wildcards string | array[string]

    The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as open,hidden.

  • fields string | array[string]

    A comma-separated list of fields to retrieve capabilities for. Wildcard (*) expressions are supported.

  • If true, missing or closed indices are not included in the response.

  • If true, unmapped fields are included in the response.

  • filters string

    A comma-separated list of filters to apply to the response.

  • types array[string]

    A comma-separated list of field types to include. Any fields that do not match one of these types will be excluded from the results. It defaults to empty, meaning that all field types are returned.

  • If false, empty fields are not included in the response.

application/json

Body

  • fields string | array[string]
  • An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • Hide runtime_mappings attribute Show runtime_mappings attribute object
    • * object Additional properties

      Additional properties are allowed.

      Hide * attributes Show * attributes object
      • fields object

        For type composite

        Hide fields attribute Show fields attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attribute Show * attribute object
          • type string Required

            Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

      • fetch_fields array[object]

        For type lookup

        Hide fetch_fields attributes Show fetch_fields attributes object
        • field string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • format string
      • format string

        A custom format for date type runtime fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • script object

        Additional properties are allowed.

        Hide script attributes Show script attributes object
        • source string

          The script source.

        • id string
        • params object

          Specifies any named parameters that are passed into the script as variables. Use parameters instead of hard-coded values to decrease compile time.

          Hide params attribute Show params attribute object
          • * object Additional properties

            Additional properties are allowed.

        • lang string

          Any of:

          Values are painless, expression, mustache, or java.

        • options object
          Hide options attribute Show options attribute object
          • * string Additional properties
      • type string Required

        Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • indices string | array[string] Required
    • fields object Required
      Hide fields attribute Show fields attribute object
      • * object Additional properties
        Hide * attribute Show * attribute object
        • * object Additional properties

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • aggregatable boolean Required

            Whether this field can be aggregated on all indices.

          • indices string | array[string]
          • meta object
            Hide meta attribute Show meta attribute object
            • * object Additional properties

              Additional properties are allowed.

          • non_aggregatable_indices string | array[string]
          • non_searchable_indices string | array[string]
          • searchable boolean Required

            Whether this field is indexed for search on all indices.

          • type string Required
          • Whether this field is registered as a metadata field.

          • time_series_dimension boolean Technical preview

            Whether this field is used as a time series dimension.

          • Values are gauge, counter, summary, histogram, or position.

          • non_dimension_indices array[string] Technical preview

            If this list is present in response then some indices have the field marked as a dimension and other indices, the ones in this list, do not.

          • metric_conflicts_indices array[string] Technical preview

            The list of indices where this field is present if these indices don’t have the same time_series_metric value for this field.

GET /_field_caps
curl \
 --request GET http://api.example.com/_field_caps \
 --header "Content-Type: application/json" \
 --data '"{\n  \"index_filter\": {\n    \"range\": {\n      \"@timestamp\": {\n        \"gte\": \"2018\"\n      }\n    }\n  }\n}"'
Request example
Run `POST my-index-*/_field_caps?fields=rating` to get field capabilities and filter indices with a query. Indices that rewrite the provided filter to `match_none` on every shard will be filtered from the response.
{
  "index_filter": {
    "range": {
      "@timestamp": {
        "gte": "2018"
      }
    }
  }
}
Response examples (200)
A successful response from `GET _field_caps?fields=rating,title`. The field `rating` is defined as a long in `index1` and `index2` and as a `keyword` in `index3` and `index4`. The field `rating` is not aggregatable in `index1`. The field `rating` is not searchable in `index4`. The field `title` is defined as text in all indices.
{
  "indices": [ "index1", "index2", "index3", "index4", "index5" ],
  "fields": {
    "rating": {                                   
      "long": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [ "index1", "index2" ],
        "non_aggregatable_indices": [ "index1" ]  
      },
      "keyword": {
        "metadata_field": false,
        "searchable": false,
        "aggregatable": true,
        "indices": [ "index3", "index4" ],
        "non_searchable_indices": [ "index4" ]    
      }
    },
    "title": {                                    
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    }
  }
}
A successful response from `GET _field_caps?fields=rating,title&include_unmapped`. The response contains an entry for each field that is present in some indices but not all. For example, the `rating` and `title` fields are unmapped in `index5`.
{
  "indices": [ "index1", "index2", "index3", "index4", "index5" ],
  "fields": {
    "rating": {                                   
      "long": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false,
        "indices": [ "index1", "index2" ],
        "non_aggregatable_indices": [ "index1" ]  
      },
      "keyword": {
        "metadata_field": false,
        "searchable": false,
        "aggregatable": true,
        "indices": [ "index3", "index4" ],
        "non_searchable_indices": [ "index4" ]    
      }
    },
    "title": {                                    
      "text": {
        "metadata_field": false,
        "searchable": true,
        "aggregatable": false
      }
    }
  }
}
















Run a knn search Deprecated Technical preview

POST /{index}/_knn_search

NOTE: The kNN search API has been replaced by the knn option in the search API.

Perform a k-nearest neighbor (kNN) search on a dense_vector field and return the matching documents. Given a query vector, the API finds the k closest vectors and returns those documents as search hits.

Elasticsearch uses the HNSW algorithm to support efficient kNN search. Like most kNN algorithms, HNSW is an approximate method that sacrifices result accuracy for improved search speed. This means the results returned are not always the true k closest neighbors.

The kNN search API supports restricting the search using a filter. The search will return the top k documents that also match the filter query.

A kNN search response has the exact same structure as a search API response. However, certain sections have a meaning specific to kNN search:

  • The document _score is determined by the similarity between the query and document vector.
  • The hits.total object contains the total number of nearest neighbor candidates considered, which is num_candidates * num_shards. The hits.total.relation will always be eq, indicating an exact value.

Path parameters

  • index string | array[string] Required

    A comma-separated list of index names to search; use _all or to perform the operation on all indices.

Query parameters

  • routing string

    A comma-separated list of specific routing values.

application/json

Body

  • _source boolean | object

    Defines how to fetch a source. Fetching can be disabled entirely, or the source can be filtered.

    One of:
  • docvalue_fields array[object]

    The request returns doc values for field names matching these patterns in the hits.fields property of the response. It accepts wildcard (*) patterns.

    Hide docvalue_fields attributes Show docvalue_fields attributes object
    • field string Required

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • format string

      The format in which the values are returned.

  • stored_fields string | array[string]
  • fields string | array[string]
  • filter object | array[object]

    A query to filter the documents that can match. The kNN search will return the top k documents that also match this filter. The value can be a single query or a list of queries. If filter isn't provided, all documents are allowed to match.

    One of:

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    Additional properties are allowed.

  • knn object Required

    Additional properties are allowed.

    Hide knn attributes Show knn attributes object
    • field string Required

      Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

    • query_vector array[number] Required
    • k number Required

      The final number of nearest neighbors to return as top hits

    • num_candidates number Required

      The number of nearest neighbor candidates to consider per shard

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • took number Required

      The milliseconds it took Elasticsearch to run the request.

    • timed_out boolean Required

      If true, the request timed out before completion; returned results may be partial or empty.

    • _shards object Required

      Additional properties are allowed.

      Hide _shards attributes Show _shards attributes object
      • failed number Required
      • successful number Required
      • total number Required
      • failures array[object]
        Hide failures attributes Show failures attributes object
        • index string
        • node string
        • reason object Required

          Additional properties are allowed.

          Hide reason attributes Show reason attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]

            Additional properties are allowed.

          • suppressed array[object]

            Additional properties are allowed.

        • shard number Required
        • status string
      • skipped number
    • hits object Required

      Additional properties are allowed.

      Hide hits attributes Show hits attributes object
    • fields object

      The field values for the documents. These fields must be specified in the request using the fields parameter.

      Hide fields attribute Show fields attribute object
      • * object Additional properties

        Additional properties are allowed.

    • The highest returned document score. This value is null for requests that do not sort by score.

POST /{index}/_knn_search
curl \
 --request POST http://api.example.com/{index}/_knn_search \
 --header "Content-Type: application/json" \
 --data '{"":true,"docvalue_fields":[{"field":"string","format":"string","include_unmapped":true}],"stored_fields":"string","fields":"string","filter":{},"knn":{"field":"string","query_vector":[42.0],"k":42.0,"num_candidates":42.0}}'
Request examples
{
  "": true,
  "docvalue_fields": [
    {
      "field": "string",
      "format": "string",
      "include_unmapped": true
    }
  ],
  "stored_fields": "string",
  "fields": "string",
  "filter": {},
  "knn": {
    "field": "string",
    "query_vector": [
      42.0
    ],
    "k": 42.0,
    "num_candidates": 42.0
  }
}
Response examples (200)
{
  "took": 42.0,
  "timed_out": true,
  "_shards": {
    "failed": 42.0,
    "successful": 42.0,
    "total": 42.0,
    "failures": [
      {
        "index": "string",
        "node": "string",
        "reason": {
          "type": "string",
          "reason": "string",
          "stack_trace": "string",
          "caused_by": {},
          "root_cause": [
            {}
          ],
          "suppressed": [
            {}
          ]
        },
        "shard": 42.0,
        "status": "string"
      }
    ],
    "skipped": 42.0
  },
  "hits": {
    "total": {
      "relation": "eq",
      "value": 42.0
    },
    "hits": [
      {
        "_index": "string",
        "_id": "string",
        "_score": 42.0,
        "_explanation": {
          "description": "string",
          "details": [
            {}
          ],
          "value": 42.0
        },
        "fields": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "highlight": {
          "additionalProperty1": [
            "string"
          ],
          "additionalProperty2": [
            "string"
          ]
        },
        "inner_hits": {
          "additionalProperty1": {
            "hits": {}
          },
          "additionalProperty2": {
            "hits": {}
          }
        },
        "matched_queries": [
          "string"
        ],
        "_nested": {
          "field": "string",
          "offset": 42.0,
          "_nested": {}
        },
        "_ignored": [
          "string"
        ],
        "ignored_field_values": {
          "additionalProperty1": [
            {}
          ],
          "additionalProperty2": [
            {}
          ]
        },
        "_shard": "string",
        "_node": "string",
        "_routing": "string",
        "_source": {},
        "_rank": 42.0,
        "_seq_no": 42.0,
        "_primary_term": 42.0,
        "_version": 42.0,
        "sort": [
          42.0
        ]
      }
    ],
    "max_score": 42.0
  },
  "fields": {
    "additionalProperty1": {},
    "additionalProperty2": {}
  },
  "max_score": 42.0
}








































Evaluate ranked search results Added in 6.2.0

POST /_rank_eval

Evaluate the quality of ranked search results over a set of typical search queries.

Query parameters

  • If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.

  • expand_wildcards string | array[string]

    Whether to expand wildcard expression to concrete indices that are open, closed or both.

  • If true, missing or closed indices are not included in the response.

  • Search operation type

application/json

Body Required

  • requests array[object] Required

    A set of typical search requests, together with their provided ratings.

    Hide requests attributes Show requests attributes object
    • id string Required
    • request object

      Additional properties are allowed.

      Hide request attributes Show request attributes object
    • ratings array[object] Required

      List of document ratings

      Hide ratings attributes Show ratings attributes object
      • _id string Required
      • _index string Required
      • rating number Required

        The document’s relevance with regard to this search request.

    • params object

      The search template parameters.

      Hide params attribute Show params attribute object
      • * object Additional properties

        Additional properties are allowed.

  • metric object

    Additional properties are allowed.

    Hide metric attributes Show metric attributes object
    • Additional properties are allowed.

      Hide precision attributes Show precision attributes object
      • k number

        Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

      • Sets the rating threshold above which documents are considered to be "relevant".

      • Controls how unlabeled documents in the search results are counted. If set to true, unlabeled documents are ignored and neither count as relevant or irrelevant. Set to false (the default), they are treated as irrelevant.

    • recall object

      Additional properties are allowed.

      Hide recall attributes Show recall attributes object
      • k number

        Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

      • Sets the rating threshold above which documents are considered to be "relevant".

    • Additional properties are allowed.

      Hide mean_reciprocal_rank attributes Show mean_reciprocal_rank attributes object
      • k number

        Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

      • Sets the rating threshold above which documents are considered to be "relevant".

    • dcg object

      Additional properties are allowed.

      Hide dcg attributes Show dcg attributes object
      • k number

        Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

      • normalize boolean

        If set to true, this metric will calculate the Normalized DCG.

    • Additional properties are allowed.

      Hide expected_reciprocal_rank attributes Show expected_reciprocal_rank attributes object
      • k number

        Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

      • maximum_relevance number Required

        The highest relevance grade used in the user-supplied relevance judgments.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • metric_score number Required

      The overall evaluation quality calculated by the defined metric

    • details object Required

      The details section contains one entry for every query in the original requests section, keyed by the search request id

      Hide details attribute Show details attribute object
      • * object Additional properties

        Additional properties are allowed.

        Hide * attributes Show * attributes object
        • metric_score number Required

          The metric_score in the details section shows the contribution of this query to the global quality metric score

        • unrated_docs array[object] Required

          The unrated_docs section contains an _index and _id entry for each document in the search result for this query that didn’t have a ratings value. This can be used to ask the user to supply ratings for these documents

          Hide unrated_docs attributes Show unrated_docs attributes object
        • hits array[object] Required

          The hits section shows a grouping of the search results with their supplied ratings

          Hide hits attributes Show hits attributes object
        • metric_details object Required

          The metric_details give additional information about the calculated quality metric (e.g. how many of the retrieved documents were relevant). The content varies for each metric but allows for better interpretation of the results

          Hide metric_details attribute Show metric_details attribute object
          • * object Additional properties
            Hide * attribute Show * attribute object
            • * object Additional properties

              Additional properties are allowed.

    • failures object Required
      Hide failures attribute Show failures attribute object
      • * object Additional properties

        Additional properties are allowed.

POST /_rank_eval
curl \
 --request POST http://api.example.com/_rank_eval \
 --header "Content-Type: application/json" \
 --data '{"requests":[{"id":"string","request":{"query":{},"size":42.0},"ratings":[{"_id":"string","_index":"string","rating":42.0}],"template_id":"string","params":{"additionalProperty1":{},"additionalProperty2":{}}}],"metric":{"":{"k":42.0,"maximum_relevance":42.0}}}'
Request examples
{
  "requests": [
    {
      "id": "string",
      "request": {
        "query": {},
        "size": 42.0
      },
      "ratings": [
        {
          "_id": "string",
          "_index": "string",
          "rating": 42.0
        }
      ],
      "template_id": "string",
      "params": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      }
    }
  ],
  "metric": {
    "": {
      "k": 42.0,
      "maximum_relevance": 42.0
    }
  }
}
Response examples (200)
{
  "metric_score": 42.0,
  "details": {
    "additionalProperty1": {
      "metric_score": 42.0,
      "unrated_docs": [
        {
          "_id": "string",
          "_index": "string"
        }
      ],
      "hits": [
        {
          "hit": {
            "_id": "string",
            "_index": "string",
            "_score": 42.0
          },
          "rating": 42.0
        }
      ],
      "metric_details": {
        "additionalProperty1": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "additionalProperty2": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      }
    },
    "additionalProperty2": {
      "metric_score": 42.0,
      "unrated_docs": [
        {
          "_id": "string",
          "_index": "string"
        }
      ],
      "hits": [
        {
          "hit": {
            "_id": "string",
            "_index": "string",
            "_score": 42.0
          },
          "rating": 42.0
        }
      ],
      "metric_details": {
        "additionalProperty1": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        },
        "additionalProperty2": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      }
    }
  },
  "failures": {
    "additionalProperty1": {},
    "additionalProperty2": {}
  }
}







































































































































































































































Create a service account token

POST /_security/service/{namespace}/{service}/credential/token/{name}

Create a service accounts token for access without requiring basic authentication.

NOTE: Service account tokens never expire. You must actively delete them if they are no longer needed.

Path parameters

  • namespace string Required

    The name of the namespace, which is a top-level grouping of service accounts.

  • service string Required

    The name of the service.

  • name string Required

    The name for the service account token. If omitted, a random name will be generated.

    Token names must be at least one and no more than 256 characters. They can contain alphanumeric characters (a-z, A-Z, 0-9), dashes (-), and underscores (_), but cannot begin with an underscore.

    NOTE: Token names must be unique in the context of the associated service account. They must also be globally unique with their fully qualified names, which are comprised of the service account principal and token name, such as <namespace>/<service>/<token-name>.

Query parameters

  • refresh string

    If true then refresh the affected shards to make this operation visible to search, if wait_for (the default) then wait for a refresh to make this operation visible to search, if false then do nothing with refreshes.

    Values are true, false, or wait_for.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • created boolean Required
    • token object Required

      Additional properties are allowed.

      Hide token attributes Show token attributes object
POST /_security/service/{namespace}/{service}/credential/token/{name}
curl \
 --request POST http://api.example.com/_security/service/{namespace}/{service}/credential/token/{name}
Response examples (200)
{
  "created": true,
  "token": {
    "name": "string",
    "value": "string"
  }
}








































































































































































































Check user profile privileges Added in 8.3.0

POST /_security/profile/_has_privileges

Determine whether the users associated with the specified user profile IDs have all the requested privileges.

NOTE: The user profile feature is designed only for use by Kibana and Elastic's Observability, Enterprise Search, and Elastic Security solutions. Individual users and external applications should not call this API directly. Elastic reserves the right to change or remove this feature in future releases without prior notice.

application/json

Body Required

  • uids array[string] Required

    A list of profile IDs. The privileges are checked for associated users of the profiles.

  • privileges object Required

    Additional properties are allowed.

    Hide privileges attributes Show privileges attributes object
    • application array[object]
      Hide application attributes Show application attributes object
      • application string Required

        The name of the application.

      • privileges array[string] Required

        A list of the privileges that you want to check for the specified resources. It may be either application privilege names or the names of actions that are granted by those privileges

      • resources array[string] Required

        A list of resource names against which the privileges should be checked.

    • cluster array[string]

      A list of the cluster privileges that you want to check.

    • index array[object]
      Hide index attributes Show index attributes object
      • names string | array[string] Required
      • privileges array[string] Required

        A list of the privileges that you want to check for the specified indices.

      • This needs to be set to true (default is false) if using wildcards or regexps for patterns that cover restricted indices. Implicitly, restricted indices do not match index patterns because restricted indices usually have limited privileges and including them in pattern tests would render most such tests false. If restricted indices are explicitly included in the names list, privileges will be checked against them regardless of the value of allow_restricted_indices.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • has_privilege_uids array[string] Required

      The subset of the requested profile IDs of the users that have all the requested privileges.

    • errors object

      Additional properties are allowed.

      Hide errors attributes Show errors attributes object
      • count number Required
      • details object Required
        Hide details attribute Show details attribute object
        • * object

          Additional properties are allowed.

          Hide * attributes Show * attributes object
          • type string Required

            The type of error

          • reason string

            A human-readable explanation of the error, in English.

          • The server stack trace. Present only if the error_trace=true parameter was sent with the request.

          • Additional properties are allowed.

          • root_cause array[object]

            Additional properties are allowed.

          • suppressed array[object]

            Additional properties are allowed.

POST /_security/profile/_has_privileges
curl \
 --request POST http://api.example.com/_security/profile/_has_privileges \
 --header "Content-Type: application/json" \
 --data '{"uids":["string"],"privileges":{"application":[{"application":"string","privileges":["string"],"resources":["string"]}],"cluster":["string"],"index":[{"names":"string","privileges":["string"],"allow_restricted_indices":true}]}}'
Request examples
{
  "uids": [
    "string"
  ],
  "privileges": {
    "application": [
      {
        "application": "string",
        "privileges": [
          "string"
        ],
        "resources": [
          "string"
        ]
      }
    ],
    "cluster": [
      "string"
    ],
    "index": [
      {
        "names": "string",
        "privileges": [
          "string"
        ],
        "allow_restricted_indices": true
      }
    ]
  }
}
Response examples (200)
{
  "has_privilege_uids": [
    "string"
  ],
  "errors": {
    "count": 42.0,
    "details": {
      "*": {
        "type": "string",
        "reason": "string",
        "stack_trace": "string",
        "caused_by": {},
        "root_cause": [
          {}
        ],
        "suppressed": [
          {}
        ]
      }
    }
  }
}








































Logout of SAML completely Added in 7.14.0

POST /_security/saml/complete_logout

Verifies the logout response sent from the SAML IdP.

NOTE: This API is intended for use by custom web applications other than Kibana. If you are using Kibana, refer to the documentation for configuring SAML single-sign-on on the Elastic Stack.

The SAML IdP may send a logout response back to the SP after handling the SP-initiated SAML Single Logout. This API verifies the response by ensuring the content is relevant and validating its signature. An empty response is returned if the verification process is successful. The response can be sent by the IdP with either the HTTP-Redirect or the HTTP-Post binding. The caller of this API must prepare the request accordingly so that this API can handle either of them.

application/json

Body Required

  • realm string Required

    The name of the SAML realm in Elasticsearch for which the configuration is used to verify the logout response.

  • If the SAML IdP sends the logout response with the HTTP-Redirect binding, this field must be set to the query string of the redirect URI.

  • content string

    If the SAML IdP sends the logout response with the HTTP-Post binding, this field must be set to the value of the SAMLResponse form parameter from the logout response.

Responses

POST /_security/saml/complete_logout
curl \
 --request POST http://api.example.com/_security/saml/complete_logout \
 --header "Content-Type: application/json" \
 --data '{"realm":"string","":"string","query_string":"string","content":"string"}'
Request examples
{
  "realm": "string",
  "": "string",
  "query_string": "string",
  "content": "string"
}


































































































































Run a policy Added in 7.4.0

PUT /_slm/policy/{policy_id}/_execute

Immediately create a snapshot according to the snapshot lifecycle policy without waiting for the scheduled time. The snapshot policy is normally applied according to its schedule, but you might want to manually run a policy before performing an upgrade or other maintenance.

Path parameters

  • policy_id string Required

    The id of the snapshot lifecycle policy to be executed

Query parameters

  • The period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.

  • timeout string

    The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
PUT /_slm/policy/{policy_id}/_execute
curl \
 --request PUT http://api.example.com/_slm/policy/{policy_id}/_execute
Response examples (200)
{
  "snapshot_name": "string"
}




















Stop snapshot lifecycle management Added in 7.6.0

POST /_slm/stop

Stop all snapshot lifecycle management (SLM) operations and the SLM plugin. This API is useful when you are performing maintenance on a cluster and need to prevent SLM from performing any actions on your data streams or indices. Stopping SLM does not stop any snapshots that are in progress. You can manually trigger snapshots with the run snapshot lifecycle policy API even if SLM is stopped.

The API returns a response as soon as the request is acknowledged, but the plugin might continue to run until in-progress operations complete and it can be safely stopped. Use the get snapshot lifecycle management status API to see if SLM is running.

Query parameters

  • The period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to -1.

  • timeout string

    The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to -1.

Responses

  • 200 application/json
    Hide response attribute Show response attribute object
    • acknowledged boolean Required

      For a successful response, this value is always true. On failure, an exception is returned instead.

POST /_slm/stop
curl \
 --request POST http://api.example.com/_slm/stop
Response examples (200)
{
  "acknowledged": true
}





















































































































Get transforms Added in 7.5.0

GET /_transform

Get configuration information for transforms.

Query parameters

  • Specifies what to do when the request:

    1. Contains wildcard expressions and there are no transforms that match.
    2. Contains the _all string or no identifiers and there are no matches.
    3. Contains wildcard expressions and there are only partial matches.

    If this parameter is false, the request returns a 404 status code when there are no matches or only partial matches.

  • from number

    Skips the specified number of transforms.

  • size number

    Specifies the maximum number of transforms to obtain.

  • Excludes fields that were automatically added when creating the transform. This allows the configuration to be in an acceptable format to be retrieved and then added to another cluster.

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • count number Required
    • transforms array[object] Required
      Hide transforms attributes Show transforms attributes object
      • Additional properties are allowed.

        Hide authorization attributes Show authorization attributes object
        • api_key object

          Additional properties are allowed.

          Hide api_key attributes Show api_key attributes object
          • id string Required

            The identifier for the API key.

          • name string Required

            The name of the API key.

        • roles array[string]

          If a user ID was used for the most recent update to the transform, its roles at the time of the update are listed in the response.

        • If a service account was used for the most recent update to the transform, the account name is listed in the response.

      • Time unit for milliseconds

      • Free text description of the transform.

      • dest object Required

        Additional properties are allowed.

        Hide dest attributes Show dest attributes object
      • A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • id string Required
      • latest object

        Additional properties are allowed.

        Hide latest attributes Show latest attributes object
        • sort string Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • unique_key array[string] Required

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • pivot object

        Additional properties are allowed.

        Hide pivot attributes Show pivot attributes object
        • Defines how to aggregate the grouped data. The following aggregations are currently supported: average, bucket script, bucket selector, cardinality, filter, geo bounds, geo centroid, geo line, max, median absolute deviation, min, missing, percentiles, rare terms, scripted metric, stats, sum, terms, top metrics, value count, weighted average.

        • group_by object

          Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported: date histogram, geotile grid, histogram, terms.

          Hide group_by attribute Show group_by attribute object
      • Additional properties are allowed.

        Hide retention_policy attribute Show retention_policy attribute object
        • time object

          Additional properties are allowed.

          Hide time attributes Show time attributes object
          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

          • max_age string Required

            A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      • settings object

        Additional properties are allowed.

        Hide settings attributes Show settings attributes object
        • Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

        • Defines if dates in the ouput should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

        • Specifies whether the transform should deduce the destination index mappings from the transform configuration.

        • Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

        • Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536.

        • unattended boolean

          If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

      • source object Required

        Additional properties are allowed.

        Hide source attributes Show source attributes object
        • index string | array[string] Required
        • Hide runtime_mappings attribute Show runtime_mappings attribute object
          • * object Additional properties

            Additional properties are allowed.

            Hide * attributes Show * attributes object
            • fields object

              For type composite

            • fetch_fields array[object]

              For type lookup

            • format string

              A custom format for date type runtime fields.

            • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

            • script object

              Additional properties are allowed.

            • type string Required

              Values are boolean, composite, date, double, geo_point, ip, keyword, long, or lookup.

        • query object

          A query clause that retrieves a subset of data from the source index.

          Additional properties are allowed.

      • sync object

        Additional properties are allowed.

        Hide sync attribute Show sync attribute object
        • time object

          Additional properties are allowed.

          Hide time attributes Show time attributes object
          • delay string

            A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

          • field string Required

            Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • version string
      • _meta object
        Hide _meta attribute Show _meta attribute object
        • * object Additional properties

          Additional properties are allowed.

GET /_transform
curl \
 --request GET http://api.example.com/_transform
Response examples (200)
{
  "count": 42.0,
  "transforms": [
    {
      "authorization": {
        "api_key": {
          "id": "string",
          "name": "string"
        },
        "roles": [
          "string"
        ],
        "service_account": "string"
      },
      "": 42.0,
      "description": "string",
      "dest": {
        "index": "string",
        "op_type": "index",
        "pipeline": "string",
        "routing": "string",
        "version_type": "internal"
      },
      "frequency": "string",
      "id": "string",
      "latest": {
        "sort": "string",
        "unique_key": [
          "string"
        ]
      },
      "pivot": {
        "aggregations": {},
        "group_by": {
          "additionalProperty1": {},
          "additionalProperty2": {}
        }
      },
      "retention_policy": {
        "time": {
          "field": "string",
          "max_age": "string"
        }
      },
      "settings": {
        "align_checkpoints": true,
        "dates_as_epoch_millis": true,
        "deduce_mappings": true,
        "docs_per_second": 42.0,
        "max_page_search_size": 42.0,
        "unattended": true
      },
      "source": {
        "index": "string",
        "runtime_mappings": {
          "additionalProperty1": {
            "fields": {},
            "fetch_fields": [
              {}
            ],
            "format": "string",
            "input_field": "string",
            "target_field": "string",
            "target_index": "string",
            "script": {},
            "type": "boolean"
          },
          "additionalProperty2": {
            "fields": {},
            "fetch_fields": [
              {}
            ],
            "format": "string",
            "input_field": "string",
            "target_field": "string",
            "target_index": "string",
            "script": {},
            "type": "boolean"
          }
        },
        "query": {}
      },
      "sync": {
        "time": {
          "delay": "string",
          "field": "string"
        }
      },
      "version": "string",
      "_meta": {
        "additionalProperty1": {},
        "additionalProperty2": {}
      }
    }
  ]
}