NOTE: You are looking at documentation for an older release. For the latest information, see the current release documentation.
Elasticsearch version 6.5.0
editElasticsearch version 6.5.0
editAlso see Breaking changes in 6.5.
Breaking changes
edit- Aggregations
- Geo
- Mapping
-
- Disallow "enabled" attribute change for types in mapping update (#33566) #33933
- Plugins
Breaking Java changes
edit- Authorization
- Core
- Java High Level REST Client
- Plugins
-
- Fix generics in ScriptPlugin#getContexts() #33426
Deprecations
editNew features
edit- Aggregations
- Analysis
- Audit
-
-
Structured audit logging #31931 and #33894 (backport) (issue: #31046)
This introduces a new format for the logfile audit type which is output alongside the
previous one. Conforming to the out-of-the-box
log4j2.propertiesconfiguration, the new and deprecated audit formats are output to the<cluster_name>_access.logand<cluster_name>_audit.logfiles, respectively. See Logfile audit output for more details.
-
Structured audit logging #31931 and #33894 (backport) (issue: #31046)
This introduces a new format for the logfile audit type which is output alongside the
previous one. Conforming to the out-of-the-box
- Authentication
- Authorization
- CCR
- Index APIs
-
- Add an index setting to control TieredMergePolicy#deletesPctAllowed #32907
- Java High Level REST Client
-
- add start trial API to HLRC #32799
- Machine Learning
- Mapping
- Monitoring
-
- APM server monitoring #32515
- SQL
- Search
Enhancements
edit- Aggregations
-
- Rollup adding support for date field metrics (#34185) #34200 (issue: #34185)
- Add early termination support for min/max aggregations #33375
- Add early termination support to BucketCollector #33279
- Add interval response parameter to AutoDateInterval histogram #33254
- Scripted metric aggregations: add deprecation warning and system property to control legacy params #31597 (issues: #29328, #30111)
- Allocation
- Analysis
- Audit
- Authentication
-
- Security: reduce memory usage of DnRoleMapper #34250 (issue: #34237)
- Security: upgrade unboundid ldapsdk to 4.0.8 #34247 (issue: #33175)
- [Kerberos] Add realm name & UPN to user metadata #33338
- [SECURITY] Set Auth-scheme preference #33156 (issue: #32699)
- Token API supports the client_credentials grant #33106
- Authorization
- Beats Plugin
-
- [Monitoring] Update beats template to include apm-server metrics #33286
- CRUD
- Circuit Breakers
-
- Whitelisting / from Circuit Breaker Exception (#32325) #32666
- Core
- Discovery-Plugins
- Distributed
- Docs Infrastructure
-
- Docs: Allow snippets to have line continuation #32649
- Engine
-
- Do not alloc full buffer for small change requests #35158
- Fill LocalCheckpointTracker with Lucene commit #34474 (issues: #0, #2, #33656)
- Lock down Engine.Searcher #34363 (issue: #34357)
- Fold EngineSearcher into Engine.Searcher #34082
- Build DocStats from SegmentInfos in ReadOnlyEngine #34079 (issue: #33903)
- Move CompletionStats into the Engine #33847 (issue: #33835)
- Move DocsStats into Engine #33835
- Add read-only Engine #33563 (issues: #32844, #32867)
- Allow engine to recover from translog upto a seqno #33032 (issue: #32867)
- Index APIs
- Ingest
-
- ingest: processor stats #34724 (issue: #34202)
- ingest: better support for conditionals with simulate?verbose #34155
- ingest: correctly measure chained pipeline stats #33912
- ingest: support simulate with verbose for pipeline processor #33839
- [ingest] geo-ip performance improvements #33029
- ingest: Introduce the dissect processor #32884
- INGEST: Add Pipeline Processor #32473 (issue: #31842)
- Ingest: Add conditional per processor #32398 (issue: #21248)
- Introduce the dissect library #32297
- INGEST: Enable default pipelines #32286 (issue: #21101)
- INGEST: Implement Drop Processor #32278 (issue: #23726)
- Ingest: Support integer and long hex values in convert #32213 (issue: #32182)
- Add region ISO code to GeoIP Ingest plugin #31669
- Java High Level REST Client
-
- HLRC: Delete role API #34620
- [HLRC] Add support for Delete role mapping API #34531
- [HLRC] Add Start/Stop Watch Service APIs. #34317
- HLRC: ML Add preview datafeed api #34284 (issue: #29827)
- HLRC: ML Adding get datafeed stats API #34271 (issue: #29827)
- HLRC: Deactivate Watch API #34192 (issues: #29827, #33988)
- Create/Update role mapping API #34171 (issue: #33745)
- HLRC: Get SSL Certificates API #34135
- [ML][HLRC] Replace REST-based ML test cleanup with the ML client #34109
- HLRC: Add activate watch action #33988 (issue: #29827)
- Add support for ack watch to the HLRC. #33962 (issue: #29827)
- HLRC: Add throttling for update & delete-by-query #33951
- HLRC: ML Stop datafeed API #33946 (issue: #29827)
- HLRC: Add get rollup job #33921
- HLRC: ML start data feed API #33898 (issue: #29827)
- HLRC: Add support for reindex rethrottling #33832
-
HLRC: Reindex should support
requests_per_secondsparameter #33808 - HLRC: Delete ML calendar #33775 (issue: #29827)
- HLRC: Get ML calendars #33760 (issue: #29827)
- [HLRC] Support for role mapper expression dsl #33745
- [HLRC][ML] Add ML get datafeed API to HLRC #33715 (issue: #29827)
- REST client: introduce a strict deprecation mode #33708 (issue: #33534)
- [HLRC][ML] Add ML delete datafeed API to HLRC #33667 (issue: #29827)
- HLRC: Add support for XPack Post Start Basic Licence API #33606 (issue: #29827)
- [HLRC][ML] Add ML put datafeed API to HLRC #33603 (issue: #29827)
- Create a WatchStatus class for the high-level REST client. #33527
- HLRC: ML Delete Forecast API #33526 (issue: #29827)
- Add create rollup job api to high level rest client #33521 (issues: #29827, #32703)
- HLRC: add change password API support #33509 (issue: #33481)
- HLRC: ML Forecast Job #33506 (issue: #29827)
- HLRC: add enable and disable user API support #33481 (issue: #29827)
- HLRC: Add ML get categories API #33465 (issue: #29827)
- HLRC: ML Post Data #33443 (issue: #29827)
- add start trial API to HLRC #33406
- HLRC: ML Update Job #33392 (issue: #29827)
- HLRC: Add ML get influencers API #33389 (issue: #29827)
- HLRC: ML PUT Calendar #33362 (issue: #29827)
- HLRC: Add ML get overall buckets API #33297 (issue: #29827)
- HLRC: create base timed request class #33216
- HLRC: add client side RefreshPolicy #33209
- HLRC: ML Flush job #33187 (issue: #29827)
- HLRC: Adding ML Job stats #33183 (issue: #29827)
- HLRC: Use Optional in validation logic #33104
- HLRC: Add ML Get Records API #33085 (issue: #29827)
- HLRC: Add ML Get Buckets API #33056 (issue: #29827)
- GraphClient for the high level REST client and associated tests. #33025 (issue: #29827)
- HLRC: Clear ML data after client tests #33023 (issue: #32993)
- HLRC: Add ML Get Job #32960 (issue: #29827)
- HLRC: ML Close Job #32943 (issue: #29827)
- HLRC: Create server agnostic request and response #32912
- Add GetRollupCaps API to high level rest client #32880 (issues: #29827, #32703)
- HLRC: adding machine learning open job #32860 (issue: #29827)
- HLRC: Refactor WatchStatus #32842 (issue: #29827)
- HLRC: adding machine learning delete job #32820 (issue: #29827)
- HLRC: Refactor WatchStatus and implement activate watch #32802 (issue: #29827)
- REST high-level client: add delete by query API #32782 (issues: #27205, #32679, #32760)
- REST high-level client: add update by query API #32760 (issues: #27205, #32679)
- HLRC: migration get assistance API #32744 (issue: #29827)
- Adding ML HLRC wrapper and put_job API call #32726
- Add create rollup job api to high level rest client #32703 (issue: #29827)
- REST high-level client: add reindex API #32679 (issue: #27205)
- HLRC: Add Delete License API #32586 (issue: #29827)
- Rest HL client: Add get license action #32438 (issue: #29827)
- HLRC: Add delete watch action #32337 (issue: #29827)
- HLRest: add xpack put user API #32332 (issue: #29827)
- Rest HL client: Add put license action #32214 (issue: #29827)
- Add Restore Snapshot High Level REST API #32155 (issue: #27205)
- Add put stored script support to high-level rest client #31323 (issue: #27205)
- Logging
-
- Logging: Make node name consistent in logger #31588
- Machine Learning
-
- Adding support for lazy nodes #34538 (issue: #29991)
- Add an ingest pipeline definition to structure finder #34350
- Add a timeout option to file structure finder #34117
- Allow asynchronous job deletion #34058 (issue: #32836)
- Make certain ML node settings dynamic #33961 (issue: #33565)
- Display integers without .0 in file structure field stats #33947
- Return both Joda and Java formats from structure finder #33900
- Adding node_count to ML Usage #33863 (issue: #33850)
- Delete custom index if the only contained job is deleted #33788 (issue: #30075)
- Allow overrides for some file structure detection decisions #33630
- Minor improvements to categorization Grok pattern creation #33353
- Delete forecast API #33218 (issue: #31134)
- Perform anomaly detection on features derived from multiple bucket values to improve the robustness of detection with respect to misconfigured bucket lengths and to improve the detection of long lasting anomalies. #175
- Support decomposing a time series into a piecewise linear trend and with piecewise constant scaling of the periodic components. This extends our decomposition functionality to handle the same types of change points that our modelling capabilities do. #198
- Increased independence of anomaly scores across partitions. #182
- Avoid potential false positives at model start up when first detecting new components of the time series decomposition. #218
-
Add a new label (
multi_bucket_impact) to record level anomaly results. The value is on a scale of -5 to +5 where -5 means the anomaly is purely single bucket and +5 means the anomaly is purely multi bucket. #34233, #230 - Improve our ability to detect change points in the presence of outliers. #265
- Mapping
- Monitoring
- Network
-
-
Pass the host name on as
server_nameif proxy mode is on #34559 - Bad regex in CORS settings should throw a nicer error #34035
- Add sni name to SSLEngine in netty transport (#33144) #33513 (issue: #32517)
- Add sni name to SSLEngine in netty transport #33144 (issue: #32517)
- Add proxy support to RemoteClusterConnection #33062 (issues: #31840, #32517)
- Use a dedicated ConnectionManger for RemoteClusterConnection #32988 (issue: #31835)
-
Pass the host name on as
- Packaging
-
- Add Ubuntu 18.04 to packaging tests #34139
- Ranking
- Recovery
-
- Use soft-deleted docs to resolve strategy for engine operation #35230 (issues: #0, #1, #33656, #34474)
- Propagate auto_id_timestamp in primary-replica resync #33964 (issue: #33693)
- Restore local history from translog on promotion #33616 (issues: #32867, #33473)
- Reset replica engine to global checkpoint on promotion #33473 (issue: #32867)
- Bootstrap a new history_uuid when force allocating a stale primary #33432 (issue: #26712)
- Integrates soft-deletes into Elasticsearch #33222 (issues: #29530, #30086, #30120, #30335, #30522, #31106)
- Rollup
- SQL
-
- SQL: Improve CircuitBreaker logic for SqlParser #35300 (issue: #35299)
- SQL: Optimizer rule for folding nullable expressions #35080 (issue: #34826)
-
SQL: Improve painless script generated from
IN#35055 (issue: #34750) - SQL: Implement CAST between STRING and IP #34949 (issue: #34799)
- SQL: Fix function args verification and error msgs #34926 (issues: #33469, #34752)
- SQL: Introduce ODBC mode, similar to JDBC #34825 (issue: #34720)
- SQL: Introduce support for IP fields #34758 (issue: #32499)
-
SQL: Implement null handling for
IN(v1, v2, ...)#34750 (issue: #34582) - SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way #34736 (issue: #30009)
- SQL: Support pattern against compatible indices #34718 (issues: #31611, #31837, #33803)
- SQL: Allow min/max aggregates on date fields #34699 (issue: #34477)
- SQL: Introduce support for NULL values (#34573) #34640 (issue: #32079)
- SQL: return constants for all matching records in constants-containing SELECTs #34576 (issue: #31863)
- SQL: Functions enhancements (OCTET_LENGTH function, order functions alphabetically, RANDOM function docs) #34101 (issue: #33477)
- SQL: Internal refactoring of operators as functions #34097 (issue: #33975)
- SQL: Remove more ANTLR4 grammar ambiguities #34074 (issue: #33854)
- SQL: Move away internally from JDBCType to SQLType #33913 (issue: #33904)
- SQL: Fix ANTL4 Grammar ambiguities. #33854 (issue: #31885)
- SQL: Better handling of number parsing exceptions #33776 (issue: #33622)
- SQL: Grammar tweak for number declarations #33767 (issue: #33765)
- SQL: Return functions in JDBC driver metadata #33672 (issue: #33671)
- SQL: Make Literal a NamedExpression #33583 (issue: #33523)
- SQL: Improve alias vs index resolution #33393 (issue: #33363)
- SQL: Align SYS TABLE for ODBC SQL_ALL_* args #33364 (issue: #33312)
- SQL: Show/desc commands now support table ids #33363 (issue: #33294)
- SQL: Support multi-index format as table identifier #33278
- SQL: Multiple indices pattern #33162
- SQL: skip uppercasing/lowercasing function tests for AZ locales as well #32910 (issue: #32589)
- SQL: test coverage for JdbcResultSet #32813 (issue: #32078)
- SQL: Added support for string manipulating functions with more than one parameter #32356 (issue: #31604)
- Scripting
- Search
-
- Replace version with reader cache key in IndicesRequestCache #34189 (issues: #27650, #33473)
- Handle terms query when detecting if a query can match nested docs #34072 (issue: #34067)
- Add a limit for graph phrase query expansion #34031
- Clarify RemoteClusterService#groupIndices behaviour #33899
- Add nested and object fields to field capabilities response #33803 (issue: #33237)
-
Introduce a
search_throttledthreadpool #33732 - Upgrade remote cluster settings #33537 (issues: #33413, #33536)
- Remove unsupported group_shard_failures parameter #33208 (issue: #32598)
- Profiler: Don’t profile NEXTDOC for ConstantScoreQuery. #33196 (issue: #23430)
- Change query field expansion #33020 (issues: #31655, #31798)
-
Expose
max_concurrent_shard_requestsin_msearch#33016 (issue: #31877) - Search: Support of wildcard on docvalue_fields #32980 (issues: #26299, #26390)
- Ignore script fields when size is 0 #31917 (issue: #31824)
- Security
-
- Generate non-encrypted license public key #34626
- Security: don’t call prepare index for reads #34568 (issues: #33205, #34246)
- Enable security automaton caching #34028
- Add Debug/Trace logging to token service #34022
- Security index expands to a single replica #33131 (issues: #29712, #29933)
- Introduce fips_mode setting and associated checks #32326
- Settings
- Snapshot/Restore
- Stats
- Store
- Suggesters
- Watcher
-
- Watcher: Reduce script cache churn by checking for mustache tags #33978 (issue: #29280)
- [Watcher] Improved error messages for CronEvalTool #32800 (issue: #32735)
- Watcher: Use Bulkprocessor in HistoryStore/TriggeredWatchStore #32490
- Watcher: migrate PagerDuty v1 events API to v2 API #32285 (issue: #32243)
- ZenDiscovery
-
- Allow excluding folder names when scanning for dangling indices #34349
Bug fixes
edit- Aggregations
-
- Fix handling of empty keyword in terms aggregation #34457 (issue: #34434)
- Check self references in metric agg after last doc collection (#33593) #34001
- Unmapped aggs should not run pipelines if they delegate reduction #33528 (issue: #33514)
- For filters aggregations, make sure that rewrites preserve other_bucket. #32921 (issue: #32834)
- Allocation
- Analysis
- Authentication
- Authorization
- CAT APIs
- CRUD
- Circuit Breakers
- Core
-
- Fix AutoQueueAdjustingExecutorBuilder settings validation #33922
- Fix Javadoc issues in 6.x for JDK11 #33579
- Core: Add java time xcontent serializers #33120 (issue: #31853)
- Protect scheduler engine against throwing listeners #32998
- Fix content type detection with leading whitespace #32632 (issue: #32357)
- Distributed
- Engine
- Geo
- Index APIs
-
-
Make XContentBuilder in AliasActions build
is_write_indexfield #35071 - Do not update number of replicas on no indices #34481
- [Security] Get Alias API wildcard exclusion with Security #34144 (issues: #33518, #33805)
- Allow to clear the fielddata cache per field #33807 (issue: #33798)
- CORE: Make Pattern Exclusion Work with Aliases #33518 (issue: #33395)
- Fix IndexMetaData loads after rollover #33394 (issue: #33316)
- Copy missing segment attributes in getSegmentInfo #32396
-
Make XContentBuilder in AliasActions build
- Ingest
- Java High Level REST Client
-
- HLRC: Fixing bug when getting a missing pipeline #34286 (issue: #34119)
- Aggregations/HL Rest client fix: missing scores #32774 (issue: #32770)
- HLRC: Ban LoggingDeprecationHandler #32756 (issue: #32151)
- HLRC: Move commercial clients from XPackClient #32596
- High-level client: fix clusterAlias parsing in SearchHit #32465
- REST high-level client: parse back _ignored meta field #32362
- License
-
- Address license state update/read thread safety #33396
- Logging
- Machine Learning
- Mapping
-
- Fix field mapping updates with similarity #33634 (issue: #33611)
- Ensure that _exists queries on keyword fields use norms when they’re available. #33006
- Make sure that field collapsing supports field aliases. #32648 (issue: #32623)
- Improve the error message when an index is incompatible with field aliases. #32482
- Monitoring
-
- Typo in x-pack template for thread_pool.management #34224
- Network
-
- NETWORKING: Add SSL Handler before other Handlers #34636 (issue: #33998)
- Handle null SSLSessions during invalidation #34130 (issue: #32124)
- Support PKCS#11 tokens as keystores and truststores #34063 (issue: #11)
- Parse PEM Key files leniantly #33173 (issue: #33168)
- NETWORKING: http.publish_host Should Contain CNAME #32806 (issue: #22029)
- NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS #32764 (issue: #28858)
- Release requests in cors handle #32410
- Release requests in cors handler #32364
- Packaging
- Percolator
-
- Ignore date ranges containing now when pre-processing a percolator query #35160
- REST API
- Recovery
- Rollup
-
- [Rollup] Proactively resolve index patterns in RollupSearch endoint #34930 (issue: #34828)
- Address BWC bug due to default metrics in (#34764) #34810 (issue: #34764)
- Allowing {index}/_xpack/rollup/data to accept comma delimited list #34115
- [Rollup] Fix Caps Comparator to handle calendar/fixed time #33336 (issue: #32052)
- [Rollup] Better error message when trying to set non-rollup index #32965
- [Rollup] Return empty response when aggs are missing #32796 (issue: #32256)
- [Rollup] Improve ID scheme for rollup documents #32558 (issue: #32372)
- SQL
-
- SQL: Fix null handling for AND and OR in SELECT #35277 (issue: #35240)
-
SQL: Handle null literal for AND and OR in
WHERE#35236 (issue: #35088) - SQL: Introduce NotEquals node to simplify expressions #35234 (issues: #35210, #35233)
- SQL: handle wildcard expansion on incorrect fields #35134 (issue: #35092)
- SQL: Fix null handling for IN ⇒ painless script #35124 (issues: #35108, #35122)
- SQL: Register missing processors #35121 (issue: #35119)
- SQL: Fix NPE thrown if HAVING filter evals to null #35108 (issue: #35107)
- SQL: Proper handling of nested fields at the beginning of the columns list #35068 (issue: #32951)
- SQL: Fix incorrect AVG data type #34948 (issue: #33773)
-
SQL: Add
CASTandCONVERTtoSHOW FUNCTIONS#34940 (issue: #34939) - SQL: Handle aggregation for null group #34916 (issue: #34896)
- SQL: Provide null-safe scripts for Not and Neg #34877 (issue: #34848)
- SQL: Return error with ORDER BY on non-grouped. #34855 (issue: #34590)
- SQL: Fix queries with filter resulting in NO_MATCH #34812 (issue: #34613)
-
SQL: Fix edge case:
<field> IN (null)#34802 (issue: #34750) - SQL: Verifier allows aliases aggregates for sorting #34773 (issue: #34607)
- SQL: the SSL default configuration shouldn’t override the https protocol if used #34635 (issue: #33817)
- JDBC: Fix artifactId in pom #34478 (issue: #34399)
-
SQL: Fix grammar for
*in arithm expressions #34176 (issue: #33957) - SQL: Fix function resolution #34137 (issue: #34114)
- SQL: Fix query translation of GroupBy with Having #34010 (issue: #33520)
- SQL: Prevent StackOverflowError when parsing large statements #33902 (issue: #32942)
- SQL: Fix issue with options for QUERY() and MATCH(). #33828 (issue: #32602)
- SQL: Return correct catalog separator in JDBC #33670 (issue: #33654)
- SQL: Fix result column names for CAST #33604 (issue: #33571)
- SQL: Fix result column names for arithmetic functions #33500 (issues: #14, #31869)
- SQL: Fix bug in REPLACE function. Adds more tests to all string functions #33478
- SQL: handle differently security connection related errors in the CLI #33255 (issue: #33230)
- SQL: prevent duplicate generation for repeated aggs #33252 (issue: #30287)
- SQL: Enable aggregations to create a separate bucket for missing values #32832 (issue: #32831)
- SQL: Bug fix for the optional "start" parameter usage inside LOCATE function #32576 (issue: #32554)
- SQL: Minor fix for javadoc #32573 (issue: #32553)
- Scripting
-
- Scripting: Add back lookup vars in score script #34833
- Scripting: Add back params._source access in scripted metric aggs #34777 (issue: #33884)
- Test: Fix last reference to SearchScript #34731 (issue: #34683)
- Ensure map keys cannot be self referencing #34569
- [Painless] Add a Map for java names to classes for use in the custom classloader #34424
- [Painless] Allow statically imported methods without whitelisted class #34370
- Painless: Remove caching of Painless scripts #34116
- Painless: Fix Bindings Bug #33274
- Painless: Fix Semicolon Regression #33212 (issue: #33193)
- Scripting: Fix painless compiler loader to know about context classes #32385
- Search
-
- Fix inner_hits retrieval when stored fields are disabled #34652 (issues: #32941, #33018)
- Fix cross fields mode of the query_string query #34216 (issue: #34215)
- Support string-style queries on metadata fields when reasonable. #34089 (issue: #34062)
- Improves doc values format deprecation message #33576 (issue: #33572)
- Fix nested _source retrieval with includes/excludes #33180 (issues: #33163, #33170)
- Fix quoted exists query #33019 (issue: #28922)
- Fix multi fields empty query #33017 (issue: #33009)
- XContentBuilder to handle BigInteger and BigDecimal #32888 (issue: #32395)
- Do NOT allow termvectors on nested fields #32728 (issues: #21625, #32652)
- Cross-cluster search: preserve cluster alias in shard failures #32608
- Security
- Settings
- Snapshot/Restore
- Suggesters
- Transport API
-
- Fix serialization of empty field capabilities response #33263
- Watcher
-
- watcher: Fix integration tests to ensure correct start/stop of Watcher #35271 (issues: #29877, #30705, #33291, #34448, #34462)
- Make Watcher validation message copy/pasteable #33497 (issue: #33369)
- Watcher: Reload properly on remote shard change #33167
- Watcher: Fix race condition when reloading watches #33157
- Guard against null in email admin watches #32923 (issue: #32590)
- Watcher: Properly find next valid date in cron expressions #32734
- ZenDiscovery