IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Elasticsearch version 6.3.1 Elasticsearch version 6.2.4 »
Elastic Docs ›Elasticsearch Guide [6.4] ›Release Notes

Elasticsearch version 6.3.0

edit

Elasticsearch version 6.3.0

edit

Known issues

edit

Upgrades from any 5.x version will fail for indexes which are prepared using the _synced flush API, or were automatically sync-flushed due to inactivity. If upgrading from those versions, please wait for 6.3.1 to be released.

Clusters with a Gold or Platinum license that are upgrading to 6.3 will need to explicitly set xpack.security.enabled: true in elasticsearch.yml to upgrade successfully. If this value is not set, the cluster will be unable to form after upgrade.

Also see 6.3.

Breaking changes

edit
Core
  • Rename the bulk thread pool to write thread pool #29593
Monitoring
  • By default when you install X-Pack, monitoring is enabled but data collection is disabled. To enable data collection, use the new xpack.monitoring.collection.enabled setting. You can update this setting by using the Cluster Update Settings API. For more information, see Monitoring Settings.
Packaging
  • Create keystore on package install #28928
Plugins
  • Remove silent batch mode from install plugin #29359
Security
  • The legacy XPackExtension extension mechanism has been removed and replaced with an SPI based extension mechanism that is installed and built as an elasticsearch plugin.

Breaking Java changes

edit
Aggregations
  • Add a shallow copy method to aggregation builders #28430 (issue: #27782)

Deprecations

edit
Analysis
  • Deprecate use of htmlStrip as name for HtmlStripCharFilter #27429
Core
  • Deprecate the index thread pool #29540
  • Add deprecation message for config prompt #28000 (issue: #27216)
Mapping
  • Deprecate the update_all_types option. #28284
Network
  • Networking: Deprecate http.enabled setting #29591 (issue: #12792)
  • Deprecate large max content length truncation #29339 (issue: #29337)
REST API
  • REST : deprecate field_data for Clear Indices Cache API #28943 (issue: #17804)
Search
  • Deprecate slicing on _uid. #29353
  • Generate deprecation warning for _search request with trailing tokens #29428 (issue: #28995)
Stats

New features

edit
Aggregations
  • Adds the ability to specify a format on composite date_histogram source #28310 (issue: #27923)
  • Calculate sum in Kahan summation algorithm in aggregations (#27807) #27848 (issue: #27807)
Geo
  • Add Z value support to geo_point and geo_shape #25738 (issue: #22917)
Ingest
  • Add ingest-attachment support for per document indexed_chars limit #28977 (issue: #28942)
Java High Level REST Client
  • Add ranking evaluation API to High Level Rest Client #28357
  • Add Indices Aliases API to the high level REST client #27876 (issue: #27205)
Java Low Level REST Client
  • Client: Wrap synchronous exceptions #28919 (issue: #28399)
Network
REST API
  • REST: Include suppressed exceptions on failures #29115 (issue: #23392)
Ranking
  • Add indices options support to _rank_eval #29386
Scripting
Search
  • Search - new flag: allow_partial_search_results #27906 (issue: #27435)
Task Management
  • Add new setting to disable persistent tasks allocations #29137

Enhancements

edit
Aggregations
  • Build global ordinals terms bucket from matching ordinals #30166 (issue: #30117)
  • Reject query if top hits result window exceeds index max result window #29199 (issue: #29190)
  • Optimize the composite aggregation for match_all and range queries #28745 (issue: #28688)
  • Always return the after_key in composite aggregation response #28358
  • Upgrade t-digest to 3.2 #28305 (issue: #28295)
CRUD
  • Don’t swallow exceptions on replication #31179 (issue: #28571)
Core
  • Implement Iterator#remove for Cache values iter #29633
  • Introduce analyze thread pool #29541
  • Add useful message when no input from terminal #29369 (issues: #29359, #29365)
  • Improve exception handling on TransportMasterNodeAction #29314 (issue: #1)
  • Add generic array support to AbstractObjectParser #28552
  • Introduce secure security manager to project #28453
  • XContent: Factor deprecation handling into callback #28449 (issue: #27955)
  • Add settings to control size and count of warning headers in responses #28427 (issue: #28301)
  • Trim down usages of ShardOperationFailedException interface #28312 (issue: #27799)
  • Enforce that java.io.tmpdir exists on startup #28217
  • Add Writeable.Reader support to TransportResponseHandler #28010 (issue: #26315)
Discovery-Plugins
  • Add information when master node left to DiscoveryNodes' shortSummary() #28197 (issue: #28169)
Distributed
  • Only log warning when actually failing shards #28558 (issue: #28534)
  • Allows failing shards without marking as stale #28054 (issue: #24841)
Engine
  • Never leave stale delete tombstones in version map #29619
  • Avoid side-effect in VersionMap when assertion enabled #29585
  • Enforce access to translog via engine #29542
  • ElasticsearchMergePolicy should extend from MergePolicyWrapper #29476
  • Track Lucene operations in engine explicitly #29357
  • Allow _update and upsert to read from the transaction log #29264 (issue: #26802)
  • Move trimming unsafe commits from the Engine constructor to Store #29260 (issue: #28245)
  • Add primary term to translog header #29227
  • Fold EngineDiskUtils into Store, for better lock semantics #29156 (issue: #28245)
  • Do not renew sync-id if all shards are sealed #29103 (issue: #27838)
  • Prune only gc deletes below the local checkpoint #28790
  • Do not optimize append-only operation if normal operation with higher seq# was seen #28787
  • Try if tombstone is eligable for pruning before locking on it’s key #28767
  • Simplify Engine.Searcher creation #28728
  • Revisit deletion policy after release the last snapshot #28627 (issue: #28140)
  • Index shard should roll generation via the engine #28537
  • Add lower bound for translog flush threshold #28382 (issues: #23779, #28350)
  • Untangle Engine Constructor logic #28245
  • Clean up commits when global checkpoint advanced #28140 (issue: #10708)
  • Replicate writes only to fully initialized shards #28049
  • Track deletes only in the tombstone map instead of maintaining as copy #27868
Geo
  • Add null_value support to geo_point type #29451 (issue: #12998)
Highlighting
Ingest
  • Reduce heap-memory usage of ingest-geoip plugin #28963 (issue: #28782)
  • Forbid trappy methods from java.time #28476
  • version set in ingest pipeline #27573 (issue: #27242)
Java High Level REST Client
  • Remove flatSettings support from request classes #29560
  • REST high-level client: add support for Indices Update Settings API [take 2] #29327 (issue: #27205)
  • REST high-level client: add force merge API #28896 (issue: #27205)
  • REST high-level client: add support for Indices Update Settings API #28892 (issue: #27205)
  • REST high-level client: add clear cache API #28866 (issue: #27205)
  • REST high-level client: add flush API #28852 (issue: #27205)
  • REST high-level client: add support for Rollover Index API #28698 (issue: #27205)
  • Add Cluster Put Settings API to the high level REST client #28633 (issue: #27205)
  • REST high-level Client: add missing final modifiers #28572
  • REST high-level client: add support for split and shrink index API #28425 (issue: #27205)
  • Java high-level REST : minor code clean up #28409
  • High level rest client : code clean up #28386
  • REST high-level client: add support for exists alias #28332 (issue: #27205)
  • Added Put Mapping API to high-level Rest client (#27205) #27869 (issue: #27205)
  • Add Refresh API for RestHighLevelClient #27799 (issue: #27205)
  • Add support for indices exists to REST high level client #27384
License
  • Require acknowledgement to start_trial license #30135 (issue: #30134)
Logging
  • Fix missing node id prefix in startup logs #29534
  • Do not swallow fail to convert exceptions #29043 (issue: #19573)
  • Add total hits to the search slow log #29034 (issue: #20648)
  • Remove interning from prefix logger #29031 (issue: #16831)
  • Log template creation and deletion #29027 (issue: #10795)
  • Disallow logger methods with Object parameter #28969
Mapping
  • Restrict Document list access in ParseContext #29463
  • Check presence of multi-types before validating new mapping #29316 (issue: #29313)
  • Validate regular expressions in dynamic templates. #29013 (issue: #24749)
Machine Learning
  • Synchronize long and short tests for periodicity #62
  • Improvements to trend modelling and periodicity testing for forecasting #7 (issue: #5)
  • Clean left behind model state docs #30659 (issue: #30551)
  • Hide internal Job update options from the REST API #30537 (issue: #30512)
Packaging
  • Configure heap dump path for archive packages #29130 (issue: #26755)
  • Configure error file for archive packages #29129 (issues: #29028, #29032)
  • Put JVM crash logs in the default log directory #29028 (issue: #13982)
  • Stop sourcing scripts during installation/removal #28918 (issue: #14630)
Plugins
  • Ensure that azure stream has socket privileges (#28751) #28773 (issue: #28662)
  • Plugins: Remove intermediate "elasticsearch" directory within plugin zips #28589
  • Plugins: Store elasticsearch and java versions in PluginInfo #28556
  • Plugins: Use one confirmation of all meta plugin permissions #28366
  • Replace jvm-example by two plugin examples #28339
  • Improve error message when installing an offline plugin #28298 (issue: #27401)
REST API
  • REST : Split RestUpgradeAction into two actions #29124 (issue: #29062)
  • Change BroadcastResponse from ToXContentFragment to ToXContentObject #28878 (issues: #27799, #3889)
  • Remove AcknowledgedRestListener in favour of RestToXContentListener #28724 (issue: #3889)
  • Standardize underscore requirements in parameters #27040 (issue: #26886)
Ranking
  • RankEvalRequest should implement IndicesRequest #29188
  • Move indices field from RankEvalSpec to RankEvalRequest #28341
  • Simplify RankEvalResponse output #28266
Recovery
  • Require translogUUID when reading global checkpoint #28587 (issue: #28435)
  • Do not ignore shard not-available exceptions in replication #28571 (issues: #28049, #28534)
  • Make primary-replica resync failures less lenient #28534 (issues: #24841, #28049, #28054)
  • Synced-flush should not seal index of out of sync replicas #28464 (issue: #10032)
  • Don’t refresh shard on activation #28013 (issue: #26055)
Rollup
  • Allow rollup job creation only if cluster is x-pack ready #30963 (issue: #30743)
Scripting
  • Modify Painless grammar to support right brackets as statement delimiters #29566
Search
  • Improve explanation in rescore #30629 (issue: #28725)
  • Add support to match_phrase query for zero_terms_query. #29598 (issue: #29344)
  • Improve similarity integration. #29187 (issues: #23208, #29035)
  • Store offsets in index prefix fields when stored in the parent field #29067 (issue: #28994)
  • Add QueryBuilders.matchNoneQuery(), #28679 #28680
  • Adds SpanGapQueryBuilder. Feature #27862 #28636 (issue: #27862)
  • Provide a better error message for the case when all shards failed #28333
  • Add ability to index prefixes on text fields #28290 (issue: #28222)
  • Add index_prefix option to text fields #28222
Settings
  • Enhance error for out of bounds byte size settings #29338 (issue: #29337)
  • Settings: Reimplement keystore format to use FIPS compliant algorithms #28255
Snapshot/Restore
  • Do not fail snapshot when deleting a missing snapshotted file #30332 (issue: #28322)
  • Update secure settings for the repository azure repository plugin #29319 (issue: #29135)
  • Use client settings in repository-gcs #28575
Stats
  • Add periodic flush count to flush stats #29360 (issue: #29125)
  • Enable selecting adaptive selection stats #28721
  • Add translog files age to Translog Stats (#28613) #28613 (issue: #28189)
Task Management
  • Make Persistent Tasks implementations version and feature aware #31045 (issues: #30731, #31020)
Transport API
  • Add remote cluster client #29495
  • Java api clean-up : consistency for shards_acknowledged getters #27819 (issue: #27784)
Watcher
  • Move watcher-history version setting to _meta field #30832 (issue: #30731)
  • Only allow x-pack metadata if all nodes are ready #30743 (issues: #30728, #30731)
ZenDiscovery
  • Add support for skippable named writeables #30948

Bug fixes

edit
Aggregations
  • Fix date and ip sources in the composite aggregation #29370
  • Pass through script params in scripted metric agg #29154 (issue: #28819)
  • Force depth_first mode execution for terms aggregation under a nested context #28421 (issue: #28394)
  • StringTerms.Bucket.getKeyAsNumber detection type #28118 (issue: #28012)
Allocation
  • Move allocation awareness attributes to list setting #30626 (issue: #30617)
  • Grammar matters.. #29462
  • Don’t break allocation if resize source index is missing #29311 (issue: #26931)
  • Add check when trying to reroute a shard to a non-data discovery node #28886
Audit
  • Fix audit index template upgrade loop #30779
Authentication
  • Security: fix dynamic mapping updates with aliases #30787 (issue: #30597)
  • [Security] Include an empty json object in an json array when FLS filters out all fields #30709 (issue: #30624)
  • Security: cleanup code in file stores #30348
  • Security: fix TokenMetaData equals and hashcode #30347
Authorization
  • Security: reduce garbage during index resolution #30180
CRUD
  • Bulk operation fail to replicate operations when a mapping update times out #30244
Core
  • Create default ES_TMPDIR on Windows #30325 (issues: #27609, #28217)
  • Core: Pick inner most parse exception as root cause #30270 (issues: #29373, #30261)
  • Fix the version ID for v5.6.10 (backport to 6.x). #29571
  • Fix the version ID for v5.6.10. #29570
  • Fix EsAbortPolicy to conform to API #29075 (issue: #19508)
  • Remove special handling for _all in nodes info #28971 (issue: #28797)
  • Handle throws on tasks submitted to thread pools #28667
  • Fix size blocking queue to not lie about its weight #28557 (issue: #28547)
  • Further minor bug fixes found by lgtm.com #27772
Engine
  • Add an escape hatch to increase the maximum amount of memory that IndexWriter gets. #31132 (issue: #31105)
  • Avoid self-deadlock in the translog #29520 (issues: #29401, #29509)
  • Close translog writer if exception on write channel #29401 (issue: #29390)
  • Harden periodically check to avoid endless flush loop #29125 (issues: #1, #2, #28350, #29097, #3)
  • Avoid class cast exception from index writer #28989
  • Maybe die before failing engine #28973 (issues: #27265, #28967)
  • Never block on key in LiveVersionMap#pruneTombstones #28736 (issue: #28714)
  • Inc store reference before refresh #28656
  • Replica recovery could go into an endless flushing loop #28350
Geo
  • Fix overflow error in parsing of long geohashes #29418 (issue: #24616)
  • Fix bwc in GeoDistanceQuery serialization #29325 (issues: #22876, #29301)
  • Allow using distance measure in the geo context precision #29273 (issue: #24807)
  • Fix incorrect geohash for lat 90, lon 180 #29256 (issue: #22163)
  • [GEO] Fix points_only indexing failure for GeoShapeFieldMapper #28774 (issues: #27415, #28744)
Index APIs
  • Propagate mapping.single_type setting on shrinked index #29202
  • Fix Parsing Bug with Update By Query for Stored Scripts #29039 (issue: #28002)
Ingest
  • Don’t allow referencing the pattern bank name in the pattern bank #29295 (issue: #29257)
  • Continue registering pipelines after one pipeline parse failure. #28752 (issue: #28269)
  • Guard accessDeclaredMembers for Tika on JDK 10 #28603 (issue: #28602)
  • Fix for bug that prevents pipelines to load that use stored scripts after a restart #28588
Java High Level REST Client
  • Bulk processor#awaitClose to close scheduler #29263
  • REST high-level client: encode path parts #28663 (issue: #28625)
  • Fix parsing of script fields #28395 (issue: #28380)
  • Move to POST when calling API to retrieve which support request body #28342 (issue: #28326)
Java Low Level REST Client
  • REST client: hosts marked dead for the first time should not be immediately retried #29230
License
  • Do not serialize basic license exp in x-pack info #30848
  • Require acknowledgement to start_trial license #30198 (issue: #30134)
Machine Learning
  • By-fields should respect model_plot_config.terms #86 (issue: #30004)
  • Function description for population lat_long results should be lat_long instead of mean #81 (issue: #80)
  • Fix error causing us to overestimate effective history length #66 (issue: #57)
  • Clearing JSON memory allocators #30 (issue: #26)
  • Fix sparse data edge cases for periodicity testing #28 (issue: #20)
  • Impose an absolute cutoff on the minimum variance #8 (issue: #488)
  • Check accesses in bounds when clearing recycled models #79 (issue: #76)
  • Set forecast progress to 100% and status finished in the case of insufficient history (data) #44
  • Add control message to start background persistence #19
  • Fail start up if state is missing #4
  • Do not log incorrect model memory limit #3
  • Fix background persistence - in 6.1 and 6.2 background persisted models were only visible after jobs were closed
  • Fix anomaly detection for buckets with fewer than two values when using the varp function
  • Fix typical values and model plot bounds when using the multivariate_by_fields option
  • Fix typical values, forecast confidence intervals and model plot bounds for models with multiple modes
Mapping
  • Delay _uid field data deprecation warning #30651 (issue: #30625)
  • Ignore null value for range field (#27845) #28116 (issue: #27845)
  • Fix a type check that is always false #27726
Network
Packaging
  • Fix #29057 CWD to ES_HOME does not change drive #29086
  • Allow overriding JVM options in Windows service #29044 (issue: #23484)
  • CLI: Close subcommands in MultiCommand #28954
  • Delay path expansion on Windows #28753 (issues: #27675, #28748)
  • Fix using relative custom config path #28700 (issue: #27610)
  • Disable console logging in the Windows service #28618 (issue: #20422)
Percolator
  • Fixed bug when non percolator docs end up in the search hits #29447 (issue: #29429)
  • Fixed a msm accounting error that can occur during analyzing a percolator query #29415 (issue: #29393)
  • Fix more query extraction bugs. #29388 (issues: #28353, #29376)
  • Fix some query extraction bugs. #29283
  • Fix percolator query analysis for function_score query #28854
  • Improved percolator’s random candidate query duel test #28840
  • Do not take duplicate query extractions into account for minimum_should_match attribute #28353 (issue: #28315)
Plugins
  • Template upgrades should happen in a system context #30621 (issue: #30603)
  • Plugins: Fix native controller confirmation for non-meta plugin #29434
  • Plugins: Fix module name conflict check for meta plugins #29146
  • Ensure that azure stream has socket privileges #28751 (issue: #28662)
  • Fix handling of mandatory meta plugins #28710 (issue: #28022)
  • Fix the ability to remove old plugin #28540 (issue: #28538)
REST API
  • Respect accept header on no handler #30383 (issue: #30329)
  • Protect against NPE in RestNodesAction #29059
  • REST api specs : remove unsupported wait_for_merge param #28959 (issue: #27158)
  • Rest api specs : remove unsupported parameter parent_node #28841
  • Rest api specs : remove a common param from nodes.usage.json #28835 (issue: #28226)
  • Missing timeout parameter from the REST API spec JSON files (#28200) #28328
Ranking
  • Fix NDCG for empty search results #29267
Recovery
  • Cancelling a peer recovery on the source can leak a primary permit #30318
  • ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled #30316
  • Do not log warn shard not-available exception in replication #30205 (issues: #28049, #28571)
  • Fix outgoing NodeID #28779 (issue: #28777)
  • Fsync directory after cleanup #28604 (issue: #28435)
Security
  • Reduces the number of object allocations made by X-Pack security when resolving the indices and aliases for a request (#30180)
  • Respects accept header on requests with no handler (#30383)
SQL
  • SQL: Verify GROUP BY ordering on grouped columns #30585 (issue: #29900)
  • SQL: Fix parsing of dates with milliseconds #30419 (issue: #30002)
  • SQL: Fix bug caused by empty composites #30343 (issue: #30292)
  • SQL: Correct error message #30138 (issue: #30016)
  • SQL: Add BinaryMathProcessor to named writeables list #30127 (issue: #30014)
Scripting
  • Correct class to name string conversion #28997
  • Painless: Fix For Loop NullPointerException #28506 (issue: #28501)
  • Scripts: Fix security for deprecation warning #28485 (issue: #28408)
Search
  • Ensure that index_prefixes settings cannot be changed #30967
  • Fix TermsSetQueryBuilder.doEquals() method #29629 (issue: #29620)
  • Fix binary doc values fetching in _search #29567 (issues: #26775, #29565)
  • Fixes query_string query equals timezone check #29406 (issue: #29403)
  • Fixed quote_field_suffix in query_string #29332 (issue: #29324)
  • Search: Validate script query is run with a single script #29304
  • Propagate ignore_unmapped to inner_hits #29261 (issue: #29071)
  • Restore tiebreaker for cross fields query #28935 (issues: #25115, #28933)
  • Fix (simple)_query_string to ignore removed terms #28871 (issues: #28855, #28856)
  • Search option terminate_after does not handle post_filters and aggregations correctly #28459 (issue: #28411)
  • Fix AIOOB on indexed geo_shape query #28458 (issue: #28456)
Settings
  • Archive unknown or invalid settings on updates #28888 (issue: #28609)
  • Settings: Introduce settings updater for a list of settings #28338 (issue: #28047)
Snapshot/Restore
Stats
  • Fix AdaptiveSelectionStats serialization bug #28718 (issue: #28713)
Suggesters
  • Fix merging logic of Suggester Options #29514
Transport API
  • Fix interoperability with < 6.3 transport clients #30971 (issue: #30731)
Watcher
  • Watcher: Prevent triggering watch when using activate API #30613
ZenDiscovery
  • Fsync state file before exposing it #30929
  • Do not return metadata customs by default #30857 (issue: #30731)
  • Use correct cluster state version for node fault detection #30810

Regressions

edit
Snapshot/Restore
  • S3 repo plugin populate SettingsFilter #30652

Upgrades

edit
Network
  • Update Netty to 4.1.16.Final #28345
Search
  • Upgrade to lucene-7.3.1 #30729
« Elasticsearch version 6.3.1 Elasticsearch version 6.2.4 »