Elasticsearch version 7.7.0
editElasticsearch version 7.7.0
editAlso see Breaking changes in 7.7.
Known issues
edit-
Slow loggers can cause Log4j loggers to leak over time. When a new index is created, a new Log4j logger is associated with it. However, when an index is deleted, Log4j keeps an internal reference to its loggers that results in a memory leak #57216 (issue: #56171)
This issue is fixed in Elasticsearch 6.8.10 and 7.7.1.
-
SQL: If a
WHERE
clause contains at least two relational operators joined byAND
, of which one is a comparison (<=
,<
,>=
,>
) and another one is an inequality (!=
,<>
), both against literals or foldable expressions, the inequality will be ignored. The workaround is to substitute the inequality with aNOT IN
operator.We have fixed this issue in Elasticsearch 7.10.1 and later versions. For more details, see #65488.
Breaking changes
edit- Highlighting
- Infra/Core
- Mapping
- Search
Breaking Java changes
editDeprecations
edit- Allocation
- Authentication
- Engine
- Features/Features
- Infra/Core
- Infra/REST API
- Machine Learning
-
- Renaming inference processor field field_mappings to new name field_map #53433
- Search
New features
editEnhancements
edit- Aggregations
-
- Fixed rewrite of time zone without DST #54398
- Try to save memory on aggregations #53793
- Speed up partial reduce of terms aggregations #53216 (issue: #51857)
- Simplify SiblingPipelineAggregator #53144
- Add histogram field type support to boxplot aggs #52265 (issues: #33112, #52233)
- Percentiles aggregation validation checks for range #51871 (issue: #51808)
- Begin moving date_histogram to offset rounding (take two) #51271 (issues: #50609, #50873)
- Password-protected Keystore Feature Branch PR #51123 (issues: #32691, #49340)
- Implement top_metrics agg #51155 (issue: #48069)
- Bucket aggregation circuit breaker optimization. #46751
- Analysis
-
- Removes old Lucene’s experimental flag from analyzer documentations #53217
- Authentication
-
- Add exception metadata for disabled features #52811 (issues: #47759, #52311, #55255)
- Validate role templates before saving role mapping #52636 (issue: #48773)
- Add support for secondary authentication #52093
- Expose API key name to the ingest pipeline #51305 (issues: #46847, #49106)
- Disallow Password Change when authenticated by Token #49694 (issue: #48752)
- Authorization
-
- Allow kibana_system to create and invalidate API keys on behalf of other users #53824 (issue: #48716)
- Add "grant_api_key" cluster privilege #53527 (issues: #48716, #52886)
- Giving kibana user privileges to create custom link index #53221 (issue: #59305)
- Allow kibana to collect APM telemetry in background task #52917 (issue: #50757)
- Add the new maintenance privilege containing 4 actions (#29998) #50643
- Cluster Coordination
- Distributed
- Engine
-
- Restore off-heap loading for term dictionary in ReadOnlyEngine #53713 (issues: #43158, #51247)
- Separate translog from index deletion conditions #52556
- Always rewrite search shard request outside of the search thread pool #51708 (issue: #49601)
-
Move the terms index of
_id
off-heap. #52405 (issue: #42838) - Cache completion stats between refreshes #51991 (issue: #51915)
- Use local checkpoint to calculate min translog gen for recovery #51905 (issue: #49970)
- Features/CAT APIs
- Features/Features
-
- Enable deprecation checks for removed settings #53317
- Features/ILM+SLM
-
- Use Priority.IMMEDIATE for stop ILM cluster update #54909
- Add cluster update timeout on step retry #54878
- Hide ILM & SLM history aliases #53564
- Avoid race condition in ILMHistorySotre #53039 (issues: #50353, #52853)
- Make FreezeStep retryable #52540
- Make DeleteStep retryable #52494
- Allow forcemerge in the hot phase for ILM policies #52073 (issue: #43165)
- Stop policy on last PhaseCompleteStep instead of TerminalPolicyStep #51631 (issue: #48431)
- Convert ILM and SLM histories into hidden indices #51456
- Make UpdateSettingsStep retryable #51235 (issues: #44135, #48183)
- Expose master timeout for ILM actions #51130 (issue: #44136)
- Wait for active shards on rolled index in a separate step #50718 (issues: #44135, #48183)
- Adding best_compression #49974
- Features/Indices APIs
- Features/Ingest
- Features/Java High Level REST Client
- Features/Monitoring
- Features/Watcher
- Infra/Core
-
- Enable helpful null pointer exceptions #54853
- Allow keystore add to handle multiple settings #54229 (issue: #54191)
- Report parser name and location in XContent deprecation warnings #53805
- Report parser name and location in XContent deprecation warnings #53752
- Deprecate all variants of a ParseField with no replacement #53722
- Allow specifying an exclusive set of fields on ObjectParser #52893
- Support joda style date patterns in 7.x #52555
- Implement hidden aliases #52547 (issue: #52304)
- Allow ObjectParsers to specify required sets of fields #49661 (issue: #48824)
- Infra/Logging
- Infra/Packaging
- Infra/Plugins
- Infra/Scripting
-
- Scripting: Context script cache unlimited compile #53769 (issue: #50152)
- Scripting: Increase ingest script cache defaults #53765 (issue: #50152)
- Scripting: Per-context script cache, default off (#52855) #53756 (issues: #50152, #52855)
- Scripting: Per-context script cache, default off #52855 (issue: #50152)
- Improve Painless compilation performance for nested conditionals #52056
- Scripting: Add char position of script errors #51069 (issue: #50993)
- Infra/Settings
- License
- Machine Learning
-
- Stratified cross validation split for classification #54087
- Data frame analytics data counts #53998
- Verify that the field is aggregatable before attempting cardinality aggregation #53874 (issue: #53876)
- Adds multi-class feature importance support #53803
- Data frame analytics analysis stats #53788
- Add a model memory estimation endpoint for anomaly detection #53507 (issue: #53219)
- Adds new default_field_map field to trained models #53294
- Improve DF analytics audits and logging #53179
- Add indices_options to datafeed config and update #52793 (issue: #48056)
- Parse and report memory usage for DF Analytics #52778
- Adds the class_assignment_objective parameter to classification #52763 (issue: #52427)
- Add reason to DataFrameAnalyticsTask updateState log message #52659 (issue: #52654)
- Add support for multi-value leaves to the tree model #52531
- Make ml internal indices hidden #52423 (issue: #52420)
- Add _cat/ml/data_frame/analytics API #52260 (issue: #51413)
- Adds feature importance option to inference processor #52218
- Switch poor categorization audit warning to use status field #52195 (issues: #50749, #51146, #51879)
- Retry persisting DF Analytics results #52048
- Improve multiline_start_pattern for CSV in find_file_structure #51737
- Add _cat/ml/trained_models API #51529 (issue: #51414)
- Add GET _cat/ml/datafeeds #51500 (issue: #51411)
- Use CSV ingest processor in find_file_structure ingest pipeline #51492 (issue: #56038)
- Add _cat/ml/anomaly_detectors API #51364
- Add tags url param to GET #51330
- Add parsers for inference configuration classes #51300
- Make datafeeds work with nanosecond time fields #51180 (issue: #49889)
- Adds support for a global calendars #50372 (issue: #45013)
- Speed up computation of feature importance 1005
- Improve initialization of learn rate for better and more stable results in regression and classification #948
- Add number of processed training samples to the definition of decision tree nodes #991
- Add new model_size_stats fields to instrument categorization #948, #51879 (issue: #50749)
- Improve upfront memory estimation for all data frame analyses, which were higher than necessary. This will improve the allocation of data frame analyses to cluster nodes #1003
- Upgrade the compiler used on Linux from gcc 7.3 to gcc 7.5, and the binutils used in the build from version 2.20 to 2.34 #1013
- Add instrumentation of the peak memory consumption for data frame analytics jobs #1022
- Remove all memory overheads for computing tree SHAP values #1023
- Distinguish between empty and missing categorical fields in classification and regression model training #1034
- Add instrumentation information for supervised learning data frame analytics jobs #1031
- Add instrumentation information for outlier detection data frame analytics jobs #1068
- Write out feature importance for multi-class models #1071
- Enable system call filtering to the native process used with data frame analytics #1098
- Mapping
- Network
- Ranking
- SQL
-
- JDBC debugging enhancement #53880
- Transfer version compatibility decision to the server #53082 (issue: #52766)
- Use a proper error message for queries directed at empty mapping indices #52967 (issue: #52865)
- Use calendar_interval of 1d for HISTOGRAMs with 1 DAY intervals #52749 (issue: #52713)
- Use a calendar interval for histograms over 1 month intervals #52586 (issue: #51538)
- Make parsing of date more lenient #52137 (issue: #49379)
- Enhance timestamp escaped literal parsing #52097 (issue: #46069)
- Handle uberjar scenario where the ES jdbc driver file is bundled in another jar #51856 (issue: #50201)
- Verify Full-Text Search functions not allowed in SELECT #51568 (issue: #47446)
- Extend the optimisations for equalities #50792 (issue: #49637)
- Add trace logging for search responses coming from server #50530
- Extend DATE_TRUNC to also operate on intervals(elastic - #46632 ) #47720 (issue: #46632)
- Search
-
- HLRC: Don’t send defaults for SubmitAsyncSearchRequest #54200
- Reduce performance impact of ExitableDirectoryReader #53978 (issues: #52822, #53166, #53496)
- Add heuristics to compute pre_filter_shard_size when unspecified #53873 (issue: #39835)
- Add async_search get and delete APIs to HLRC #53828 (issue: #49091)
- Increase step between checks for cancellation #53712 (issues: #52822, #53496)
- Refine SearchProgressListener internal API #53373
- Check for query cancellation during rewrite #53166 (issue: #52822)
- Implement Cancellable DirectoryReader #52822
- Address MinAndMax generics warnings #52642 (issue: #49092)
- Clarify when shard iterators get sorted #52633
-
Generalize how queries on
_index
are handled at rewrite time #52486 (issues: #49254, #49713) - Remove the query builder serialization from QueryShardException message #51885 (issues: #48910, #51843)
- Short circuited to MatchNone for non-participating slice #51207
- Add "did you mean" to unknown queries #51177
- Exclude unmapped fields during max clause limit checking for querying #49523 (issue: #49002)
- Security
-
- Add error message in JSON response #54389
- Snapshot/Restore
-
- Use Azure Bulk Deletes in Azure Repository #53919 (issue: #53865)
- Only link fd* files during source-only snapshot #53463 (issue: #50231)
- Add Blob Download Retries to GCS Repository #52479 (issues: #46589, #52319)
- Better Incrementality for Snapshots of Unchanged Shards #52182
- Add Region and Signer Algorithm Overrides to S3 Repos #52112 (issue: #51861)
- Allow Parallel Snapshot Restore And Delete #51608 (issue: #41463)
- Store
-
- HybridDirectory should mmap postings. #52641
- Transform
-
- Transition Transforms to using hidden indices for notifcations index #53773 (issue: #53762)
- Add processing stats to record the time spent for processing results #53770
- Create GET _cat/transforms API Issue #53643 (issue: #51412)
- Add support for script in group_by #53167 (issue: #43152)
- Implement node.transform to control where to run a transform #52712 (issues: #48734, #50033, #52200)
- Add support for filter aggregation #52483 (issue: #52151)
- Provide exponential_avg* stats for batch transforms #52041 (issue: #52037)
- Improve irrecoverable error detection - part 2 #52003 (issue: #51820)
- Mark transform API’s stable #51862
- Improve irrecoverable error detection #51820 (issue: #50135)
- Add support for percentile aggs #51808 (issue: #51663)
- Disallow fieldnames with a dot at start and/or end #51369
- Avoid mapping problems with index templates #51368 (issue: #51321)
- Handle permanent bulk indexing errors #51307 (issue: #50122)
- Improve force stop robustness in case of an error #51072
Bug fixes
edit- Aggregations
-
- Fix date_nanos in composite aggs #53315 (issue: #53168)
- Fix composite agg sort bug #53296 (issue: #52480)
- Decode max and min optimization more carefully #52336 (issue: #52220)
- Fix a DST error in date_histogram #52016 (issue: #50265)
- Use #name() instead of #simpleName() when generating doc values #51920 (issues: #50307, #51847)
- Fix a sneaky bug in rare_terms #51868 (issue: #51020)
- Support time_zone on composite’s date_histogram #51172 (issues: #45199, #45200)
- Allocation
- Analysis
- Audit
- Authentication
-
- Fix responses for the token APIs #54532 (issue: #53323)
- Fix potential bug in concurrent token refresh support #53668
- Respect runas realm for ApiKey security operations #52178 (issue: #51975)
- Preserve ApiKey credentials for async verification #51244
- Don’t fallback to anonymous for tokens/apikeys #51042 (issue: #50171)
- Fail gracefully on invalid token strings #51014
- Authorization
- CCR
- CRUD
- Cluster Coordination
- Distributed
- Engine
-
- Update translog policy before the next safe commit #54839 (issue: #52223)
- Fix doc_stats and segment_stats of ReadOnlyEngine #53345 (issues: #51303, #51331)
- Do not wrap soft-deletes reader for segment stats #51331 (issues: #51192, #51303)
- Account soft-deletes in FrozenEngine #51192 (issue: #50775)
- Fixed an index corruption bug that would occur when applying deletes or updates on an index after it has been shrunk. More details can be found on the corresponding issue.
- Features/CAT APIs
- Features/ILM+SLM
-
- Ensure error handler is called during SLM retention callback failure #55252 (issue: #55217)
- Ignore ILM indices in the TerminalPolicyStep #55184 (issue: #51631)
- Disallow negative TimeValues #53913 (issue: #54041)
- Fix null config in SnapshotLifecyclePolicy.toRequest #53328 (issues: #44465, #53171)
- Freeze step retry when not acknowledged #53287
- Make the set-single-node-allocation retryable #52077 (issue: #43401)
- Fix the init step to actually be retryable #52076
- Features/Indices APIs
-
- Read the index.number_of_replicas from template so that wait_for_active_shards is interpreted correctly #54231
- Features/Ingest
-
- Fix ingest pipeline _simulate api with empty docs never returns a response #52937 (issue: #52833)
- Handle errors when evaluating if conditions in processors #52543 (issue: #52339)
- Fix delete enrich policy bug #52179 (issue: #51228)
- Fix ignore_missing in CsvProcessor #51600
- Missing suffix for German Month "Juli" in Grok Pattern MONTH #51591 (issue: #51579)
- Don’t overwrite target field with SetSecurityUserProcessor #51454 (issue: #51428)
- Features/Java High Level REST Client
- Features/Java Low Level REST Client
- Features/Monitoring
- Features/Stats
- Features/Watcher
- Geo
-
- Handle properly indexing rectangles that crosses the dateline #53810
- Highlighting
- Infra/Core
-
- Make feature usage version aware #55246 (issues: #44589, #55248)
- Avoid StackOverflowError if write circular reference exception #54147 (issue: #53589)
- Fix Joda compatibility in stream protocol #53823 (issue: #53586)
- Avoid self-suppression on grouped action listener #53262 (issue: #53174)
- Ignore virtual ethernet devices that disappear #51581 (issue: #49914)
- Fix ingest timezone logic #51215 (issue: #51108)
- Infra/Logging
- Infra/Packaging
- Infra/Plugins
- Machine Learning
-
- Fix node serialization on GET df-nalytics stats without id #54808 (issue: #54807)
- Allow force stopping failed and stopping DF analytics #54650
- Take more care that normalize processes use unique named pipes #54636 (issue: #43830)
- Do not fail Evaluate API when the actual and predicted fields' types differ #54255 (issue: #54079)
- Get ML filters size should default to 100 #54207 (issues: #39976, #54206)
- Introduce a "starting" datafeed state for lazy jobs #53918 (issue: #53763)
- Only retry persistence failures when the failure is intermittent and stop retrying when analytics job is stopping #53725 (issue: #53687)
- Fix number inference models returned in x-pack info API #53540
- Make classification evaluation metrics work when there is field mapping type mismatch #53458 (issue: #53485)
- Perform evaluation in multiple steps when necessary #53295
- Specifying missing_field_value value and using it instead of empty_string #53108 (issue: #1034)
- Use event.timezone in ingest pipeline from find_file_structure #52720 (issue: #9458)
- Better error when persistent task assignment disabled #52014 (issue: #51956)
- Fix possible race condition starting datafeed #51646 (issues: #50886, #51302)
- Fix possible race condition when starting datafeed #51302 (issue: #51285)
- Address two edge cases for categorization.GrokPatternCreator#findBestGrokMatchFromExamples #51168
- Calculate results and snapshot retention using latest bucket timestamps #51061
- Use largest ordered subset of categorization tokens for category reverse search regex #970 (issue: #949)
- Account for the data frame’s memory when estimating the peak memory used by classification and regression model training #996
- Rename classification and regression parameter maximum_number_trees to max_trees #1047
- Mapping
- Network
-
- Add support for more named curves #55179 (issue: #55031)
- Allow proxy mode server name to be updated #54107
- Invoke response handler on failure to send #53631
- Do not log no-op reconnections at DEBUG #53469
- Fix RemoteConnectionManager size() method #52823 (issue: #52029)
- Remove seeds dependency for remote cluster settings #52796
- Add host address to BindTransportException message #51269 (issue: #48001)
- Percolator
- Reindex
- SQL
-
- Fix ODBC metadata for DATE & TIME data types #55316 (issue: #41086)
- Fix NPE for parameterized LIKE/RLIKE #53573 (issue: #53557)
- Add support for index aliases for SYS COLUMNS command #53525 (issue: #31609)
- Fix issue with LIKE/RLIKE as painless script #53495 (issue: #53486)
- Fix column size for IP data type #53056 (issue: #52762)
- Fix sql cli sourcing of x-pack-env #52613 (issue: #47803)
- Supplement input checks on received request parameters #52229
- Fix issue with timezone when paginating #52101 (issue: #51258)
- Fix ORDER BY on aggregates and GROUPed BY fields #51894 (issue: #50355)
- Fix milliseconds handling in intervals #51675 (issue: #41635)
- Fix ORDER BY YEAR() function #51562 (issue: #51224)
- Change the way unsupported data types fields are handled #50823
- Selecting a literal from grouped by query generates error #41964 (issues: #41413, #41951)
- Search
-
- Improve robustness of Query Result serializations #54692 (issue: #54665)
- Fix Term Vectors with artificial docs and keyword fields #53504 (issue: #53494)
- Fix concurrent requests race over scroll context limit #53449
- Fix pre-sorting of shards in the can_match phase #53397
- Fix potential NPE in FuzzyTermsEnum #53231 (issue: #52894)
- Fix inaccurate total hit count in _search/template api #53155 (issue: #52801)
- Harden search context id #53143
-
Correct boost in
script_score
query and error on negative scores #52478 (issue: #48465)
- Snapshot/Restore
-
- Exclude Snapshot Shard Status Update Requests from Circuit Breaker #55376 (issue: #54714)
- Fix Snapshot Completion Listener Lost on Master Failover #54286
- Fix Non-Verbose Snapshot List Missing Empty Snapshots #52433
- Fix Inconsistent Shard Failure Count in Failed Snapshots #51416 (issue: #47550)
- Fix Overly Aggressive Request DeDuplication #51270 (issue: #51253)
- Store
-
- Fix synchronization in ByteSizeCachingDirectory #52512
- Transform
-
- Fixing naming in HLRC and _cat to match API content #54300 (issue: #53946)
- Transform optmize date histogram #54068 (issue: #54254)
- Add version guards around Transform hidden index settings #54036 (issue: #53931)
- Fix NPE in derive stats if shouldStopAtNextCheckpoint is set #52940
- Fix mapping deduction for scaled_float #51990 (issue: #51780)
- Fix stats can return old state information if security is enabled #51732 (issue: #51728)