2.2.0 Release Notes

edit

Also see Breaking changes in 2.2.

Breaking changes

edit
Index APIs
  • Change Field stats API response to include both number and string based min and max values #14674 (issue: #14404)
Scripting
  • Filter classes loaded by scripts #15262
  • Lock down javascript and python script engines better #13924

Deprecations

edit
Java API
  • Deprecate the count api in favour of search with size 0 #14165 (issue: #13928)
  • Deprecates defaultRescoreWindowSize #13995
Plugin Discovery Multicast
Query DSL
Search
  • Deprecate _search/exists in favour of regular _search with size 0 and terminate_after 1 #13910 (issue: #13682)

New features

edit
Geo
  • Add CONTAINS relation to geo_shape query #14810 (issue: #14713)
  • Add support for Lucene 5.4 GeoPoint queries #14537
  • Add GeoPointV2 Field Mapping #14536
Network
  • Allow binding to multiple addresses. #13954 (issue: #13592)
Plugin Analysis Phonetic
  • Add support for daitch_mokotoff #14834
Plugin Repository S3
Plugins
  • Decentralize plugin security #14108
Search

Enhancements

edit
Aliases
  • Add support to _aliases endpoint to specify multiple indices and aliases in one action #15305 (issue: #15186)
Allocation
  • Simplify delayed shard allocation #14808
  • Add cluster-wide setting for total shard limit #14563 (issue: #14456)
  • Early terminate high disk watermark checks on single data node cluster #13882 (issue: #9391)
Analysis
  • Add detail response support for _analyze API #11660 (issue: #11076)
CAT API
  • Add sync_id to cat shards API #14712 (issue: #14705)
  • Add duration field to /_cat/snapshots #14385
  • Add cat API for repositories and snapshots #14247 (issue: #13919)
  • Adds disk used by indices to _cat/allocation #13783 (issue: #13529)
Cluster
  • Safe cluster state task notifications #15777
  • Reroute once per batch of shard failures #15510
  • Add callback for publication of new cluster state #15494 (issue: #15482)
  • Split cluster state update tasks into roles #15159
  • Use general cluster state batching mechanism for shard started #15023 (issues: #14725, #14899)
  • Use general cluster state batching mechanism for shard failures #15016 (issues: #14725, #14899)
  • Set an newly created IndexShard’s ShardRouting before exposing it to operations #14918 (issue: #10708)
  • Uniform exceptions for TransportMasterNodeAction #14737
Core
  • If we can’t get a MAC address for the node, use a dummy one #15266 (issue: #10099)
  • Simplify shard inactive logging #15259 (issue: #15252)
  • Simplify IndexingMemoryController#checkIdle #15252 (issue: #15251)
  • IndexingMemoryController should not track shard index states #15251 (issues: #13918, #15225)
  • Make PerThreadIDAndVersionLookup per-segment #14070
  • Verify Checksum once it has been fully written to fail as soon as possible #13896
Discovery
  • Don’t allow nodes with missing custom meta data to join cluster #15401 (issue: #13445)
Exceptions
  • Deduplicate cause if already contained in shard failures #14432
  • Give a better exception when running from freebsd jail without enforce_statfs=1 #14135 (issue: #12018)
  • Make root_cause of field conflicts more obvious #13976 (issue: #12839)
Fielddata
  • Update GeoPoint FieldData for GeoPointV2 #14345
Geo
  • Geo: Fix toString() in GeoDistanceRangeQuery and GeoPolygonQuery #15026
  • Enable GeoPointV2 with backward compatibility testing #14667 (issues: #10761, #11159, #9859)
  • Refactor Geo utilities to Lucene 5.4 #14339
Index APIs
  • Add option to disable closing indices #14169 (issue: #14168)
Index Templates
  • Disallow index template pattern to be the same as an alias name #15184 (issue: #14842)
Internal
  • Simplify the Text API. #15511
  • Simpler using compressed oops flag representation #15509 (issue: #15489)
  • Info on compressed ordinary object pointers #15489 (issues: #13187, #455)
  • Explicitly log cluster state update failures #15428 (issues: #14899, #15016, #15023)
  • Use transport service to handle RetryOnReplicaException to execute replica action on the current node #15363
  • Make IndexShard operation be more explicit about whether they are expected to run on a primary or replica #15282
  • Avoid trace logging allocations in TransportBroadcastByNodeAction #15221
  • Only trace log shard not available exceptions #14950 (issue: #14927)
  • Transport options should be immutable #14760
  • Fix dangling comma in ClusterBlock#toString #14483
  • Improve some logging around master election and cluster state #14481
  • Add System#exit(), Runtime#exit() and Runtime#halt() to forbidden APIs #14473 (issue: #12596)
  • Simplify XContent detection. #14472
  • Add threadgroup isolation. #14353
  • Cleanup plugin security #14311
  • Add workaround for JDK-8014008 #14274
  • Refactor retry logic for TransportMasterNodeAction #14222
  • Remove MetaDataSerivce and it’s semaphores #14159 (issue: #1296)
  • Cleanup IndexMetaData #14119
  • TransportNodesAction shouldn’t hold on to cluster state #13948
  • Add SpecialPermission to guard exceptions to security policy. #13854
  • Clean up scripting permissions. #13844
  • Factor groovy out of core into lang-groovy #13834 (issue: #13725)
  • Factor expressions scripts out to lang-expression plugin #13726 (issue: #13725)
Java API
  • TransportClient: Add exception when using plugin.types, to help migration to addPlugin #15943 (issue: #15693)
  • Align handling of interrupts in BulkProcessor #15527 (issue: #14833)
  • BulkProcessor backs off exponentially by default #15513 (issue: #14829)
  • Add java API for synced flush #15396 (issue: #12812)
  • Allow to get and set ttl as a time value/string #15239 (issue: #15047)
  • Reject refresh usage in bulk items when using and fix NPE when no source #15082 (issue: #7361)
  • BulkProcessor retries after request handling has been rejected due to a full thread pool #14829 (issue: #14620)
Logging
  • Add circuit breaker name to logging package #14661
  • Log cluster health status changes #14557 (issue: #11657)
  • Map log-level trace to JDK-Level FINEST #14234
Mapping
  • Make mapping serialization more robust. #15480
  • Make mappings immutable. #15313 (issue: #9365)
  • Make MappedFieldType.checkTypeName part of MappedFieldType.checkCompatibility. #15245
  • throw exception if a copy_to is within a multi field #15213 (issues: #10802, #14946)
  • Register field mappers at the node level. #14896 (issue: #14828)
Network
  • only allow code to bind to the user’s configured port numbers/ranges #14549
  • Port of publishAddress should match port of corresponding boundAddress #14535 (issues: #14503, #14513, #14514)
Packaging
  • Default standard output to the journal in systemd #16159 (issues: #15315, #16134)
  • Use egrep instead of grep -E for Solaris #15755 (issue: #15628)
  • punch thru symlinks when loading plugins/modules #15311
  • set ActiveProcessLimit=1 on windows #15055
  • set RLIMIT_NPROC = 0 on bsd/os X systems. #15039
  • Drop ability to execute on Solaris #14200
  • Packaging: change permissions/ownership of config dir #14017 (issue: #11016)
  • Release: Fix package repo path to only consist of major version #13971 (issue: #12493)
  • Nuke ES_CLASSPATH appending, JarHell fail on empty classpath elements #13880 (issues: #13812, #13864)
  • Get lang-javascript, lang-python, securemock ready for script refactoring #13695
  • Remove some bogus permissions only needed for tests. #13620
  • Remove java.lang.reflect.ReflectPermission "suppressAccessChecks" #13603
Plugin Cloud AWS
Plugin Cloud Azure
Plugin Cloud GCE
Plugin Discovery EC2
  • Add ap-northeast-2 (seoul) endpoints for EC2 discovery and S3 snapshots #16167 (issue: #16166)
  • Adding US-Gov-West #14358
  • Improved building of disco nodes #14155
Plugin Mapper Attachment
  • Backport mapper-attachments plugin to 2.x #14902
Plugin Repository S3
Plugins
  • Expose http.type setting, and collapse al(most all) modules relating to transport/http #15434 (issue: #14148)
  • Ban RuntimePermission("getClassLoader") #15253
  • Add nicer error message when a plugin descriptor is missing #15200 (issue: #15197)
  • Don’t be lenient in PluginService#processModule(Module) #14306
  • Check "plugin already installed" before jar hell check. #14207 (issue: #14205)
  • Plugin script to set proper plugin bin dir attributes #14088 (issue: #11016)
  • Plugin script to set proper plugin config dir attributes #14048 (issue: #11016)
Query DSL
  • Allow CIDR notation in query string query #14773 (issue: #7464)
REST
Recovery
  • Handle cancel exceptions on recovery target if the cancel comes from the source #15309
  • Decouple routing and primary operation logic in TransportReplicationAction #14852
Scripting
  • Add property permissions so groovy scripts can serialize json #14500 (issue: #14488)
  • Remove ScriptEngineService.unwrap. #13958
  • Remove ScriptEngineService.execute. #13956
Search
  • Support * wildcard to retrieve stored fields in the fields option #15119 (issues: #10783, #14489)
  • Caching Weight wrappers should propagate the BulkScorer. #14317
  • fix numerical issue in function score query #14085
Snapshot/Restore
  • Support wildcards for getting repositories and snapshots #15151 (issue: #4758)
  • Add ignore_unavailable parameter to skip unavailable snapshot #14471 (issue: #13887)
Stats
  • Add system CPU percent to OS stats #15302
  • Pull Fields instance once from LeafReader in completion stats #15090 (issue: #6593)
  • Add os.allocated_processors stats #14409 (issue: #13917)
Top Hits
  • Put method addField on TopHitsBuilder #14597 (issue: #12962)
Translog
  • Check for tragic event on all kinds of exceptions not only ACE and IOException #15535
Tribe Node
  • Tribe nodes should apply cluster state updates in batches #14993 (issues: #14725, #14899)

Bug fixes

edit
Aggregations
  • Filter(s) aggregation should create weights only once. #15998
  • Make missing on terms aggs work with all execution modes. #15746 (issue: #14882)
  • Run pipeline aggregations for empty buckets added in the Range Aggregation #15519 (issue: #15471)
  • [Children agg] fix bug that prevented all child docs from being evaluated #15457
  • Correct typo in class name of StatsAggregator #15321 (issue: #14730)
  • Fix significant terms reduce for long terms #14948 (issue: #13522)
  • Pass extended bounds into HistogramAggregator when creating an unmapped aggregator #14742 (issue: #14735)
  • Added correct generic type parameter on ScriptedMetricBuilder #14018 (issue: #13986)
Aliases
  • Check lenient_expand_open after aliases have been resolved #15882 (issue: #13278)
Allocation
  • Prevent peer recovery from node with older version #15775
  • Fix calculation of next delay for delayed shard allocation #14765
  • Take ignored unallocated shards into account when making allocation decision #14678 (issue: #14670)
  • Only allow rebalance operations to run if all shard store data is available #14591 (issue: #14387)
  • Delayed allocation can miss a reroute #14494 (issues: #14010, #14011, #14445)
  • Check rebalancing constraints when shards are moved from a node they can no longer remain on #14259 (issue: #14057)
Bulk
  • Do not release unacquired semaphore #14909 (issue: #14908)
CAT API
  • Properly set indices and indicesOptions on subrequest made by /_cat/indices #14360
CRUD
  • Throw exception if content type could not be determined in Update API #15904 (issue: #15822)
Cluster
  • Only fail the relocation target when a replication request on it fails #15791 (issue: #15790)
  • Handle shards assigned to nodes that are not in the cluster state #14586 (issue: #14584)
  • Bulk cluster state updates on index deletion #11258 (issue: #7295)
Core
  • BitSetFilterCache duplicates its content. #15836 (issue: #15820)
  • Limit the max size of bulk and index thread pools to bounded number of processors #15585 (issue: #15582)
  • AllTermQuery’s scorer should skip segments that never saw the requested term #15506
  • Include root-cause exception when we fail to change shard’s index buffer #14867
  • Restore thread interrupt flag after an InterruptedException #14799 (issue: #14798)
  • Use fresh index settings instead of relying on @IndexSettings #14578 (issue: #14319)
  • Record all bytes of the checksum in VerifyingIndexOutput #13923 (issues: #13848, #13896)
  • When shard becomes active again, immediately increase its indexing buffer #13918 (issue: #13802)
  • Close TokenStream in finally clause #13870 (issue: #11947)
Engine
  • Never wrap searcher for internal engine operations #14071
Exceptions
  • Fix ensureNodesAreAvailable’s error message #14007 (issue: #13957)
Fielddata
  • Don’t cache top level field data for fields that don’t exist #14693
Geo
  • Geo: Allow numeric parameters enclosed in quotes for geohash_grid aggregation #14440 (issue: #13132)
Highlighting
  • Fix spans extraction to not also include individual terms. #15516 (issues: #13239, #15291)
Index APIs
  • Field stats: Index constraints should remove indices in the response if the field to evaluate is empty #14868
  • Field stats: Fix NPE for index constraint on empty index #14841
  • Field stats: Added format option for index constraints #14823 (issue: #14804)
  • Restore previous optimize transport action name for bw comp #14221 (issue: #13778)
  • Forbid index name . and .. #13862 (issue: #13858)
Internal
  • Log uncaught exceptions from scheduled once tasks #15824 (issue: #15814)
  • Make sure the remaining delay of unassigned shard is updated with every reroute #14890 (issue: #14808)
  • Throw a meaningful error when loading metadata and an alias and index have the same name #14842 (issue: #14706)
  • fixup issues with 32-bit jvm #14609
  • Failure to update the cluster state with the recovered state should make sure it will be recovered later #14485
  • Properly bind ClassSet extensions as singletons #14232 (issue: #14194)
  • Streamline top level reader close listeners and forbid general usage #14084
  • Gateway: a race condition can prevent the initial cluster state from being recovered #13997
  • Verify actually written checksum in VerifyingIndexOutput #13848
  • Remove all setAccessible in tests and forbid #13539
  • Remove easy uses of setAccessible in tests. #13537
  • Ban setAccessible from core code, restore monitoring stats under java 9 #13531 (issue: #13527)
Logging
  • Settings in log config file should not overwrite custom parameters #13934
  • Plugin cli tool should not create empty log files #13933
Mapping
  • Fix MapperService#searchFilter(…​) #15923 (issue: #15757)
  • Fix initial sizing of BytesStreamOutput. #15864 (issue: #15789)
  • MetaDataMappingService should call MapperService.merge with the original mapping update. #15508
  • Fix copy_to when the target is a dynamic object field. #15385 (issue: #11237)
  • Only text fields should accept analyzer and term vector settings. #15308
  • Mapper parsers should not check for a tokenized property. #15289
  • Validate that fields are defined only once. #15243 (issue: #15057)
  • Check mapping compatibility up-front. #15175 (issue: #15049)
  • Don’t treat default as a regular type. #15156 (issue: #15049)
  • Don’t ignore mapping merge failures. #15144 (issue: #15049)
  • Treat mappings at an index-level feature. #15142
  • Multi field names may not contain dots #15118 (issue: #14957)
  • Mapping: Allows upgrade of indexes with only search_analyzer specified #14677 (issue: #14383)
Packaging
  • Fix Windows service installation failure #15549 (issue: #15349)
  • fix command line options for windows bat file #15320 (issues: #15284, #16086)
  • Handle system policy correctly #14704 (issue: #14690)
  • Startup script exit status should catch daemonized startup failures #14170 (issue: #14163)
Plugin Cloud AWS
Plugin Delete By Query
  • Fix Delete-by-Query with Shield #14658 (issue: #14527)
  • Delete by query to not wrap the inner query into an additional query element #14302 (issue: #13326)
Plugin Mapper Attachment
  • Fix toXContent() for mapper attachments field #15110
Plugin Repository S3
  • Hack around aws security hole of accessing sun.security.ssl, s3 repository works on java 9 again #13538 (issue: #432)
Plugins
  • Fix plugin list command error message #14288 (issue: #14287)
  • Parse Java system properties in plugin.bat #13989 (issue: #13616)
  • PluginManager: Dont leave leftover files on unsuccessful installs #12851 (issue: #12749)
  • Fix HTML response during redirection #11374 (issue: #11370)
Query DSL
  • Fix FunctionScore equals/hashCode to include minScore and friends #15676
  • Min should match greater than the number of optional clauses should return no result #15571 (issue: #15521)
  • Return a better exception message when regexp query is used on a numeric field #14910 (issue: #14782)
REST
  • Throw exception when trying to write map with null keys #15479 (issue: #14346)
  • XContentFactory.xContentType: allow for possible UTF-8 BOM for JSON XContentType #14611 (issue: #14442)
  • Restore support for escaped / as part of document id #14216 (issues: #13665, #13691, #14177)
  • Add missing REST spec for detect_noop #14004
  • Expose nodes operation timeout in REST API #13981
  • Ensure XContent is consistent across platforms #13816
Recovery
  • sync translog to disk after recovery from primary #15832
  • CancellableThreads should also treat ThreadInterruptedException as InterruptedException #15318
Scripting
  • Security permissions for Groovy closures #16196 (issues: #16194)
Search
  • Fix NPE when a segment with an empty cache gets closed. #15202 (issue: #15043)
  • Fix the quotes in the explain message for a script score function without parameters #11398
Settings
  • TransportClient should use updated setting for initialization of modules and service #16095
Shadow Replicas
  • Don’t pull translog from shadow engine #14000 (issue: #12730)
Snapshot/Restore
  • Snapshot restore and index creates should keep index settings and cluster blocks in sync #13931 (issue: #13213)
  • Snapshot restore operations throttle more than specified #13828 (issue: #6018)
  • Fix blob size in writeBlob() method #13574 (issue: #13434)
Stats
  • Fix calculation of age of pending tasks #15995 (issue: #15988)
  • serialize os name, arch and version too #15454 (issue: #15422)
  • Add extra validation into cluster/stats #14699 (issue: #7390)
  • Omit current* stats for OldShardStats #13801 (issue: #13386)
Translog
  • Make sure IndexShard is active during recovery so it gets its fair share of the indexing buffer #16209 (issue: #16206)
  • Avoid circular reference in exception #15952 (issue: #15941)
  • Initialize translog before scheduling the sync to disk #15881
  • Translog base flushes can be disabled after replication relocation or slow recovery #15830 (issues: #10624, #15814)
  • Catch tragic even inside the checkpoint method rather than on the caller side #15825
  • Never delete translog-N.tlog file when creation fails #15788
  • Close recovered translog readers if createWriter fails #15762 (issue: #15754)
  • Fail and close translog hard if writing to disk fails #15420 (issue: #15333)
  • Prevent writing to closed channel if translog is already closed #15012 (issue: #14866)
  • Don’t delete temp recovered checkpoint file if it was renamed #14872 (issue: #14695)
  • Translog recovery can repeatedly fail if we run out of disk #14695
Tribe Node
  • Tribe node: pass path.conf to inner tribe clients #16258 (issue: #16253)
  • Fix tribe node to load config file for internal client nodes #15300 (issues: #13383, #14573)

Regressions

edit
Analysis
  • Add PathHierarchy type back to path_hierarchy tokenizer for backward compatibility with 1.x #15785 (issue: #15756)
Internal
Plugin Cloud Azure
Query DSL
  • RangeQueryParser should accept _name in inner field #15394 (issue: #15306)

Upgrades

edit
Core
  • Upgrade to lucene-5.4.1. #16160
  • Upgrade to lucene-5.4.0. #15477
  • Upgrade Lucene to 5.4.0-snapshot-1715952 #14951
  • Upgrade Lucene to 5.4.0-snapshot-1714615 #14784
  • Upgrade to lucene-5.4.0-snapshot-1712973. #14619
  • update to lucene-5.4.x-snapshot-1711508 #14398
  • Upgrade to lucene-5.4-snapshot-1710880. #14320
  • Upgrade to lucene-5.4-snapshot-1708254. #14074
  • upgrade lucene to r1702265 #13439
  • Upgrade master to lucene 5.4-snapshot r1701068 #13324
Geo
  • Update to spatial4j 0.5 for correct Multi-Geometry #14269 (issue: #9904)
Network
  • Upgrade Netty to 3.10.5.final #14105
Plugin Discovery Azure
  • Update Azure Service Management API to 0.9.0 #15232 (issue: #15209)
Plugin Discovery EC2
  • Upgrade to aws 1.10.33 #14672
Plugin Lang JS
  • upgrade rhino for plugins/lang-javascript #14466