- Elastic Cloud Enterprise - Elastic Cloud on your Infrastructure: other versions:
- Introducing Elastic Cloud Enterprise
- Preparing your installation
- Installing Elastic Cloud Enterprise
- Identify the deployment scenario
- Install ECE with Ansible
- Install ECE on a public cloud
- Install ECE on your own premises
- Log into the Cloud UI
- Install ECE on additional hosts
- Post-installation steps
- Configuring your installation
- System clusters configuration
- Configure deployment templates
- Tag your allocators
- Edit instance configurations
- Create instance configurations
- Create deployment templates
- Configure system deployment templates
- Configure index management for templates
- Updating custom templates to support
node_roles
and autoscaling - Updating custom templates to support Integrations Server
- Default instance configurations
- Include additional Kibana plugins
- Manage snapshot repositories
- Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 6.4 - 6.8
- Snapshotting to Google Cloud Storage (GCS) for Elasticsearch 7.0 and later
- Snapshotting to Azure Storage for Elasticsearch 5.x and 6.4 - 6.8
- Snapshotting to Azure Storage for Elasticsearch 7.0 and later
- Snapshotting to Minio on-premise storage
- Manage licenses
- Change the ECE API URL
- Change endpoint URLs
- Enable custom endpoint aliases
- Configure allocator affinity
- Securing your installation
- Monitoring your installation
- Administering your installation
- Getting started with deployments
- Create your deployment
- Access Kibana
- Manage data from the command line
- Configure the Java transport client (deprecated)
- Next steps
- Migrate your Elasticsearch data
- Indexing data into Elasticsearch
- Ingest data with Node.js on Elastic Cloud Enterprise
- Ingest data with Python on Elastic Cloud Enterprise
- Ingest data from Beats to Elastic Cloud Enterprise with Logstash as a proxy
- Ingest data from a relational database into Elastic Cloud Enterprise
- Ingest logs from a Python application using Filebeat
- Ingest logs from a Node.js web application using Filebeat
- Administering deployments
- Change your deployment configuration
- Stop routing requests or pause nodes
- Terminate a deployment
- Restart a deployment
- Restore a deployment
- Delete a deployment
- Migrate to index lifecycle management
- Disable a data tier
- Access the Elasticsearch API console
- Work with snapshots
- Upgrade versions
- Upgrade App Search to Enterprise Search
- Editing your user settings
- Deployment autoscaling
- Configure Beats and Logstash with Cloud ID
- Keep your clusters healthy
- Secure your clusters
- Reset the password for the
elastic
user - Secure your settings
- Secure your 7.x clusters with LDAP
- Secure your 7.x clusters with Active Directory
- Secure your 5.x and 6.x clusters with LDAP
- Secure your 5.x and 6.x clusters with Active Directory
- Secure your clusters with SAML
- Secure your clusters with OpenID Connect
- Secure your clusters with Kerberos
- Reset the password for the
- Trust management
- Deployment heap dumps
- Deployment thread dumps
- Traffic Filtering
- Connect to your cluster
- Manage your Kibana instance
- Manage your APM & Fleet Server
- Manage your APM Server (versions before 7.13)
- Manage your Integrations Server
- Enable logging and monitoring
- Enable cross-cluster search and cross-cluster replication
- Enable App Search
- Enable Enterprise Search
- Enable Graph (versions before 5.0)
- Troubleshooting
- RESTful API
- Authentication
- API calls
- How to access the API
- API examples
- Setting up your environment
- A first API call: What deployments are there?
- Create a first Deployment: Elasticsearch and Kibana
- Applying a new plan: Resize and add high availability
- Updating a deployment: Checking on progress
- Applying a new deployment configuration: Upgrade
- Enable more stack features: Add Enterprise Search to a deployment
- Dipping a toe into platform automation: Generate a roles token
- Customize your deployment
- Remove unwanted deployment templates and instance configurations
- Secure your settings
- API reference
- API - Base Url
- APM routes
- Accounts
- App Search - Read-only
- Authentication
- User authentication information
- Disable elevated permissions
- Enable elevated permissions
- Login to ECE
- Logout from ECE
- Refresh authentication token
- Delete API keys
- Get all API keys
- Create API key
- Delete API keys of multiple users
- Get all API keys for all users
- Delete API key
- Get API key
- Available authentication methods
- SAML callback
- Initiate SAML protocol
- Delete API keys for a user
- Get API key metadata for all keys created by the user
- Delete an API key for a user
- Get a user API key
- Billing Costs Analysis
- Comments
- Deployment - CRUD
- Deployment - Commands
- Search Deployments
- Migrate Elasticsearch and associated Kibana resources to enable CCR
- Migrate Elasticsearch resource to use ILM
- Migrate Elasticsearch resource to use SLM
- Reset elastic user password
- Restart Deployment Elasticsearch Resource
- Shutdown Deployment Elasticsearch Resource
- Move Elasticsearch resource instances
- Get details about heap dumps for a Deployment
- Start all instances
- Stop all instances
- Start maintenance mode (all instances)
- Stop maintenance mode (all instances)
- Set size overrides
- Start instances
- Stop instances
- Start maintenance mode
- Stop maintenance mode
- Set size overrides
- Capture a new on-demand heap dump for the given instance
- Download the given instance’s heap dump
- Capture a new thread dump for the given instance
- Restart Deployment Stateless Resource
- Shutdown Deployment Stateless Resource
- Upgrade Kibana, APM, Integrations Server, AppSearch, Enterprise Search inside Deployment
- Deployment - Diagnostics
- Deployment - Info
- Deployment - Remote Clusters
- Deployment - Upgrade Assistant
- Deployment - resync
- Deployment Resources - CRUD
- Deployment Resources - Proxy
- Deployments - Extensions
- Deployments - Notes
- Deployments - Templates
- Deployments - Traffic Filter - Associations
- Deployments - Traffic Filter - CRUD
- Integrations Server routes
- Networking - Deployment Domain Names
- Platform
- Platform - Adminconsoles
- Platform - Allocators
- Get allocators
- Resynchronize allocators
- Search allocators
- Delete allocator
- Get allocator
- Resynchronize allocator
- Move clusters
- Move clusters by type
- Delete allocator logging settings
- Get allocator logging settings
- Update allocator logging settings
- Set allocator logging settings
- Start maintenance mode
- Stop maintenance mode
- Get allocator metadata
- Set allocator metadata
- Delete allocator metadata item
- Set allocator metadata item
- Get allocator settings
- Update allocator settings
- Set allocator settings
- Platform - Configuration - Instances - CRUD
- Platform - Configuration - Security
- Platform - Configuration - Security - Extra Certificates - CRUD
- Platform - Configuration - Security Deployment
- Platform - Configuration - Security Realms
- List security realm configurations
- Reorder security realms
- Create Active Directory configuration
- Delete Active Directory configuration
- Get Active Directory configuration
- Update Active Directory configuration
- Create LDAP configuration
- Delete LDAP configuration
- Get LDAP configuration
- Update LDAP configuration
- Create SAML configuration
- Delete SAML configuration
- Get SAML configuration
- Update SAML configuration
- Platform - Configuration - TLS
- Platform - Configuration - Trust relationships
- Platform - Constructors
- Platform - License
- Platform - Repository - CRUD
- Platform - Runners
- Platform - configuration - Store
- Platform - proxies
- Roles
- Stack - Instance Types - CRUD
- Stack - Versions - CRUD
- Telemetry
- Templates - Deployments - Global
- Users
- Definitions
AbsoluteRefId
AccountResponse
AccountTrustRelationship
AccountTrustSettings
AccountUpdateRequest
ActiveDirectoryGroupSearch
ActiveDirectorySecurityRealmLoadBalance
ActiveDirectorySecurityRealmRoleMappingRule
ActiveDirectorySecurityRealmRoleMappingRules
ActiveDirectorySettings
ActiveDirectoryUserSearch
AdminconsoleInfo
AdminconsolesOverview
AllocatedInstancePlansInfo
AllocatedInstanceStatus
AllocatorBuildInfo
AllocatorCapacity
AllocatorCapacityMemory
AllocatorHealthStatus
AllocatorInfo
AllocatorMoveRequest
AllocatorOverview
AllocatorSettings
AllocatorZoneInfo
AllocatorsSummary
AllocatorsZoneSummary
ApiBaseUrlData
ApiKeyResponse
ApiKeysResponse
Apm
ApmConfiguration
ApmCrudResponse
ApmInfo
ApmPayload
ApmPlan
ApmPlanControlConfiguration
ApmPlanInfo
ApmPlansInfo
ApmResourceInfo
ApmSettings
ApmSubInfo
ApmSystemSettings
ApmTopologyElement
AppSearch
AppSearchConfiguration
AppSearchInfo
AppSearchNodeTypes
AppSearchPayload
AppSearchPlan
AppSearchPlanControlConfiguration
AppSearchPlanInfo
AppSearchPlansInfo
AppSearchResourceInfo
AppSearchSettings
AppSearchSubInfo
AppSearchSystemSettings
AppSearchTopologyElement
AuthenticationInfo
AutodetectStrategyConfig
AvailableAuthenticationMethods
Balance
BasicFailedReply
BasicFailedReplyElement
Blessing
Blessings
BlessingsWithMeta
BoolQuery
CapacityConstraintsResource
ChainStatus
ChangeSourceInfo
ClusterCredentials
ClusterCurationSettings
ClusterCurationSpec
ClusterInstanceConfigurationInfo
ClusterInstanceDiskInfo
ClusterInstanceInfo
ClusterInstanceMemoryInfo
ClusterLicenseInfo
ClusterMetadataCpuResourcesSettings
ClusterMetadataInfo
ClusterMetadataPortInfo
ClusterMetadataResourcesSettings
ClusterMetadataSettings
ClusterPlanStepInfo
ClusterPlanStepLogMessageInfo
ClusterPlanWarning
ClusterSnapshotRepositoryDefault
ClusterSnapshotRepositoryInfo
ClusterSnapshotRepositoryReference
ClusterSnapshotRepositoryStatic
ClusterSnapshotRetention
ClusterSnapshotSettings
ClusterSystemAlert
ClusterTopologyInfo
Comment
CommentCreateRequest
CommentUpdateRequest
CommentWithMeta
CommentsWithMetas
CompatibleNodeTypesResource
CompatibleVersionResource
ConfigStoreOption
ConfigStoreOptionData
ConfigStoreOptionList
ConstructorHealthStatus
ConstructorInfo
ConstructorOverview
ContainerConfigHostConfig
ContainerSetSummary
ContainerSetsSummary
ContainersEntry
ContainersEntryOptions
ContainersEntryOptionsACL
ContainersEntryOptionsAuth
ContainersEntryOptionsContainerConfig
ContainersEntryOptionsOverrides
CoordinatorSummary
CoordinatorsSummary
Costs
CostsOverview
CreateApiKeyRequest
CreateExtensionRequest
Creates
DeleteApiKeysRequest
DeleteUsersApiKeysRequest
DeploymentCosts
DeploymentCreateMetadata
DeploymentCreateRequest
DeploymentCreateResources
DeploymentCreateResponse
DeploymentCreateSettings
DeploymentDeleteResponse
DeploymentDiagnostics
DeploymentDomainName
DeploymentGetResponse
DeploymentHeapDumps
DeploymentLogging
DeploymentLoggingSettings
DeploymentMetadata
DeploymentMetrics
DeploymentMetricsSettings
DeploymentObservability
DeploymentObservabilitySettings
DeploymentResource
DeploymentResourceCommandResponse
DeploymentResourceCrudResponse
DeploymentResourceUpgradeResponse
DeploymentResources
DeploymentRestoreResponse
DeploymentSearchResponse
DeploymentSettings
DeploymentShutdownResponse
DeploymentTemplateInfoV2
DeploymentTemplateReference
DeploymentTemplateRequestBody
DeploymentUpdateMetadata
DeploymentUpdateRequest
DeploymentUpdateResources
DeploymentUpdateResponse
DeploymentUpdateSettings
DeploymentUpgradeAssistantStatusResponse
DeploymentsCosts
DeploymentsListResponse
DeploymentsListingData
DeploymentsSearchResponse
DiscreteSizes
DtsDimensionCosts
DtsQuantity
DtsRate
Elasticsearch
ElasticsearchClusterBlockingIssueElement
ElasticsearchClusterBlockingIssues
ElasticsearchClusterInfo
ElasticsearchClusterPlan
ElasticsearchClusterPlanInfo
ElasticsearchClusterPlansInfo
ElasticsearchClusterRole
ElasticsearchClusterSecurityInfo
ElasticsearchClusterSettings
ElasticsearchClusterTopologyElement
ElasticsearchClusterTrustSettings
ElasticsearchClusterUser
ElasticsearchConfiguration
ElasticsearchCuration
ElasticsearchDependant
ElasticsearchElasticUserPasswordResetResponse
ElasticsearchInfo
ElasticsearchMasterElement
ElasticsearchMasterInfo
ElasticsearchMonitoringInfo
ElasticsearchNodeType
ElasticsearchPayload
ElasticsearchPlanControlConfiguration
ElasticsearchReplicaElement
ElasticsearchResourceInfo
ElasticsearchScriptTypeSettings
ElasticsearchScriptingUserSettings
ElasticsearchShardElement
ElasticsearchShardsInfo
ElasticsearchSystemSettings
ElasticsearchUserBundle
ElasticsearchUserPlugin
ElevatePermissionsRequest
ElevatedPermissions
EmptyResponse
EnableIlmRequest
EnrollmentTokenRequest
EnterpriseSearch
EnterpriseSearchConfiguration
EnterpriseSearchInfo
EnterpriseSearchNodeTypes
EnterpriseSearchPayload
EnterpriseSearchPlan
EnterpriseSearchPlanControlConfiguration
EnterpriseSearchPlanInfo
EnterpriseSearchPlansInfo
EnterpriseSearchResourceInfo
EnterpriseSearchSettings
EnterpriseSearchSubInfo
EnterpriseSearchSystemSettings
EnterpriseSearchTopologyElement
ExistsQuery
Extension
ExtensionFileMetadata
Extensions
ExternalHyperlink
ExternalTrustRelationship
FilterAssociation
GenericResponse
GlobalDeploymentTemplateInfo
GlobalDeploymentTemplateRegion
GrowShrinkStrategyConfig
HeapDump
Hyperlink
IdResponse
IndexPattern
IndexSynchronizationResults
InstanceConfiguration
InstanceConfigurationInfo
InstanceMoveRequest
InstanceOverrides
InstanceTypeResource
IntegrationsServerConfiguration
IntegrationsServerCrudResponse
IntegrationsServerInfo
IntegrationsServerPayload
IntegrationsServerPlan
IntegrationsServerPlanControlConfiguration
IntegrationsServerPlanInfo
IntegrationsServerPlansInfo
IntegrationsServerResourceInfo
IntegrationsServerSettings
IntegrationsServerSystemSettings
IntegrationsServerTopologyElement
ItemCosts
ItemsCosts
KeystoreContents
KeystoreSecret
Kibana
KibanaClusterInfo
KibanaClusterPlan
KibanaClusterPlanInfo
KibanaClusterPlansInfo
KibanaClusterSettings
KibanaClusterTopologyElement
KibanaConfiguration
KibanaDeeplink
KibanaPayload
KibanaPlanControlConfiguration
KibanaResourceInfo
KibanaSubClusterInfo
KibanaSystemSettings
LdapGroupSearch
LdapSecurityRealmLoadBalance
LdapSecurityRealmRoleMappingRule
LdapSecurityRealmRoleMappingRules
LdapSettings
LdapUserSearch
LicenseInfo
LicenseObject
ListEnrollmentTokenElement
ListEnrollmentTokenReply
LoggingSettings
LoginRequest
LoginState
ManagedMonitoringSettings
MatchAllQuery
MatchNoneQuery
MatchQuery
Metadata
MetadataItem
MetadataItemValue
MetadataItems
ModelVersionIndexSynchronizationResults
MoveApmClusterConfiguration
MoveApmClusterDetails
MoveAppSearchConfiguration
MoveAppSearchDetails
MoveClustersCommandResponse
MoveClustersDetails
MoveClustersRequest
MoveElasticsearchClusterConfiguration
MoveElasticsearchClusterDetails
MoveEnterpriseSearchConfiguration
MoveEnterpriseSearchDetails
MoveKibanaClusterConfiguration
MoveKibanaClusterDetails
NestedQuery
NodeTypeResource
Note
Notes
ObservabilityIssue
Orphaned
OrphanedElasticsearch
PendingState
PendingStates
PendingStatesWithMeta
Period
PlanStrategy
PlatformInfo
PlatformResourcesSummary
PlatformServiceImageInfo
PlatformServiceInfo
PortBinding
PrefixQuery
ProxiesAllocationsInfo
ProxiesFilter
ProxiesFilteredGroup
ProxiesFilteredGroupHealth
ProxiesHealth
ProxiesHttpSettings
ProxiesSSOSettings
ProxiesSettings
ProxiesSummary
ProxyAllocationCounts
ProxyAllocationInfo
ProxyInfo
ProxyOverview
ProxySummary
PublicCertChain
PublicCertChainCollection
QueryContainer
QueryStringQuery
RangeQuery
ReadOnlyRequest
ReadOnlyResponse
RegionInfo
RemoteResourceInfo
RemoteResourceRef
RemoteResources
ReplyWarning
RepositoryConfig
RepositoryConfigs
RequestEnrollmentTokenReply
ResourceHeapDumps
RestartPolicy
RestoreSnapshotApiConfiguration
RestoreSnapshotConfiguration
RestoreSnapshotRepoConfiguration
Role
RoleAggregate
RoleAggregateCreateData
RoleAggregates
RoleWithMeta
RollingGrowShrinkStrategyConfig
RollingStrategyConfig
RulesetAssociations
RunnerBuildInfo
RunnerContainerInfo
RunnerInfo
RunnerOverview
RunnerRoleInfo
RunnerRolesInfo
RunnersSummary
SamlAttributeSettings
SamlIdpSettings
SamlSecurityRealmRoleMappingRule
SamlSecurityRealmRoleMappingRules
SamlSettings
SamlSpSettings
SearchRequest
SecurityDeployment
SecurityDeploymentCreateRequest
SecurityDeploymentTopology
SecurityDeploymentUpdateRequest
SecurityRealmInfo
SecurityRealmInfoList
SecurityRealmsReorderRequest
ServiceUrl
SimplifiedLineItem
SnapshotRepositoryConfiguration
SnapshotStatusInfo
SsoAuthenticationMethodInfo
StackVersionApmConfig
StackVersionAppSearchConfig
StackVersionArchiveProcessingError
StackVersionArchiveProcessingResult
StackVersionConfig
StackVersionConfigPost
StackVersionConfigs
StackVersionElasticsearchConfig
StackVersionEnterpriseSearchConfig
StackVersionInstanceCapacityConstraint
StackVersionIntegrationsServerConfig
StackVersionKibanaConfig
StackVersionMetadata
StackVersionNodeType
StackVersionTemplateFileHash
StackVersionTemplateInfo
TargetElasticsearchCluster
TelemetryConfig
TelemetryConfigRequest
TermQuery
TlsPublicCertChain
TokenResponse
TopologyElementControl
TopologySize
TrafficFilterEgressRule
TrafficFilterRule
TrafficFilterRulesetInfo
TrafficFilterRulesetRequest
TrafficFilterRulesetResponse
TrafficFilterRulesets
TrafficFilterSettings
TransientApmPlanConfiguration
TransientAppSearchPlanConfiguration
TransientElasticsearchPlanConfiguration
TransientEnterpriseSearchPlanConfiguration
TransientIntegrationsServerPlanConfiguration
TransientKibanaPlanConfiguration
TrustRelationshipCreateRequest
TrustRelationshipCreateResponse
TrustRelationshipGetResponse
TrustRelationshipUpdateRequest
TrustRelationshipUpdateResponse
TrustRelationshipsListResponse
UnreachableRegionInfo
UpdateExtensionRequest
UpdatedTlsChain
Updates
UsageStats
User
UserApiKey
UserList
UserMetadata
UserSecurity
UserSecurityRealm
ZookeeperNodeInfo
ZookeeperSummary
- Changes to index allocation and API
- Script reference
- Glossary
- Release notes
- Elastic Cloud Enterprise 3.0.0
- Elastic Cloud Enterprise 2.13.4
- Elastic Cloud Enterprise 2.13.3
- Elastic Cloud Enterprise 2.13.2
- Elastic Cloud Enterprise 2.13.1
- Elastic Cloud Enterprise 2.13.0
- Elastic Cloud Enterprise 2.12.4
- Elastic Cloud Enterprise 2.12.3
- Elastic Cloud Enterprise 2.12.2
- Elastic Cloud Enterprise 2.12.1
- Elastic Cloud Enterprise 2.12.0
- Elastic Cloud Enterprise 2.11.2
- Elastic Cloud Enterprise 2.11.1
- Elastic Cloud Enterprise 2.11.0
- Elastic Cloud Enterprise 2.10.1
- Elastic Cloud Enterprise 2.10.0
- Elastic Cloud Enterprise 2.9.2
- Elastic Cloud Enterprise 2.9.1
- Elastic Cloud Enterprise 2.9.0
- Elastic Cloud Enterprise 2.8.1
- Elastic Cloud Enterprise 2.8.0
- Elastic Cloud Enterprise 2.7.2
- Elastic Cloud Enterprise 2.7.1
- Elastic Cloud Enterprise 2.7.0
- Elastic Cloud Enterprise 2.6.2
- Elastic Cloud Enterprise 2.6.1
- Elastic Cloud Enterprise 2.6.0
- Elastic Cloud Enterprise 2.5.1
- Elastic Cloud Enterprise 2.5.0
- Elastic Cloud Enterprise 2.4.3
- Elastic Cloud Enterprise 2.4.2
- Elastic Cloud Enterprise 2.4.1
- Elastic Cloud Enterprise 2.4.0
- Elastic Cloud Enterprise 2.3.2
- Elastic Cloud Enterprise 2.3.1
- Elastic Cloud Enterprise 2.3.0
- Elastic Cloud Enterprise 2.2.3
- Elastic Cloud Enterprise 2.2.2
- Elastic Cloud Enterprise 2.2.1
- Elastic Cloud Enterprise 2.2.0
- Elastic Cloud Enterprise 2.1.1
- Elastic Cloud Enterprise 2.1.0
- Elastic Cloud Enterprise 2.0.1
- Elastic Cloud Enterprise 2.0.0
- Elastic Cloud Enterprise 1.1.5
- Elastic Cloud Enterprise 1.1.4
- Elastic Cloud Enterprise 1.1.3
- Elastic Cloud Enterprise 1.1.2
- Elastic Cloud Enterprise 1.1.1
- Elastic Cloud Enterprise 1.1.0
- Elastic Cloud Enterprise 1.0.2
- Elastic Cloud Enterprise 1.0.1
- Elastic Cloud Enterprise 1.0.0
- What’s new with the Elastic Stack
- About this product
Deployment autoscaling
editDeployment autoscaling
editAutoscaling helps you to more easily manage your deployments by adjusting their available resources automatically. Currently, autoscaling is supported for Elasticsearch data tiers and machine learning nodes. Check the following sections to learn more:
You can also have a look at our autoscaling example, as well as a sample request to create an autoscaled deployment through the API.
Overview
editWhen you first create a deployment it can be challenging to determine the amount of storage your data nodes will require. The same is relevant for the amount of memory you want to allocate to your machine learning nodes. It can become even more challenging to predict these requirements for weeks or months into the future. In an ideal scenario, these resources should be sized to both ensure efficient performance and resiliency, and to avoid excess costs. Autoscaling can help with this balance by adjusting the resources available to a deployment automatically as loads change over time, reducing the need for monitoring and manual intervention.
Currently, autoscaling behavior is as follows:
-
Data tiers
- Each Elasticsearch data tier scales upward based on the amount of available storage. When we detect more storage is needed, autoscaling will scale up each data tier independently to ensure you can continue and ingest more data to your hot and content tier, or move data to the warm, cold, or frozen data tiers.
- In addition to scaling up existing data tiers, a new data tier will be automatically added when necessary, based on your index lifecycle management policies.
- To control the maximum size of each data tier and ensure it will not scale above a certain size, you can use the maximum size per zone field.
- Autoscaling based on memory or CPU, as well as autoscaling downward, is not currently supported. In case you want to adjust the size of your data tier to add more memory or CPU, or in case you deleted data and want to scale it down, you can set the current size per zone of each data tier manually.
-
Machine learning nodes
- Machine learning nodes can scale upward and downward based on the configured machine learning jobs.
- When a machine learning job is created, if there are no machine learning nodes in your deployment, the autoscaling mechanism will automatically add machine learning nodes. Similarly, after a period of no active machine learning jobs, any enabled machine learning nodes are disabled automatically.
- To control the maximum size of your machine learning nodes and ensure they will not scale above a certain size, you can use the maximum size per zone field.
- To control the minimum size of your machine learning nodes and ensure the autoscaling mechanism will not scale machine learning below a certain size, you can use the minimum size per zone field.
- The determination of when to scale is based on the expected memory requirements for the currently configured machine learning jobs.
For any Elastic Cloud Enterprise Elasticsearch component the number of availability zones is not affected by autoscaling. You can always set the number of availability zones manually and the autoscaling mechanism will add or remove capacity per availability zone.
When does autoscaling occur?
editSeveral factors determine when data tiers or machine learning nodes are scaled.
For a data tier, an autoscaling event can be triggered in the following cases:
- Based on an assessment of how shards are currently allocated, and the amount of storage and buffer space currently available.
- When past behavior on a hot tier, over the duration of the forecast window, indicates that the influx of data can increase significantly in the near future. Refer to Reactive storage decider and Proactive storage decider for more detail.
- Through ILM policies. For example, if a deployment has only hot nodes and autoscaling is enabled, it automatically creates warm or cold nodes, if an ILM policy is trying to move data from hot to warm or cold nodes.
On machine learning nodes, scaling is determined by an estimate of the memory requirement for the currently configured jobs. When a new machine learning job tries to start, it looks for a node with adequate native memory capacity. If one cannot be found, it stays in an opening
state. If this waiting job exceeds the queueing limit set in the machine learning decider, a scale up is requested. Conversely, as machine learning jobs run, their memory usage might decrease or other running jobs might finish or close. In this case, if the duration of decreased resource usage exceeds the set value for down_scale_delay
, a scale down is requested. Check Machine learning decider for more detail. To learn more about machine learning jobs in general, check Create anomaly detection jobs.
On a highly available deployment, autoscaling events are always applied to instances in each availability zone simultaneously, to ensure consistency.
Notifications
editIn the event that a data tier or machine learning node scales up to its maximum possible size, a notice appears on the deployment overview page prompting you to adjust your autoscaling settings in order to ensure optimal performance.
A warning is also issued in the ECE service-constructor
logs with the field labels.autoscaling_notification_type
and a value of data-tier-at-limit
(for a fully scaled data tier) or ml-tier-at-limit
(for a fully scaled machine learning node). The warning is indexed in the logging-and-metrics
deployment, so you can use that event to configure an email notification.
Restrictions and limitations
editThe following are known limitations and restrictions with autoscaling:
- Autoscaling is available starting in Elastic stack version 7.11.
- Autoscaling will not run if the cluster is unhealthy or if the last Elasticsearch plan failed.
- Autoscaling is not available in Elastic Cloud Enterprise on custom deployment templates. To enable autoscaling you need to use one of the built-in deployment templates. Note that you can modify the built-in deployment templates, for example to adjust some of the autoscaling default settings.
- In the event that an override is set for the disk quota multiplier for an instance by means of the Instance Overrides API, autoscaling does not use the new override settings. Autoscaling always adjusts storage and memory capacity based on the memory to storage ratio that is set for each default instance configuration. It’s recommended to avoid adjusting the disk quote multiplier for an instance that uses autoscaling, since the setting is not used.
Enable or disable autoscaling
editTo enable or disable autoscaling on a deployment:
- Log into the Cloud UI.
-
On the Deployments page, select your deployment.
Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.
- In your deployment menu, select Edit.
- Select or deselect Autoscale this deployment.
- Select Confirm to have the autoscaling change and any other settings take effect. All plan changes are shown on the Deployment Activity page.
When autoscaling has been enabled, the data tiers and machine learning nodes resize according to the autoscaling settings. Current sizes are shown on the deployment overview page.
When autoscaling has been disabled, you need to adjust the size of data tiers and machine learning nodes manually.
Update your autoscaling settings
editEach autoscaling setting is configured with a default value. You can adjust these if necessary, as follows:
- Log into the Cloud UI.
-
On the Deployments page, select your deployment.
Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.
- In your deployment menu, select Edit.
-
To update a data tier:
- Select Edit Settings on the data tier that you want to adjust. A panel opens, showing both the current size per availability zone and the maximum size per availability zone.
- Set the Maximum size per zone to the largest amount of resources that should be allocated to the data tier automatically. The resources will not scale above this value.
- You can also update the Current size per zone. If you update this setting to match the Maximum size per zone, the data tier will remain fixed at that size.
- For a hot data tier, under Advanced settings you can also adjust the Forecast window. This is the duration of time, up to the present, for which past storage usage is assessed in order to predict when additional storage is needed.
- Select Update. The Edit Settings pane closes.
- Select Save to apply the changes to your deployment.
-
To update machine learning nodes:
- Select Edit Settings on the machine learning nodes. A panel opens, showing the minimum and maximum size per availability zone.
- Set the Minimum size per zone and Maximum size per zone to the smallest and largest amount of resources, respectively, that should be allocated to the nodes automatically. The resources allocated to machine learning will not exceed these values. If you set these two settings to the same value, the machine learning node will remain fixed at that size.
- Select Update. The Edit Settings pane closes.
- Select Save to apply the changes to your deployment.
You can also view our example of how the autoscaling settings work.
On Elastic Cloud Enterprise, system-owned deployment templates include the default values for all deployment autoscaling settings.
On this page