- Elasticsearch Guide: other versions:
- What is Elasticsearch?
- What’s new in 8.4
- Set up Elasticsearch
- Installing Elasticsearch
- Run Elasticsearch locally
- Configuring Elasticsearch
- Important Elasticsearch configuration
- Secure settings
- Auditing settings
- Circuit breaker settings
- Cluster-level shard allocation and routing settings
- Cross-cluster replication settings
- Discovery and cluster formation settings
- Field data cache settings
- Health Diagnostic settings
- Index lifecycle management settings
- Index management settings
- Index recovery settings
- Indexing buffer settings
- License settings
- Local gateway settings
- Logging
- Machine learning settings
- Monitoring settings
- Node
- Networking
- Node query cache settings
- Search settings
- Security settings
- Shard request cache settings
- Snapshot and restore settings
- Transforms settings
- Thread pools
- Watcher settings
- Advanced configuration
- Important system configuration
- Bootstrap Checks
- Heap size check
- File descriptor check
- Memory lock check
- Maximum number of threads check
- Max file size check
- Maximum size virtual memory check
- Maximum map count check
- Client JVM check
- Use serial collector check
- System call filter check
- OnError and OnOutOfMemoryError checks
- Early-access check
- G1GC check
- All permission check
- Discovery configuration check
- Bootstrap Checks for X-Pack
- Starting Elasticsearch
- Stopping Elasticsearch
- Discovery and cluster formation
- Add and remove nodes in your cluster
- Full-cluster restart and rolling restart
- Remote clusters
- Plugins
- Upgrade Elasticsearch
- Index modules
- Mapping
- Text analysis
- Overview
- Concepts
- Configure text analysis
- Built-in analyzer reference
- Tokenizer reference
- Token filter reference
- Apostrophe
- ASCII folding
- CJK bigram
- CJK width
- Classic
- Common grams
- Conditional
- Decimal digit
- Delimited payload
- Dictionary decompounder
- Edge n-gram
- Elision
- Fingerprint
- Flatten graph
- Hunspell
- Hyphenation decompounder
- Keep types
- Keep words
- Keyword marker
- Keyword repeat
- KStem
- Length
- Limit token count
- Lowercase
- MinHash
- Multiplexer
- N-gram
- Normalization
- Pattern capture
- Pattern replace
- Phonetic
- Porter stem
- Predicate script
- Remove duplicates
- Reverse
- Shingle
- Snowball
- Stemmer
- Stemmer override
- Stop
- Synonym
- Synonym graph
- Trim
- Truncate
- Unique
- Uppercase
- Word delimiter
- Word delimiter graph
- Character filters reference
- Normalizers
- Index templates
- Data streams
- Ingest pipelines
- Example: Parse logs
- Enrich your data
- Processor reference
- Append
- Attachment
- Bytes
- Circle
- Community ID
- Convert
- CSV
- Date
- Date index name
- Dissect
- Dot expander
- Drop
- Enrich
- Fail
- Fingerprint
- Foreach
- GeoIP
- Grok
- Gsub
- HTML strip
- Inference
- Join
- JSON
- KV
- Lowercase
- Network direction
- Pipeline
- Registered domain
- Remove
- Rename
- Script
- Set
- Set security user
- Sort
- Split
- Trim
- Uppercase
- URL decode
- URI parts
- User agent
- Aliases
- Search your data
- Collapse search results
- Filter search results
- Highlighting
- Long-running searches
- Near real-time search
- Paginate search results
- Retrieve inner hits
- Retrieve selected fields
- Search across clusters
- Search multiple data streams and indices
- Search shard routing
- Search templates
- Sort search results
- kNN search
- Query DSL
- Aggregations
- Bucket aggregations
- Adjacency matrix
- Auto-interval date histogram
- Categorize text
- Children
- Composite
- Date histogram
- Date range
- Diversified sampler
- Filter
- Filters
- Frequent items
- Geo-distance
- Geohash grid
- Geohex grid
- Geotile grid
- Global
- Histogram
- IP prefix
- IP range
- Missing
- Multi Terms
- Nested
- Parent
- Random sampler
- Range
- Rare terms
- Reverse nested
- Sampler
- Significant terms
- Significant text
- Terms
- Variable width histogram
- Subtleties of bucketing range fields
- Metrics aggregations
- Pipeline aggregations
- Average bucket
- Bucket script
- Bucket count K-S test
- Bucket correlation
- Bucket selector
- Bucket sort
- Change point
- Cumulative cardinality
- Cumulative sum
- Derivative
- Extended stats bucket
- Inference bucket
- Max bucket
- Min bucket
- Moving function
- Moving percentiles
- Normalize
- Percentiles bucket
- Serial differencing
- Stats bucket
- Sum bucket
- Bucket aggregations
- EQL
- SQL
- Overview
- Getting Started with SQL
- Conventions and Terminology
- Security
- SQL REST API
- SQL Translate API
- SQL CLI
- SQL JDBC
- SQL ODBC
- SQL Client Applications
- SQL Language
- Functions and Operators
- Comparison Operators
- Logical Operators
- Math Operators
- Cast Operators
- LIKE and RLIKE Operators
- Aggregate Functions
- Grouping Functions
- Date/Time and Interval Functions and Operators
- Full-Text Search Functions
- Mathematical Functions
- String Functions
- Type Conversion Functions
- Geo Functions
- Conditional Functions And Expressions
- System Functions
- Reserved keywords
- SQL Limitations
- Scripting
- Data management
- ILM: Manage the index lifecycle
- Tutorial: Customize built-in policies
- Tutorial: Automate rollover
- Index management in Kibana
- Overview
- Concepts
- Index lifecycle actions
- Configure a lifecycle policy
- Migrate index allocation filters to node roles
- Troubleshooting index lifecycle management errors
- Start and stop index lifecycle management
- Manage existing indices
- Skip rollover
- Restore a managed data stream or index
- Data tiers
- Autoscaling
- Monitor a cluster
- Roll up or transform your data
- Set up a cluster for high availability
- Snapshot and restore
- Secure the Elastic Stack
- Elasticsearch security principles
- Start the Elastic Stack with security enabled automatically
- Manually configure security
- Updating node security certificates
- User authentication
- Built-in users
- Service accounts
- Internal users
- Token-based authentication services
- User profiles
- Realms
- Realm chains
- Security domains
- Active Directory user authentication
- File-based user authentication
- LDAP user authentication
- Native user authentication
- OpenID Connect authentication
- PKI user authentication
- SAML authentication
- Kerberos authentication
- JWT authentication
- Integrating with other authentication systems
- Enabling anonymous access
- Looking up users without authentication
- Controlling the user cache
- Configuring SAML single-sign-on on the Elastic Stack
- Configuring single sign-on to the Elastic Stack using OpenID Connect
- User authorization
- Built-in roles
- Defining roles
- Security privileges
- Document level security
- Field level security
- Granting privileges for data streams and aliases
- Mapping users and groups to roles
- Setting up field and document level security
- Submitting requests on behalf of other users
- Configuring authorization delegation
- Customizing roles and authorization
- Enable audit logging
- Restricting connections with IP filtering
- Securing clients and integrations
- Operator privileges
- Troubleshooting
- Some settings are not returned via the nodes settings API
- Authorization exceptions
- Users command fails due to extra arguments
- Users are frequently locked out of Active Directory
- Certificate verification fails for curl on Mac
- SSLHandshakeException causes connections to fail
- Common SSL/TLS exceptions
- Common Kerberos exceptions
- Common SAML issues
- Internal Server Error in Kibana
- Setup-passwords command fails due to connection failure
- Failures due to relocation of the configuration files
- Limitations
- Watcher
- Command line tools
- elasticsearch-certgen
- elasticsearch-certutil
- elasticsearch-create-enrollment-token
- elasticsearch-croneval
- elasticsearch-keystore
- elasticsearch-node
- elasticsearch-reconfigure-node
- elasticsearch-reset-password
- elasticsearch-saml-metadata
- elasticsearch-service-tokens
- elasticsearch-setup-passwords
- elasticsearch-shard
- elasticsearch-syskeygen
- elasticsearch-users
- How to
- Troubleshooting
- Fix common cluster issues
- Diagnose unassigned shards
- Add a missing tier to the system
- Allow Elasticsearch to allocate the data in the system
- Allow Elasticsearch to allocate the index
- Indices mix index allocation filters with data tiers node roles to move through data tiers
- Not enough nodes to allocate all shard replicas
- Total number of shards for an index on a single node exceeded
- Total number of shards per node has been reached
- Troubleshooting corruption
- Start index lifecycle management
- Start Snapshot Lifecycle Management
- Restore from snapshot
- Multiple deployments writing to the same snapshot repository
- Troubleshooting discovery
- Troubleshooting monitoring
- Troubleshooting transforms
- Troubleshooting Watcher
- Troubleshooting searches
- REST APIs
- API conventions
- Common options
- REST API compatibility
- Autoscaling APIs
- Compact and aligned text (CAT) APIs
- cat aliases
- cat allocation
- cat anomaly detectors
- cat count
- cat data frame analytics
- cat datafeeds
- cat fielddata
- cat health
- cat indices
- cat master
- cat nodeattrs
- cat nodes
- cat pending tasks
- cat plugins
- cat recovery
- cat repositories
- cat segments
- cat shards
- cat snapshots
- cat task management
- cat templates
- cat thread pool
- cat trained model
- cat transforms
- Cluster APIs
- Cluster allocation explain
- Cluster get settings
- Cluster health
- Health
- Cluster reroute
- Cluster state
- Cluster stats
- Cluster update settings
- Nodes feature usage
- Nodes hot threads
- Nodes info
- Nodes reload secure settings
- Nodes stats
- Pending cluster tasks
- Remote cluster info
- Task management
- Voting configuration exclusions
- Create or update desired nodes
- Get desired nodes
- Delete desired nodes
- Cross-cluster replication APIs
- Data stream APIs
- Document APIs
- Enrich APIs
- EQL APIs
- Features APIs
- Fleet APIs
- Find structure API
- Graph explore API
- Index APIs
- Alias exists
- Aliases
- Analyze
- Analyze index disk usage
- Clear cache
- Clone index
- Close index
- Create index
- Create or update alias
- Create or update component template
- Create or update index template
- Create or update index template (legacy)
- Delete component template
- Delete dangling index
- Delete alias
- Delete index
- Delete index template
- Delete index template (legacy)
- Exists
- Field usage stats
- Flush
- Force merge
- Get alias
- Get component template
- Get field mapping
- Get index
- Get index settings
- Get index template
- Get index template (legacy)
- Get mapping
- Import dangling index
- Index recovery
- Index segments
- Index shard stores
- Index stats
- Index template exists (legacy)
- List dangling indices
- Open index
- Refresh
- Resolve index
- Rollover
- Shrink index
- Simulate index
- Simulate template
- Split index
- Unfreeze index
- Update index settings
- Update mapping
- Index lifecycle management APIs
- Create or update lifecycle policy
- Get policy
- Delete policy
- Move to step
- Remove policy
- Retry policy
- Get index lifecycle management status
- Explain lifecycle
- Start index lifecycle management
- Stop index lifecycle management
- Migrate indices, ILM policies, and legacy, composable and component templates to data tiers routing
- Ingest APIs
- Info API
- Licensing APIs
- Logstash APIs
- Machine learning APIs
- Machine learning anomaly detection APIs
- Add events to calendar
- Add jobs to calendar
- Close jobs
- Create jobs
- Create calendars
- Create datafeeds
- Create filters
- Delete calendars
- Delete datafeeds
- Delete events from calendar
- Delete filters
- Delete forecasts
- Delete jobs
- Delete jobs from calendar
- Delete model snapshots
- Delete expired data
- Estimate model memory
- Flush jobs
- Forecast jobs
- Get buckets
- Get calendars
- Get categories
- Get datafeeds
- Get datafeed statistics
- Get influencers
- Get jobs
- Get job statistics
- Get model snapshots
- Get model snapshot upgrade statistics
- Get overall buckets
- Get scheduled events
- Get filters
- Get records
- Open jobs
- Post data to jobs
- Preview datafeeds
- Reset jobs
- Revert model snapshots
- Start datafeeds
- Stop datafeeds
- Update datafeeds
- Update filters
- Update jobs
- Update model snapshots
- Upgrade model snapshots
- Machine learning data frame analytics APIs
- Create data frame analytics jobs
- Delete data frame analytics jobs
- Evaluate data frame analytics
- Explain data frame analytics
- Get data frame analytics jobs
- Get data frame analytics jobs stats
- Preview data frame analytics
- Start data frame analytics jobs
- Stop data frame analytics jobs
- Update data frame analytics jobs
- Machine learning trained model APIs
- Migration APIs
- Node lifecycle APIs
- Reload search analyzers API
- Repositories metering APIs
- Rollup APIs
- Script APIs
- Search APIs
- Searchable snapshots APIs
- Security APIs
- Authenticate
- Change passwords
- Clear cache
- Clear roles cache
- Clear privileges cache
- Clear API key cache
- Clear service account token caches
- Create API keys
- Create or update application privileges
- Create or update role mappings
- Create or update roles
- Create or update users
- Create service account tokens
- Delegate PKI authentication
- Delete application privileges
- Delete role mappings
- Delete roles
- Delete service account token
- Delete users
- Disable users
- Enable users
- Enroll Kibana
- Enroll node
- Get API key information
- Get application privileges
- Get builtin privileges
- Get role mappings
- Get roles
- Get service accounts
- Get service account credentials
- Get token
- Get user privileges
- Get users
- Grant API keys
- Has privileges
- Invalidate API key
- Invalidate token
- OpenID Connect prepare authentication
- OpenID Connect authenticate
- OpenID Connect logout
- Query API key information
- Update API key
- SAML prepare authentication
- SAML authenticate
- SAML logout
- SAML invalidate
- SAML complete logout
- SAML service provider metadata
- SSL certificate
- Activate user profile
- Disable user profile
- Enable user profile
- Get user profile
- Suggest user profile
- Update user profile data
- Has privileges user profile
- Snapshot and restore APIs
- Snapshot lifecycle management APIs
- SQL APIs
- Transform APIs
- Usage API
- Watcher APIs
- Definitions
- Migration guide
- Release notes
- Elasticsearch version 8.4.3
- Elasticsearch version 8.4.2
- Elasticsearch version 8.4.1
- Elasticsearch version 8.4.0
- Elasticsearch version 8.3.3
- Elasticsearch version 8.3.2
- Elasticsearch version 8.3.1
- Elasticsearch version 8.3.0
- Elasticsearch version 8.2.3
- Elasticsearch version 8.2.2
- Elasticsearch version 8.2.1
- Elasticsearch version 8.2.0
- Elasticsearch version 8.1.3
- Elasticsearch version 8.1.2
- Elasticsearch version 8.1.1
- Elasticsearch version 8.1.0
- Elasticsearch version 8.0.1
- Elasticsearch version 8.0.0
- Elasticsearch version 8.0.0-rc2
- Elasticsearch version 8.0.0-rc1
- Elasticsearch version 8.0.0-beta1
- Elasticsearch version 8.0.0-alpha2
- Elasticsearch version 8.0.0-alpha1
- Dependencies and versions
Get data frame analytics jobs statistics API
editGet data frame analytics jobs statistics API
editRetrieves usage information for data frame analytics jobs.
Request
editGET _ml/data_frame/analytics/<data_frame_analytics_id>/_stats
GET _ml/data_frame/analytics/<data_frame_analytics_id>,<data_frame_analytics_id>/_stats
GET _ml/data_frame/analytics/_stats
GET _ml/data_frame/analytics/_all/_stats
GET _ml/data_frame/analytics/*/_stats
Prerequisites
editRequires the monitor_ml
cluster privilege. This privilege is included in the
machine_learning_user
built-in role.
Path parameters
edit-
<data_frame_analytics_id>
- (Optional, string) Identifier for the data frame analytics job. If you do not specify this option, the API returns information for the first hundred data frame analytics jobs.
Query parameters
edit-
allow_no_match
-
(Optional, Boolean) Specifies what to do when the request:
- Contains wildcard expressions and there are no data frame analytics jobs that match.
-
Contains the
_all
string or no identifiers and there are no matches. - Contains wildcard expressions and there are only partial matches.
The default value is
true
, which returns an emptydata_frame_analytics
array when there are no matches and the subset of results when there are partial matches. If this parameter isfalse
, the request returns a404
status code when there are no matches or only partial matches. -
from
-
(Optional, integer)
Skips the specified number of data frame analytics jobs. The default value is
0
. -
size
-
(Optional, integer)
Specifies the maximum number of data frame analytics jobs to obtain. The default value
is
100
. -
verbose
-
(Optional, Boolean)
Defines whether the stats response should be verbose. The default value is
false
.
Response body
edit-
data_frame_analytics
-
(array) An array of objects that contain usage information for data frame analytics jobs, which are sorted by the
id
value in ascending order.Properties of data frame analytics job usage resources
-
analysis_stats
-
(object) An object containing information about the analysis job.
Properties of
analysis_stats
-
classification_stats
-
(object) An object containing information about the classification analysis job.
Properties of
classification_stats
-
hyperparameters
-
(object) An object containing the parameters of the classification analysis job.
Properties of
hyperparameters
-
alpha
- (double) Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.
-
class_assignment_objective
-
(string)
Defines the objective to optimize when assigning class labels:
maximize_accuracy
ormaximize_minimum_recall
. When maximizing accuracy, class labels are chosen to maximize the number of correct predictions. When maximizing minimum recall, labels are chosen to maximize the minimum recall for any class. Defaults tomaximize_minimum_recall
. -
downsample_factor
- (double) Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. For more information about shrinkage, refer to this wiki article. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.
-
eta
- (double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. For more information about shrinkage, refer to this wiki article. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.
-
eta_growth_rate_per_tree
-
(double)
Advanced configuration option. Specifies the rate at which
eta
increases for each new tree that is added to the forest. For example, a rate of 1.05 increaseseta
by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2. -
feature_bag_fraction
- (double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.
-
gamma
- (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.
-
lambda
- (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.
-
max_attempts_to_add_tree
- (integer) If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.
-
max_optimization_rounds_per_hyperparameter
- (integer) Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.
-
max_trees
- (integer) Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.
-
num_folds
- (integer) The maximum number of folds for the cross-validation procedure.
-
num_splits_per_feature
- (integer) Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.
-
soft_tree_depth_limit
-
(double)
Advanced configuration option. Machine learning uses loss guided tree growing, which
means that the decision trees grow where the regularized loss decreases most
quickly. This soft limit combines with the
soft_tree_depth_tolerance
to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0. -
soft_tree_depth_tolerance
-
(double)
Advanced configuration option. This option controls how quickly the regularized
loss increases when the tree depth exceeds
soft_tree_depth_limit
. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.
-
-
iteration
- (integer) The number of iterations on the analysis.
-
timestamp
- (date) The timestamp when the statistics were reported in milliseconds since the epoch.
-
timing_stats
-
(object) An object containing time statistics about the data frame analytics job.
Properties of
timing_stats
-
elapsed_time
- (integer) Runtime of the analysis in milliseconds.
-
iteration_time
- (integer) Runtime of the latest iteration of the analysis in milliseconds.
-
-
validation_loss
-
(object) An object containing information about validation loss.
Properties of
validation_loss
-
fold_values
- Validation loss values for every added decision tree during the forest growing procedure.
-
loss_type
-
(string)
The type of the loss metric. For example,
binomial_logistic
.
-
-
-
outlier_detection_stats
-
(object) An object containing information about the outlier detection job.
Properties of
outlier_detection_stats
-
parameters
-
(object) The list of job parameters specified by the user or determined by algorithmic heuristics.
Properties of
parameters
-
compute_feature_influence
-
(Boolean)
Specifies whether the feature influence calculation is enabled. Defaults to
true
. -
feature_influence_threshold
-
(double)
The minimum outlier score that a document needs to have in order to calculate its
feature influence score. Value range: 0-1 (
0.1
by default). -
method
-
(string)
The method that outlier detection uses. Available methods are
lof
,ldof
,distance_kth_nn
,distance_knn
, andensemble
. The default value isensemble
, which means that outlier detection uses an ensemble of different methods and normalises and combines their individual outlier scores to obtain the overall outlier score. -
n_neighbors
- (integer) Defines the value for how many nearest neighbors each method of outlier detection uses to calculate its outlier score. When the value is not set, different values are used for different ensemble members. This default behavior helps improve the diversity in the ensemble; only override it if you are confident that the value you choose is appropriate for the data set.
-
outlier_fraction
- (double) The proportion of the data set that is assumed to be outlying prior to outlier detection. For example, 0.05 means it is assumed that 5% of values are real outliers and 95% are inliers.
-
standardization_enabled
-
(Boolean)
If
true
, the following operation is performed on the columns before computing outlier scores: (x_i - mean(x_i)) / sd(x_i). Defaults totrue
. For more information about this concept, see Wikipedia.
-
-
timestamp
- (date) The timestamp when the statistics were reported in milliseconds since the epoch.
-
timing_stats
-
(object) An object containing time statistics about the data frame analytics job.
Property of
timing_stats
-
elapsed_time
- (integer) Runtime of the analysis in milliseconds.
-
-
-
regression_stats
-
(object) An object containing information about the regression analysis job.
Properties of
regression_stats
-
hyperparameters
-
(object) An object containing the parameters of the regression analysis job.
Properties of
hyperparameters
-
alpha
- (double) Advanced configuration option. Machine learning uses loss guided tree growing, which means that the decision trees grow where the regularized loss decreases most quickly. This parameter affects loss calculations by acting as a multiplier of the tree depth. Higher alpha values result in shallower trees and faster training times. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to zero.
-
downsample_factor
- (double) Advanced configuration option. Controls the fraction of data that is used to compute the derivatives of the loss function for tree training. A small value results in the use of a small fraction of the data. If this value is set to be less than 1, accuracy typically improves. However, too small a value may result in poor convergence for the ensemble and so require more trees. For more information about shrinkage, refer to this wiki article. By default, this value is calculated during hyperparameter optimization. It must be greater than zero and less than or equal to 1.
-
eta
- (double) Advanced configuration option. The shrinkage applied to the weights. Smaller values result in larger forests which have a better generalization error. However, larger forests cause slower training. For more information about shrinkage, refer to this wiki article. By default, this value is calculated during hyperparameter optimization. It must be a value between 0.001 and 1.
-
eta_growth_rate_per_tree
-
(double)
Advanced configuration option. Specifies the rate at which
eta
increases for each new tree that is added to the forest. For example, a rate of 1.05 increaseseta
by 5% for each extra tree. By default, this value is calculated during hyperparameter optimization. It must be between 0.5 and 2. -
feature_bag_fraction
- (double) Advanced configuration option. Defines the fraction of features that will be used when selecting a random bag for each candidate split. By default, this value is calculated during hyperparameter optimization.
-
gamma
- (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies a linear penalty associated with the size of individual trees in the forest. A high gamma value causes training to prefer small trees. A small gamma value results in larger individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.
-
lambda
- (double) Advanced configuration option. Regularization parameter to prevent overfitting on the training data set. Multiplies an L2 regularization term which applies to leaf weights of the individual trees in the forest. A high lambda value causes training to favor small leaf weights. This behavior makes the prediction function smoother at the expense of potentially not being able to capture relevant relationships between the features and the dependent variable. A small lambda value results in large individual trees and slower training. By default, this value is calculated during hyperparameter optimization. It must be a nonnegative value.
-
max_attempts_to_add_tree
- (integer) If the algorithm fails to determine a non-trivial tree (more than a single leaf), this parameter determines how many of such consecutive failures are tolerated. Once the number of attempts exceeds the threshold, the forest training stops.
-
max_optimization_rounds_per_hyperparameter
- (integer) Advanced configuration option. A multiplier responsible for determining the maximum number of hyperparameter optimization steps in the Bayesian optimization procedure. The maximum number of steps is determined based on the number of undefined hyperparameters times the maximum optimization rounds per hyperparameter. By default, this value is calculated during hyperparameter optimization.
-
max_trees
- (integer) Advanced configuration option. Defines the maximum number of decision trees in the forest. The maximum value is 2000. By default, this value is calculated during hyperparameter optimization.
-
num_folds
- (integer) The maximum number of folds for the cross-validation procedure.
-
num_splits_per_feature
- (integer) Determines the maximum number of splits for every feature that can occur in a decision tree when the tree is trained.
-
soft_tree_depth_limit
-
(double)
Advanced configuration option. Machine learning uses loss guided tree growing, which
means that the decision trees grow where the regularized loss decreases most
quickly. This soft limit combines with the
soft_tree_depth_tolerance
to penalize trees that exceed the specified depth; the regularized loss increases quickly beyond this depth. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0. -
soft_tree_depth_tolerance
-
(double)
Advanced configuration option. This option controls how quickly the regularized
loss increases when the tree depth exceeds
soft_tree_depth_limit
. By default, this value is calculated during hyperparameter optimization. It must be greater than or equal to 0.01.
-
-
iteration
- (integer) The number of iterations on the analysis.
-
timestamp
- (date) The timestamp when the statistics were reported in milliseconds since the epoch.
-
timing_stats
-
(object) An object containing time statistics about the data frame analytics job.
Propertis of
timing_stats
-
elapsed_time
- (integer) Runtime of the analysis in milliseconds.
-
iteration_time
- (integer) Runtime of the latest iteration of the analysis in milliseconds.
-
-
validation_loss
-
(object) An object containing information about validation loss.
Properties of
validation_loss
-
fold_values
- (array of strings) Validation loss values for every added decision tree during the forest growing procedure.
-
loss_type
-
(string)
The type of the loss metric. For example,
binomial_logistic
.
-
-
-
-
assignment_explanation
- (string) For running jobs only, contains messages relating to the selection of a node to run the job.
-
data_counts
-
(object) An object that provides counts for the quantity of documents skipped, used in training, or available for testing.
Properties of
data_counts
-
skipped_docs_count
- (integer) The number of documents that are skipped during the analysis because they contained values that are not supported by the analysis. For example, outlier detection does not support missing fields so it skips documents with missing fields. Likewise, all types of analysis skip documents that contain arrays with more than one element.
-
test_docs_count
- (integer) The number of documents that are not used for training the model and can be used for testing.
-
training_docs_count
- (integer) The number of documents that are used for training the model.
-
-
id
- (string) The unique identifier of the data frame analytics job.
-
memory_usage
-
(Optional, object) An object describing memory usage of the analytics. It is present only after the job is started and memory usage is reported.
Properties of
memory_usage
-
memory_reestimate_bytes
-
(long)
This value is present when the
status
ishard_limit
and it is a new estimate of how much memory the job needs. -
peak_usage_bytes
- (long) The number of bytes used at the highest peak of memory usage.
-
status
-
(string) The memory usage status. May have one of the following values:
-
ok
: usage stayed below the limit. -
hard_limit
: usage surpassed the configured memory limit.
-
-
timestamp
- (date) The timestamp when memory usage was calculated.
-
-
node
-
(object) Contains properties for the node that runs the job. This information is available only for running jobs.
Properties of
node
-
attributes
-
(object)
Lists node attributes such as
ml.machine_memory
orml.max_open_jobs
settings. -
ephemeral_id
- (string) The ephemeral ID of the node.
-
id
- (string) The unique identifier of the node.
-
name
- (string) The node name.
-
transport_address
- (string) The host and port where transport HTTP connections are accepted.
-
-
progress
-
(array) The progress report of the data frame analytics job by phase.
Properties of phase objects
-
phase
-
(string) Defines the phase of the data frame analytics job. Possible phases:
-
reindexing
, -
loading_data
, -
computing_outliers
(for outlier detection only), -
feature_selection
(for regression and classification only), -
coarse_parameter_search
(for regression and classification only), -
fine_tuning_parameters
(for regression and classification only), -
final_training
(for regression and classification only), -
writing_results
, -
inference
(for regression and classification only).To learn more about the different phases, refer to How a data frame analytics job works.
-
-
progress_percent
- (integer) The progress that the data frame analytics job has made expressed in percentage.
-
-
state
-
(string) The status of the data frame analytics job, which can be one of the following
values:
failed
,started
,starting
,stopping
,stopped
.
-
Response codes
edit-
404
(Missing resources) -
If
allow_no_match
isfalse
, this code indicates that there are no resources that match the request or only partial matches for the request.
Examples
editThe following API retrieves usage information for the outlier detection data frame analytics job example:
GET _ml/data_frame/analytics/weblog-outliers/_stats
{ "count" : 1, "data_frame_analytics" : [ { "id" : "weblog-outliers", "state" : "stopped", "progress" : [ { "phase" : "reindexing", "progress_percent" : 100 }, { "phase" : "loading_data", "progress_percent" : 100 }, { "phase" : "computing_outliers", "progress_percent" : 100 }, { "phase" : "writing_results", "progress_percent" : 100 } ], "data_counts" : { "training_docs_count" : 1001, "test_docs_count" : 0, "skipped_docs_count" : 0 }, "memory_usage" : { "timestamp" : 1626264770206, "peak_usage_bytes" : 328011, "status" : "ok" }, "analysis_stats" : { "outlier_detection_stats" : { "timestamp" : 1626264770206, "parameters" : { "n_neighbors" : 0, "method" : "ensemble", "compute_feature_influence" : true, "feature_influence_threshold" : 0.1, "outlier_fraction" : 0.05, "standardization_enabled" : true }, "timing_stats" : { "elapsed_time" : 32 } } } } ] }