Configure tail-based sampling

edit

Enable tail-based sampling in the APM integration settings. When enabled, trace events are mapped to sampling policies. Each sampling policy must specify a sample rate, and can optionally specify other conditions. All of the policy conditions must be true for a trace event to match it.

Trace events are matched to policies in the order specified. Each policy list must conclude with a default policy — one that only specifies a sample rate. This default policy is used to catch remaining trace events that don’t match a stricter policy. Requiring this default policy ensures that traces are only dropped intentionally. If you enable tail-based sampling and send a transaction that does not match any of the policies, APM Server will reject the transaction with the error no matching policy.

Please note that from version 8.3.1 APM Server implements a default storage limit of 3GB, but, due to how the limit is calculated and enforced the actual disk space may still grow slightly over the limit.

Example configuration

edit

This example defines three tail-based sampling polices:

- sample_rate: 1 
  service.environment: production
  trace.name: "GET /very_important_route"
- sample_rate: .01 
  service.environment: production
  trace.name: "GET /not_important_route"
- sample_rate: .1 

Samples 100% of traces in production with the trace name "GET /very_important_route"

Samples 1% of traces in production with the trace name "GET /not_important_route"

Default policy to sample all remaining traces at 10%, e.g. traces in a different environment, like dev, or traces with any other name

Configuration reference

edit

Top-level tail-based sampling settings:

Enable Tail-based sampling

(bool) Enable and disable tail-based sampling.

Default: false

Interval

(duration) Synchronization interval for multiple APM Servers. Should be in the order of tens of seconds or low minutes.

Default: 1m

Policies

([]policy) Criteria used to match a root transaction to a sample rate. Order is important; the first policy on the list that an event matches is the winner. Each policy list must conclude with a default policy that only specifies a sample rate. The default policy is used to catch remaining trace events that don’t match a stricter policy.

Required when tail-based sampling is enabled.

Policy settings:

Sample rate

sample_rate

(int) The sample rate to apply to trace events matching this policy. Required in each policy.

The sample rate must be greater than 0 and less than or equal to 1. For example, a sample_rate of 0.01 means that 1% of trace events matching the policy will be sampled. A sample_rate of 1 means that 100% of trace events matching the policy will be sampled.

Trace name

trace.name

(string) The trace name for events to match a policy. A match occurs when the configured trace.name matches the transaction.name of the root transaction of a trace. A root transaction is any transaction without a parent.id.

Trace outcome

trace.outcome

(string) The trace outcome for events to match a policy. A match occurs when the configured trace.outcome matches a trace’s event.outcome field. Trace outcome can be success, failure, or unknown.

Service name

service.name

(string) The service name for events to match a policy.

Service Environment

service.environment

(string) The service environment for events to match a policy.