Elastic now providing distributions for OpenTelemetry SDKs

OpenTelemetry is a vendor-neutral observability framework and toolkit that supports telemetry signals such as traces, metrics, and logs in applications and distributed microservice-based architectures.

Driven by a set of standards, OpenTelemetry is designed to provide a consistent approach to instrumenting and observing application behavior. OpenTelemetry is an incubating project developed under the Cloud Native Computing Foundation (CNCF) umbrella and is currently the second most active project, topped only by Kubernetes.

You can read more on the OpenTelemetry website about the concepts, terminology, and techniques for adopting OpenTelemetry.

By adopting OpenTelemetry, software code can be instrumented in a vendor-agnostic fashion, with telemetry signals exported in a standardized format to one or more vendor backends, such as Elastic APM. Its design provides flexibility for application owners to switch out vendor backends with no code changes and use OpenTelemetry collectors to send telemetry data to multiple backends.

Because OpenTelemetry is not a vendor-specific solution, it is much easier for language ecosystems to adopt it and provide robust instrumentations. Vendors don’t have to implement specific instrumentations themselves anymore. OpenTelemetry is a standard, and it is in the interest of library developers to introduce and maintain instrumentations from which all consumers can benefit.

As a result, more instrumentation libraries are available and better kept up to date. If your company has open-source libraries, you can also contribute and create your own instrumentations to make it easier for your customers to adopt OpenTelemetry and benefit from richer traces, metrics, and logging in their applications.

Elastic is deeply involved in OpenTelemetry. In 2023, we donated the Elastic Common Schema, which is being merged with the Semantic Conventions. In 2024, we are in the process of donating our profiling agent based on eBPF. We also have multiple contributors to various areas of OpenTelemetry across the organization.

We are therefore committed to helping OpenTelemetry succeed, which means, in some cases, beginning to shift away from Elastic-specific components and recommend using OpenTelemetry components instead.

Elastic is committed to supporting and contributing to OpenTelemetry. Our APM solution already accepts native OTLP (OpenTelemetry Protocol) data, and many of our APM agents have already bridged data collection and transmission from applications instrumented using the OpenTelemetry APIs.

The next step on our journey is introducing Elastic distributions for the language SDKs and donating features upstream to the OpenTelemetry community by contributing to the OpenTelemetry SDK repositories.

An OpenTelemetry distribution is simply a customized version of one or more OpenTelemetry components. Each distribution extends the core functionality offered by the component while adhering to its API and existing features, utilizing built-in extension points.

With the release of Elastic distributions of the OpenTelemetry SDKs, we are extending our backing of OpenTelemetry as the preferred and recommended choice for instrumenting applications.

OpenTelemetry maintains and ships many language APIs and SDKs for observing applications using OpenTelemetry. The APIs provide a language-specific interface for instrumenting application code, while the SDK implements that API, enabling signals from observed applications to be collected and exported. 

Our current work extends the OpenTelemetry language SDKs to introduce additional features and ensure that the exported data provides the most robust compatibility with our current backend while it evolves to become more OpenTelemetry native. 

Additional features include reimplementing concepts currently available in the Elastic APM Agent but not part of the OpenTelemetry SDK. The distributions allow us to ship with opinionated defaults for all signals that are known to provide the best integration with Elastic’s Observability offering.

It’s undoubtedly possible to use the OpenTelemetry APIs to instrument code and then reference the OpenTelemetry SDK to enable the collection of the trace, metric, and log data that applications produce. Elastic APM accepts native OTLP data, so you can configure the OpenTelemetry SDK to export telemetry data directly to an Elastic backend. We refer to this setup as using the “vanilla” (a.k.a. “native”) OpenTelemetry SDK.

Work is ongoing to improve support for storing and presenting OpenTelemetry data natively in our backend so that we can drive our observability UIs directly from the data from the various telemetry signals. Our work focuses on ensuring that the Elastic-curated UIs can seamlessly handle the ECS and OpenTelemetry formats. Alongside this effort, we are working on distributions of the language SDKs to support customers looking to adopt OpenTelemetry-native instrumentation in their applications.

The current Elastic APM Agents support features such as central configuration and span compression that are not part of the OpenTelemetry specification as of today. We are investing our engineering expertise to bring those features to a broader audience by contributing them to OpenTelemetry. Because standardization takes time, we can more rapidly bring these features to the OpenTelemetry community and our customers by providing distributions.

We believe the responsible choice is to concentrate on enabling and encouraging customers to favor vendor-neutral instrumentation in their code and reap the benefits of OpenTelemetry. 

Distributions best serve our decision to fully adopt and recommend OpenTelemetry as the preferred solution for observing applications. By providing features that are currently unavailable in the “vanilla” OpenTelemetry SDK, we can support customers who want to adopt OpenTelemetry native, vendor-agnostic instrumentation in their applications while still providing the same set of features and backend capabilities they enjoy today with the existing APM Agents. By maintaining Elastic distributions, we can also better support our customers with enhancements and fixes outside of the release cycle of the “vanilla” OpenTelemetry SDKs, which we believe to be a crucial differentiating factor in choosing them.

Our vision is that Elastic will work with the OpenTelemetry community to donate features through the standardization processes and contribute the code to implement those in the native OpenTelemetry SDKs. In time, we hope to see many Elastic APM Agent-exclusive features transition into OpenTelemetry to the point where an Elastic distribution may no longer be necessary. In the meantime, we can deliver those capabilities via our OpenTelemetry distributions.

Application developers then have several options for instrumenting and collecting telemetry data from their applications:

1. Elastic APM Agent: The most fully featured, however, vendor-specific

2. Elastic APM Agent with OpenTelemetry Bridge: Vendor-neutral instrumentation API, but with known limitations:

   1. Only supports bridging of traces (no metrics support)

   2. Does not support OpenTelemetry span events

3. OpenTelemetry “vanilla” SDK: Fully supported today; however, it lacks some features of Elastic APM Agent, such as span compression

4. Elastic OpenTelemetry Distribution:

   1. Supports vendor-neutral instrumentation and no Elastic-specific configuration in code by default

   2. Recommended defaults when using Elastic Observability as a backend

   3. Use OpenTelemetry APIs to further customize our defaults; no new APIs to learn

While we continue to support all options to instrument your code for the foreseeable future, we think we are setting our customers up for success by introducing a fourth OpenTelemetry-native offering. We expect this will become the preferred default for Elastic customers in due time.

We currently have distributions in alpha release status for .NET and Java, with additional language distributions coming very soon. We encourage you to check out those repositories, try out the distributions, and provide feedback to us via issues. Your valued input allows us to refine our designs and steer our direction to ensure that our distributions delight consumers.

Learn about the alpha release of our new Elastic distribution of the OpenTelemetry SDK for .NET.