Instrumenting Go Source Code

Package apmecho provides middleware for the Echo web framework.

For each request, a transaction is stored in the request context, which can be obtained via echo.Context.Request().Context() in your handler.

import (
	"github.com/labstack/echo"
	"github.com/elastic/apm-agent-go/module/apmecho"
)

func main() {
	e := echo.New()
	e.Use(apmecho.Middleware())
	...
}

The apmecho middleware will recover panics and send them to Elastic APM, so you do not need to install the echo/middleware.Recover middleware.

Package apmgin provides middleware for the Gin web framework.

For each request, a transaction is stored in the request context, which can be obtained via gin.Context.Request.Context() in your handler.

import (
	"github.com/elastic/apm-agent-go/module/apmgin"
)

func main() {
	engine := gin.New()
	engine.Use(apmgin.Middleware(engine))
	...
}

The apmgin middleware will recover panics and send them to Elastic APM, so you do not need to install the gin.Recovery middleware.

Package apmgorilla provides middleware for the Gorilla Mux router.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/gorilla/mux"

	"github.com/elastic/apm-agent-go/module/apmgorilla"
)

func main() {
	router := mux.NewRouter()
	router.Use(apmgorilla.Middleware())
	...
}

The apmgorilla middleware will recover panics and send them to Elastic APM, so you do not need to install any other recovery middleware.

Package apmgrpc provides server and client interceptors for gRPC-Go. Server interceptors report transactions for each incoming request, while client interceptors report spans for each outgoing request. For each RPC served, a transaction is stored in the context passed into the method.

import (
	"github.com/elastic/apm-agent-go/module/apmgrpc"
)

func main() {
	server := grpc.NewServer(grpc.UnaryInterceptor(apmgrpc.NewUnaryServerInterceptor()))
	...
	conn, err := grpc.Dial(addr, grpc.WithUnaryInterceptor(apmgrpc.NewUnaryClientInterceptor()))
	...
}

The server interceptor can optionally be made to recover panics, in the same way as grpc_recovery. The apmgrpc server interceptor will always send panics it observes as errors to the Elastic APM server. If you want to recover panics but also want to continue using grpc_recovery, then you should ensure that it comes before the apmgrpc interceptor in the interceptor chain, or panics will not be captured by apmgrpc.

server := grpc.NewServer(grpc.UnaryInterceptor(
	apmgrpc.NewUnaryServerInterceptor(apmgrpc.WithRecovery()),
))
...

There is currently no support for intercepting at the stream level. Please file an issue and/or send a pull request if this is something you need.

Package apmhttp provides a low-level net/http middleware handler. Other web middleware should typically be based off this.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/elastic/apm-agent-go/module/apmhttp"
)

func main() {
	var myHandler http.Handler = ...
	tracedHandler := apmhttp.Wrap(myHandler)
}

The apmhttp handler will recover panics and send them to Elastic APM.

Package apmhttp also provides functions for instrumenting an http.Client or http.RoundTripper such that outgoing requests are traced as spans, if the request context includes a transaction.

import (
	"net/http"
	"golang.org/x/net/context/ctxhttp"
	"github.com/elastic/apm-agent-go/module/apmhttp"
)

var tracingClient = apmhttp.WrapClient(http.DefaultClient)

func serverHandler(w http.ResponseWriter, req *http.Request) {
	resp, err := ctxhttp.Get(req.Context(), tracingClient, "http://backend.local/foo")
	...
}

func main() {
	http.ListenAndServe(":8080", apmhttp.Wrap(serverHandler))
}

Package apmhttprouter provides a low-level middleware handler for httprouter.

For each request, a transaction is stored in the request context, which can be obtained via http.Request.Context() in your handler.

import (
	"github.com/julienschmidt/httprouter"

	"github.com/elastic/apm-agent-go/module/apmhttprouter"
)

func main() {
	router := httprouter.New()

	const route = "/my/route"
	router.GET(route, apmhttprouter.Wrap(h, route))
	...
}

httprouter does not provide a means of obtaining the matched route, hence the route must be passed into the wrapper.

Alternatively you can use the apmhttprouter.Router type, which wraps httprouter.Router, providing the same API and instrumenting added routes. To use this wrapper type, you should rewrite your use of httprouter.New to apmhttprouter.New; the returned type is *apmhttprouter.Router, and not *httprouter.Router.

import (
	"github.com/julienschmidt/httprouter"

	"github.com/elastic/apm-agent-go/module/apmhttprouter"
)

func main() {
	router := apmhttprouter.New()

	router.GET(route, h)
	...
}

Package apmlambda intercepts requests to your AWS Lambda function invocations.

Importing the package is enough to report the function invocations.

import (
	_ "github.com/elastic/apm-agent-go/module/apmlambda"
)

We currently do not expose the transactions via context; when we do, it will be necessary to make a small change to your code to call apmlambda.Start instead of lambda.Start.

Package apmsql provides a means of wrapping database/sql drivers so that queries and other executions are reported as spans within the current transaction.

To trace SQL queries, you should register drivers using apmsql.Register and obtain connections with apmsql.Open. The parameters are exactly the same as if you were to call sql.Register and sql.Open respectively.

As a convenience, we also provide packages which will automatically register popular drivers with apmsql.Register:

import (
	"github.com/elastic/apm-agent-go/module/apmsql"
	_ "github.com/elastic/apm-agent-go/module/apmsql/pq"
	_ "github.com/elastic/apm-agent-go/module/apmsql/sqlite3"
)

func main() {
	db, err := apmsql.Open("pq", "postgres://...")
	db, err := apmsql.Open("sqlite3", ":memory:")
}

Spans will be created for queries and other statement executions if the context methods are used, and the context includes a transaction.

Package apmgorm provides a means of instrumenting [gorm](http://gorm.io) database operations.

To trace GORM operations, import the appropriate apmgorm/dialects package (instead of the gorm/dialects package), and use apmgorm.Open (instead of gorm.Open). The parameters are exactly the same.

Once you have a *gorm.DB from apmgorm.Open, you can call apmgorm.WithContext to propagate a context containing a transaction to the operations:

import (
	"github.com/elastic/apm-agent-go/module/apmgorm"
	_ "github.com/elastic/apm-agent-go/module/apmgorm/dialects/postgres"
)

func main() {
	db, err := apmgorm.Open("postgres", "")
	...
	db = apmgorm.WithContext(ctx, db)
	db.Find(...) // creates a "SELECT FROM <foo>" span
}

Package apmgocql provides a means of instrumenting gocql so that queries are reported as spans within the current transaction.

To report gocql queries, you can construct an apmgocql.Observer and assign it to the QueryObserver and BatchObserver fields of gocql.ClusterConfig, or install it into a specific gocql.Query or gocql.Batch via their Observer methods.

Spans will be created for queries as long as they have context associated, and the context includes a transaction.

import (
	"github.com/gocql/gocql"

	"github.com/elastic/apm-agent-go/module/apmgocql"
)

func main() {
	observer := apmgocql.NewObserver()
	config := gocql.NewCluster("cassandra_host")
	config.QueryObserver = observer
	config.BatchObserver = observer

	session, err := config.CreateSession()
	...
	err = session.Query("SELECT * FROM foo").WithContext(ctx).Exec()
	...
}

To report a transaction, you call elasticapm.DefaultTracer.StartTransaction with the transaction name and type. This returns a Transaction object; the transaction can be customized with additional context before you call its End method to indicate that the transaction has completed. Once the transaction’s End method is called, it will be enqueued for sending to the Elastic APM server, and made available to the APM UI.

tx := elasticapm.DefaultTracer.StartTransaction("GET /api/v1", "request")
defer tx.End()
...
tx.Result = "HTTP 2xx"
tx.Context.SetTag("region", "us-east-1")

The agent supports sampling transactions: non-sampled transactions will be still be reported, but with limited context and without any spans. To determine whether a transaction is sampled, use the Transaction.Sampled method; if it returns false, you should avoid unnecessary storage or processing required for setting transaction context.

Once you have started a transaction, you can include it in a context object for propagating throughout the application. See context propagation for more details.

ctx = elasticapm.ContextWithTransaction(ctx, tx)

To report an operation within a transaction, you should use Transaction.StartSpan or elasticapm.StartSpan to start a span given a transaction or a context containing a transaction, respectively. Like a transaction, a span has a name and a type. In addition, a span can have a parent span within the same transaction. If the context provided to elasticapm.StartSpan contains a span, then that will be considered the parent. See context propagation for more details.

span, ctx := elasticapm.StartSpan(ctx, "SELECT FROM foo", "db.mysql.query")
defer span.End()

Transaction.StartSpan and elasticapm.StartSpan will always return a non-nil Span, even if the transaction is nil. It is always safe to defer a call to the span’s End method. If setting the span’s context would incur significant overhead, you may want to check if the span is dropped first, by calling the Span.Dropped method.

In Go, context is used to propagate request-scoped values along a call chain, potentially crossing between goroutines and between processes. For servers based on net/http, each request contains an independent context object, which allows adding values specific to that particular request.

When you start a transaction, you can add it to a context object using elasticapm.ContextWithTransaction. This context object can be later passed to elasticapm.TransactionFromContext to obtain the transaction, or into elasticapm.StartSpan to start a span.

The simplest way to create and propagate a span is by using elasticapm.StartSpan, which takes a context and returns a span. The span will be created as a child of the span most recently added to this context, or a transaction added to the context as described above. If the context contains neither a transaction nor a span, then the span will be dropped (i.e. will not be reported to the APM Server.)

For example, take a simple CRUD-type web service, which accepts requests over HTTP and then makes corresponding database queries. For each incoming request, a transaction will be started and added to the request context automatically. This context needs to be passed into method calls within the handler manually in order to create spans within that transaction, e.g. to measure the duration of SQL queries.

import (
	"net/http"

	"github.com/elastic/apm-agent-go"
	"github.com/elastic/apm-agent-go/module/apmhttp"
	"github.com/elastic/apm-agent-go/module/apmsql"
	_ "github.com/elastic/apm-agent-go/module/apmsql/pq"
)

var db *sql.DB

func init() {
	// apmsql.Open wraps sql.Open, in order
	// to add tracing to database operations.
	db, _ = apmsql.Open("postgres", "")
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", handleList)

	// apmhttp.Wrap instruments an http.Handler, in order
	// to report any request to this handler as a transaction,
	// and to store the transaction in the request's context.
	handler := apmhttp.Wrap(mux)
	http.ListenAndServe(":8080", handler)
}

func handleList(w http.ResponseWriter, req *http.Request) {
	// By passing the request context down to getList, getList can add spans to it.
	ctx := req.Context()
	getList(ctx)
	...
}

func getList(ctx context.Context) (
	// When getList is called with a context containing a transaction or span,
	// StartSpan creates a child span. In this example, getList is always called
	// with a context containing a transaction for the handler, so we should
	// expect to see something like:
	//
	//     Transaction: handleList
	//         Span: getList
	//             Span: SELECT FROM items
	//
	span, ctx := elasticapm.StartSpan(ctx, "getList", "custom")
	defer span.End()

	// NOTE: The context object ctx returned by StartSpan above contains
	// the current span now, so subsequent calls to StartSpan create new
	// child spans.

	// db was opened with apmsql, so queries will be reported as
	// spans when using the context methods.
	rows, err := db.QueryContext(ctx, "SELECT * FROM items")
	...
	rows.Close()
}

If you want to recover panics, and report them along with your transaction, you can use the Tracer.Recover or Tracer.Recovered methods. The former should be used as a deferred call, while the latter can be used if you have your own recovery logic. There are also methods for reporting non-panic errors: Tracer.NewError, Tracer.NewErrorLog, and elasticapm.CaptureError.

defer elasticapm.DefaultTracer.Recover(tx)

See the Error API for details and examples of the other methods.