Ingest application traces with Elastic Agent

edit

This guide describes how to:

  • Collect Application Performance Monitoring (APM) data
  • Send APM data to the Elastic Stack
  • Explore and visualize the data in real-time

For feedback and questions, please contact us in the discuss forum.

Prerequisites
edit

You need Elasticsearch for storing and searching your data, and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud (recommended), or self-manage the Elastic Stack on your own hardware.

Here’s what you need for each deployment type:

  • Elasticsearch Service deployment that includes an Integrations Server (included by default in every Elasticsearch Service deployment). Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.
  • Kibana user with All privileges on Fleet and Integrations. Since many Integrations assets are shared across spaces, users need the Kibana privileges in all spaces.
Step 1: Set up Fleet
edit

Use Fleet in Kibana to get APM data into the Elastic Stack.

The first time you use Fleet, you might need to set it up and add a Fleet Server:

Elastic Cloud runs a hosted version of Integrations Server that includes Fleet Server. No extra setup is required unless you want to scale your deployment.

To confirm that an Integrations Server is available in your deployment:

  1. In Kibana, go to Management > Fleet.
  2. On the Agents tab, look for the Elastic Cloud agent policy. This policy is managed by Elastic Cloud, and contains a Fleet Server integration and an Elastic APM integration. You cannot modify the policy. Confirm that the agent status is Healthy.

Don’t see the agent? Make sure your deployment includes an Integrations Server instance. This instance is required to use Fleet.

Hosted Integrations Server

For more information, refer to Fleet Server.

The APM integration does not support running Elastic Agent in standalone mode; you must use Fleet to manage Elastic Agent.

Step 2: Configure the APM integration
edit

Elastic Cloud runs a hosted version of Integrations Server that includes the APM integration. Self-managed users will need to add the APM integration before configuring it.

  1. In Kibana, navigate to Fleet > Agent policies and select the Elastic Cloud agent policy. This is the default agent policy for Elastic Agents hosted on Elastic Cloud.

    Elastic Cloud agent policy
  2. Next to Elastic APM in the Actions column, select Edit integration to configure the APM integration.

    Edit the APM integration
  3. Make a note of the predefined URL that the APM Server is listening on—​you’ll need this in the next step.
  4. If you made any changes to your configuration, click save and continue.
Step 3: Install and run an Elastic Agent on your machine
edit

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, forward data from remote services or hardware, and more. A single agent makes it easier and faster to deploy monitoring across your infrastructure. Each agent has a single policy you can update to add integrations for new data sources, security protections, and more.

Don’t confuse Elastic Agent with APM agents—​they are different components. In a later step, you’ll instrument your code with APM agents and send the data to an APM Server instance that Elastic Agent spins up.

If you plan on enabling Real User Monitoring (RUM), you must run Elastic Agent centrally. If RUM is disabled, you should run Elastic Agent on edge machines.

To send APM data to the Elastic Stack:

  1. In Kibana, go to Fleet > Agents, and click Add agent.
  2. In the Add agent flyout, select an existing agent policy or create a new one. If you create a new policy, Fleet generates a new Fleet enrollment token.

    For on-premises deployments, you can dedicate a policy to all the agents in the network boundary and configure that policy to include a specific Fleet Server (or a cluster of Fleet Servers).

    Read more in Add a Fleet Server to a policy.

  3. Make sure Enroll in Fleet is selected.
  4. Download, install, and enroll the Elastic Agent on your host by selecting your host operating system and following the Install Elastic Agent on your host step.

    1. If you are enrolling the agent in a Fleet Server that uses your organization’s certificate you must add the --certificate-authorities option to the command provided in the in-product instructions. If you do not include the certificate, you will see the following error: "x509: certificate signed by unknown authority".

      Add agent flyout in Kibana

After about a minute, the agent will enroll in Fleet, download the configuration specified in the agent policy, and start collecting data.

Notes:

  • If you encounter an "x509: certificate signed by unknown authority" error, you might be trying to enroll in a Fleet Server that uses self-signed certs. To fix this problem in a non-production environment, pass the --insecure flag. For more information, refer to the troubleshooting guide.
  • Optionally, you can use the --tag flag to specify a comma-separated list of tags to apply to the enrolled Elastic Agent. For more information, refer to Filter list of Agents by tags.
  • Refer to Installation layout for the location of installed Elastic Agent files.
  • Because Elastic Agent is installed as an auto-starting service, it will restart automatically if the system is rebooted.

To confirm that Elastic Agent is installed and running, go to the Agents tab in Fleet.

Fleet showing enrolled agents

If the status hangs at Enrolling, make sure the elastic-agent process is running.

If you run into problems:

For information about managing Elastic Agent in Fleet, refer to Centrally manage Elastic Agents in Fleet.

Step 4: Install APM agents
edit

APM agents are written in the same language as your service. To monitor a new service, you must install the agent and configure it with a service name, APM Server host, and Secret token.

  • Service name: The APM integration maps an instrumented service’s name–defined in each APM agent’s configuration– to the index that its data is stored in Elasticsearch. Service names are case-insensitive and must be unique. For example, you cannot have a service named Foo and another named foo. Special characters will be removed from service names and replaced with underscores (_).
  • APM Server URL: The host and port that APM Server listens for events on. This should match the host and port defined when setting up the APM integration.
  • Secret token: Authentication method for APM agent and APM Server communication. This should match the secret token defined when setting up the APM integration.

You can edit your APM integration settings if you need to change the APM Server URL or secret token to match your APM agents.

Download the APM agent

Download the agent jar from Maven Central. Do not add the agent as a dependency to your application.

Start your application with the javaagent flag

Add the -javaagent flag and configure the agent with system properties.

  • Set required service name
  • Set custom APM Server URL (default: http://localhost:8200)
  • Set the base package of your application
java -javaagent:/path/to/elastic-apm-agent-<version>.jar \
     -Delastic.apm.service_name=my-application \
     -Delastic.apm.server_urls=http://localhost:8200 \
     -Delastic.apm.secret_token= \
     -Delastic.apm.application_packages=org.example \
     -jar my-application.jar

Learn more in the agent reference

Step 5: View your data
edit

Back in Kibana, under Observability, select APM. You should see application performance monitoring data flowing into the Elastic Stack!

The built-in apm_user role is not compatible with the APM integration as it only provides read access to apm-* indices. For a list of indices users need access to, refer to APM data streams

APM app with data

Not seeing any data? Review our list of common problems for helpful tips.

What’s next?
edit
  • Now that data is streaming into the Elastic Stack, take your investigation to a deeper level! Use Elastic Observability to unify your logs, metrics, uptime, and application performance data.
  • Want to protect your endpoints from security threats? Try Elastic Security. Adding endpoint protection is just another integration that you add to the agent policy!
  • Are your eyes bleary from staring at a wall of screens? Create alerts and find out about problems while sipping your favorite beverage poolside.
  • Want Elastic to do the heavy lifting? Use machine learning to detect anomalies.
  • Got everything working like you want it? Roll out your agent policies to other hosts by deploying Elastic Agents across your infrastructure!