Hints annotations based autodiscover

edit

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

Make sure you are using Elastic Agent 8.5+.

Hints autodiscovery only works with Elastic Agent Standalone.

Standalone Elastic Agent supports autodiscover based on hints from the provider. The hints mechanism looks for hints in Kubernetes Pod annotations that have the prefix co.elastic.hints. As soon as the container starts, Elastic Agent checks it for hints and launches the proper configuration for the container. Hints tell Elastic Agent how to monitor the container by using the proper integration. This is the full list of supported hints:

Required hints:
edit
co.elastic.hints/package
edit

The package to use for monitoring.

co.elastic.hints/host
edit

The host to use for metrics retrieval.

Optional hints available:
edit
co.elastic.hints/data_stream
edit

The list of data streams to enable. If not specified, the integration’s default data streams are used. To find the defaults, refer to the Elastic integrations documentation.

If data streams are specified, additional hints can be defined per data stream. For example, co.elastic.hints/info.period: 5m if the data stream specified is info for the Redis module.

apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info
    co.elastic.hints/info.period: 5m

If data stream hints are not specified, the top level hints will be used in its configuration.

co.elastic.hints/metrics_path
edit

The path to retrieve the metrics from.

co.elastic.hints/period
edit

The time interval for metrics retrieval, for example, 10s.

co.elastic.hints/timeout
edit

Metrics retrieval timeout, for example, 3s.

co.elastic.hints/username
edit

The username to use for authentication.

co.elastic.hints/password
edit

The password to use for authentication. It is recommended to retrieve this sensitive information from an ENV variable and avoid placing passwords in plain text.

co.elastic.hints/stream
edit

The stream to use for logs collection, for example, stdout/stderr.

If the specified package has no logs support, a generic container’s logs input will be used as a fallback.

Available packages that support hints autodiscovery
edit

The available packages that are supported through hints can be found here.

Configure hints autodiscovery
edit

To enable hints, you must add hints.enabled: true to the provider’s configuration:

providers:
  kubernetes:
    hints.enabled: true

Then ensure that the proper volumes and volumeMounts are specified by uncommenting the appropriate sections in the Elastic Agent manifest:

volumeMounts:
- name: external-inputs
  mountPath: /etc/elastic-agent/inputs.d
...
volumes:
- name: external-inputs
  emptyDir: {}
...

An init container is also required to download the hints templates. The init container is already defined, so uncomment the respective section:

initContainers:
- name: k8s-templates-downloader
  image: busybox:1.28
  command: ['sh']
  args:
    - -c
    - >-
      mkdir -p /etc/elastic-agent/inputs.d &&
      wget -O - https://github.com/elastic/elastic-agent/archive/8.6.tar.gz | tar xz -C /etc/elastic-agent/inputs.d --strip=5 "elastic-agent-8.6/deploy/kubernetes/elastic-agent-standalone/templates.d"
  volumeMounts:
    - name: external-inputs
      mountPath: /etc/elastic-agent/inputs.d
Example: Hints autodiscovery
edit

Enabling hints allows users deploying Pods on the cluster to automatically turn on Elastic monitoring at Pod deployment time. For example, to deploy a Redis Pod on the cluster and automatically enable Elastic monitoring, add the proper hints as annotations on the Pod manifest file:

...
apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations:
    co.elastic.hints/package: redis
    co.elastic.hints/data_streams: info
    co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
    co.elastic.hints/info.period: 5s
  labels:
    k8s-app: redis
    app: redis
...

After deploying this Pod, the data will start flowing in automatically. You can find it on the index metrics-redis.info-default.

All assets (dashboards, ingest pipelines, and so on) related to the Redis integration are not installed. You need to explicitly install them through Kibana.

Troubleshooting
edit

When things do not work as expected, you may need to troubleshoot your setup. Here we provide some directions to speed up your investigation:

  1. Exec inside an Agent’s Pod and run the inspect command to verify how inputs are constructed dynamically:

    ./elastic-agent inspect --variables --variables-wait 1s -c /etc/elastic-agent/agent.yml

    Specifically, examine how the inputs are being populated.

  2. View the Elastic Agent logs:

    tail -f /etc/elastic-agent/data/logs/elastic-agent-*.ndjson

    Verify that the hints feature is enabled in the config and look for hints-related logs like: "Generated hints mappings are …​" In these logs, you can find the mappings that are extracted out of the annotations and determine if the values can populate a specific input.

  3. View the Metricbeat logs:

    tail -f /etc/elastic-agent/data/logs/default/metricbeat-*.ndjson
  4. View the Filebeat logs:

    tail -f /etc/elastic-agent/data/logs/default/filebeat-*.ndjson
  5. View the target input template. For the Redis example:

    cat f /etc/elastic-agent/inputs.d/redis.yml