Autodiscover

When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

You define autodiscover settings in the metricbeat.autodiscover section of the metricbeat.yml config file. To enable autodiscover, you specify a list of providers.

Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event to set conditions that, when met, launch specific configurations.

On start, Metricbeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.

The Docker autodiscover provider watches for Docker containers to start and stop.

These are the fields available within config templating. The docker.* fields will be available on each emitted event. event:

For example:

{
  "host": "10.4.15.9",
  "port": 6379,
  "docker": {
    "container": {
      "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51"
      "name": "redis",
      "image": "redis:3.2.11",
      "labels": {
        "io.kubernetes.pod.namespace": "default"
        ...
      }
    }
  }
}

You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.

Conditions match events from the provider. Providers use the same format for Conditions that processors use.

Configuration templates can contain variables from the autodiscover event. They can be accessed under the data namespace. For example, with the example event, "${data.port}" resolves to 6379.

Metricbeat supports templates for modules:

metricbeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"

This configuration launches a redis module for all containers running an image with redis in the name. labels.dedot defaults to be true for docker autodiscover, which means dots in docker labels are replaced with _ by default.

Also Metricbeat autodiscover supports leveraging Secrets keystore in order to retrieve sensitive data like passwords. Here is an example of how a configuration using keystore would look like:

metricbeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "${REDIS_PASSWORD}"

where REDIS_PASSWORD is a key stored in local keystore of Metricbeat.

The Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services to start, update, and stop.

These are the fields available within config templating. The kubernetes.* fields will be available on each emitted event.

If the include_annotations config is added to the provider config, then the list of annotations present in the config are added to the event.

If the include_labels config is added to the provider config, then the list of labels present in the config will be added to the event.

If the exclude_labels config is added to the provider config, then the list of labels present in the config will be excluded from the event.

if the labels.dedot config is set to be true in the provider config, then . in labels will be replaced with _. By default it is true.

if the annotations.dedot config is set to be true in the provider config, then . in annotations will be replaced with _. By default it is true.

For example:

{
  "host": "172.17.0.21",
  "port": 9090,
  "kubernetes": {
    "container": {
      "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527",
      "image": "prom/prometheus",
      "name": "prometheus"
    },
    "labels": {
      "project": "prometheus",
      ...
    },
    "namespace": "default",
    "node": {
      "name": "minikube"
    },
    "pod": {
      "name": "prometheus-2657348378-k1pnh"
    }
  },
}

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.

The kubernetes autodiscover provider has the following configuration settings:

      add_resource_metadata:
        namespace:
          include_labels: ["namespacelabel1"]
        node:
          include_labels: ["nodelabel2"]
          include_annotations: ["nodeannotation1"]
        deployment: false

Example:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      identifier: leader-election-metricbeat
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node

The above configuration when deployed on one or more Metribceat instances will enable state_node metricset only for the Metricbeat instance that will gain the leader lease/lock. With this deployment strategy we can ensure that cluster-wide metricsets are only enabled by one Beat instance when deploying a Beat as DaemonSet.

Metricbeat supports templates for modules:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      include_annotations: ["prometheus.io.scrape"]
      templates:
        - condition:
            contains:
              kubernetes.annotations.prometheus.io/scrape: "true"
          config:
            - module: prometheus
              metricsets: ["collector"]
              hosts: "${data.host}:${data.port}"

This configuration launches a prometheus module for all containers of pods annotated prometheus.io/scrape=true.

Declare exposed ports in your pod spec if possible. Otherwise, you will need to use multiple templates with complex filtering rules. The {port} variable will not be present, and you will need to hardcode ports. Example: {data.host}:1234

When ports are not declared, Autodiscover generates a config using your provided template once per pod, and once per container. These generated configs are de-duplicated after they are generated. If the generated configs for multiple containers are identical, they will be merged into one config.

Pods share an identical host. If only the {data.host} variable is interpolated, then one config will be generated per host. The configs will be identical. After they are de-duplicated, only one will be used.

In order to target one specific exposed port {data.host}:{data.ports.web} can be used in template config, where web is the name of the exposed container port.

Metricbeat autodiscover supports leveraging Secrets keystore in order to retrieve sensitive data like passwords. Here is an example of how a configuration using keystore would look like:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "${REDIS_PASSWORD}"

where REDIS_PASSWORD is a key stored in local keystore of Metricbeat.

Metricbeat autodiscover supports leveraging Kubernetes secrets in order to retrieve sensitive data like passwords. In order to enable this future add the following section in Metricbeat’s ClusterRole rules:

- apiGroups: [""]
  resources:
    - secrets
  verbs: ["get"]

One option to give permissions only for one namespace, and not cluster-scoped, is to use a specific Role for a targeted namespace so as to better control access:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: marketing-team
  name: secret-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["secrets"]
  verbs: ["get"]

One can find more info about Role and ClusterRole in the official Kubernetes documentation.

Here is an example of how a configuration using Kubernetes secrets would look like:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - condition:
            contains:
              kubernetes.labels.app: "redis"
          config:
            - module: redis
              metricsets: ["info", "keyspace"]
              hosts: "${data.host}:6379"
              password: "${kubernetes.default.somesecret.value}"

where kubernetes.default.somesecret.value specifies a key stored as Kubernetes secret as following:

This secret can be created in a Kubernetes environment using the following the command:

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: somesecret
type: Opaque
data:
  value: $(echo -n "passpass" | base64)
EOF

Note that Pods can only consume secrets that belong to the same Kubernetes namespace. For instance if Pod my-redis is running under staging namespace, it cannot access a secret under testing namespace for example kubernetes.testing.xxx.yyy.

The Jolokia autodiscover provider uses Jolokia Discovery to find agents running in your host or your network.

Jolokia Discovery mechanism is supported by any Jolokia agent since version 1.2.0, it is enabled by default when Jolokia is included in the application as a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents. In any case, this feature is controlled with two properties:

There are multiple ways of setting these properties, and they can vary from application to application, please refer to the documentation of your application to find the more suitable way to set them in your case.

Jolokia Discovery is based on UDP multicast requests. Agents join the multicast group 239.192.48.84, port 24884, and discovery is done by sending queries to this group. You have to take into account that UDP traffic between Metricbeat and the Jolokia agents has to be allowed. Also notice that this multicast address is in the 239.0.0.0/8 range, that is reserved for private use within an organization, so it can only be used in private networks.

These are the available fields during within config templating. The jolokia.* fields will be available on each emitted event.

The configuration of this provider consists in a set of network interfaces, as well as a set of templates as in other providers. The network interfaces will be the ones used for discovery probes, they have these settings:

Metricbeat supports templates for modules:

metricbeat.autodiscover:
  providers:
    - type: jolokia
      interfaces:
      - name: br*
        interval: 5s
        grace_period: 10s
      - name: en*
      templates:
      - condition:
          contains:
            jolokia.server.product: "tomcat"
        config:
        - module: jolokia
          metricsets: ["jmx"]
          hosts: "${data.jolokia.url}"
          namespace: test
          jmx.mappings:
          - mbean: "java.lang:type=Runtime"
            attributes:
            - attr: Uptime
              field: uptime

This configuration starts a jolokia module that collects the uptime of each tomcat instance discovered. Discovery probes are sent using all interfaces starting with br and en, for the br interfaces the interval and grace_period is reduced to 5 and 10 seconds respectively.

The Amazon EC2 autodiscover provider discovers EC2 instances. This is useful for users to launch Metricbeat modules to monitor services running on AWS EC2 instances.

For example, you can use this provider to gather MySQL metrics from MySQL servers running on EC2 instances that have a specific tag, service: mysql.

This provider will load AWS credentials using the standard AWS environment variables and shared credentials files see Best Practices for Managing AWS Access Keys for more information. If you do not wish to use these, you may explicitly set the access_key_id and secret_access_key variables.

These are the available fields during within config templating. The aws.ec2.* fields and cloud.* fields will be available on each emitted event.

Metricbeat supports templates for modules:

metricbeat.autodiscover:
  providers:
    - type: aws_ec2
      period: 1m
      credential_profile_name: elastic-beats
      templates:
        - condition:
            equals:
              aws.ec2.tags.service: "mysql"
          config:
            - module: mysql
              metricsets: ["status", "galera_status"]
              period: 10s
              hosts: ["root:password@tcp(${data.aws.ec2.public.ip}:3306)/"]
              username: root
              password: password

This autodiscover provider takes our standard AWS credentials options. With this configuration, mysql metricbeat module will be launched for all EC2 instances that have service: mysql as a tag.

This autodiscover provider takes our standard AWS credentials options.