- Metricbeat Reference: other versions:
- Metricbeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade Metricbeat
- How Metricbeat works
- Configure
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- metricbeat.reference.yml
- How to guides
- Modules
- ActiveMQ module
- Aerospike module
- Airflow module
- Apache module
- AWS module
- AWS billing metricset
- AWS cloudwatch metricset
- AWS dynamodb metricset
- AWS ebs metricset
- AWS ec2 metricset
- AWS elb metricset
- AWS kinesis metricset
- AWS lambda metricset
- AWS natgateway metricset
- AWS rds metricset
- AWS s3_daily_storage metricset
- AWS s3_request metricset
- AWS sns metricset
- AWS sqs metricset
- AWS transitgateway metricset
- AWS usage metricset
- AWS vpn metricset
- AWS Fargate module
- Azure module
- Azure app_insights metricset
- Azure app_state metricset
- Azure billing metricset
- Azure compute_vm metricset
- Azure compute_vm_scaleset metricset
- Azure container_instance metricset
- Azure container_registry metricset
- Azure container_service metricset
- Azure database_account metricset
- Azure monitor metricset
- Azure storage metricset
- Beat module
- Ceph module
- Ceph cluster_disk metricset
- Ceph cluster_health metricset
- Ceph cluster_status metricset
- Ceph mgr_cluster_disk metricset
- Ceph mgr_cluster_health metricset
- Ceph mgr_osd_perf metricset
- Ceph mgr_osd_pool_stats metricset
- Ceph mgr_osd_tree metricset
- Ceph mgr_pool_disk metricset
- Ceph monitor_health metricset
- Ceph osd_df metricset
- Ceph osd_tree metricset
- Ceph pool_disk metricset
- Cloudfoundry module
- CockroachDB module
- Consul module
- Containerd module
- Coredns module
- Couchbase module
- CouchDB module
- Docker module
- Dropwizard module
- Elasticsearch module
- Elasticsearch ccr metricset
- Elasticsearch cluster_stats metricset
- Elasticsearch enrich metricset
- Elasticsearch index metricset
- Elasticsearch index_recovery metricset
- Elasticsearch index_summary metricset
- Elasticsearch ml_job metricset
- Elasticsearch node metricset
- Elasticsearch node_stats metricset
- Elasticsearch pending_tasks metricset
- Elasticsearch shard metricset
- Enterprise Search module
- Envoyproxy module
- Etcd module
- Google Cloud Platform module
- Google Cloud Platform billing metricset
- Google Cloud Platform compute metricset
- Google Cloud Platform dataproc metricset
- Google Cloud Platform firestore metricset
- Google Cloud Platform gke metricset
- Google Cloud Platform loadbalancing metricset
- Google Cloud Platform metrics metricset
- Google Cloud Platform pubsub metricset
- Google Cloud Platform storage metricset
- Golang module
- Graphite module
- HAProxy module
- HTTP module
- IBM MQ module
- IIS module
- Istio module
- Jolokia module
- Kafka module
- Kibana module
- Kubernetes module
- Kubernetes apiserver metricset
- Kubernetes container metricset
- Kubernetes controllermanager metricset
- Kubernetes event metricset
- Kubernetes node metricset
- Kubernetes pod metricset
- Kubernetes proxy metricset
- Kubernetes scheduler metricset
- Kubernetes state_container metricset
- Kubernetes state_cronjob metricset
- Kubernetes state_daemonset metricset
- Kubernetes state_deployment metricset
- Kubernetes state_job metricset
- Kubernetes state_node metricset
- Kubernetes state_persistentvolumeclaim metricset
- Kubernetes state_pod metricset
- Kubernetes state_replicaset metricset
- Kubernetes state_resourcequota metricset
- Kubernetes state_service metricset
- Kubernetes state_statefulset metricset
- Kubernetes state_storageclass metricset
- Kubernetes system metricset
- Kubernetes volume metricset
- KVM module
- Linux module
- Logstash module
- Memcached module
- MongoDB module
- MSSQL module
- Munin module
- MySQL module
- NATS module
- Nginx module
- Openmetrics module
- Oracle module
- PHP_FPM module
- PostgreSQL module
- Prometheus module
- RabbitMQ module
- Redis module
- Redis Enterprise module
- SQL module
- Stan module
- Statsd module
- SyncGateway module
- System module
- System core metricset
- System cpu metricset
- System diskio metricset
- System entropy metricset
- System filesystem metricset
- System fsstat metricset
- System load metricset
- System memory metricset
- System network metricset
- System network_summary metricset
- System process metricset
- System process_summary metricset
- System raid metricset
- System service metricset
- System socket metricset
- System socket_summary metricset
- System uptime metricset
- System users metricset
- Tomcat module
- Traefik module
- uWSGI module
- vSphere module
- Windows module
- ZooKeeper module
- Exported fields
- ActiveMQ fields
- Aerospike fields
- Airflow fields
- Apache fields
- AWS fields
- AWS Fargate fields
- Azure fields
- Beat fields
- Beat fields
- Ceph fields
- Cloud provider metadata fields
- Cloudfoundry fields
- CockroachDB fields
- Common fields
- Consul fields
- Containerd fields
- Coredns fields
- Couchbase fields
- CouchDB fields
- Docker fields
- Docker fields
- Dropwizard fields
- ECS fields
- Elasticsearch fields
- Enterprise Search fields
- Envoyproxy fields
- Etcd fields
- Google Cloud Platform fields
- Golang fields
- Graphite fields
- HAProxy fields
- Host fields
- HTTP fields
- IBM MQ fields
- IIS fields
- Istio fields
- Jolokia fields
- Jolokia Discovery autodiscover provider fields
- Kafka fields
- Kibana fields
- Kubernetes fields
- Kubernetes fields
- KVM fields
- Linux fields
- Logstash fields
- Memcached fields
- MongoDB fields
- MSSQL fields
- Munin fields
- MySQL fields
- NATS fields
- Nginx fields
- Openmetrics fields
- Oracle fields
- PHP_FPM fields
- PostgreSQL fields
- Process fields
- Prometheus fields
- Prometheus typed metrics fields
- RabbitMQ fields
- Redis fields
- Redis Enterprise fields
- SQL fields
- Stan fields
- Statsd fields
- SyncGateway fields
- System fields
- Tomcat fields
- Traefik fields
- uWSGI fields
- vSphere fields
- Windows fields
- ZooKeeper fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- "open /compat/linux/proc: no such file or directory" error on FreeBSD
- Metricbeat collects system metrics for interfaces you didn’t configure
- Metricbeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Autodiscover
editAutodiscover
editWhen you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.
You define autodiscover settings in the metricbeat.autodiscover
section of the metricbeat.yml
config file. To enable autodiscover, you specify a list of providers.
Providers
editAutodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event to set conditions that, when met, launch specific configurations.
On start, Metricbeat will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.
Docker
editThe Docker autodiscover provider watches for Docker containers to start and stop.
It has the following settings:
-
host
-
(Optional) Docker socket (UNIX or TCP socket). It uses
unix:///var/run/docker.sock
by default. -
ssl
- (Optional) SSL configuration to use when connecting to the Docker socket.
-
cleanup_timeout
- (Optional) Specify the time of inactivity before stopping the running configuration for a container, disabled by default.
-
labels.dedot
-
(Optional) Default to be false. If set to true, replace dots in
labels with
_
.
These are the fields available within config templating. The docker.*
fields will be available on each emitted event.
event:
- host
- port
- docker.container.id
- docker.container.image
- docker.container.name
- docker.container.labels
For example:
{ "host": "10.4.15.9", "port": 6379, "docker": { "container": { "id": "382184ecdb385cfd5d1f1a65f78911054c8511ae009635300ac28b4fc357ce51" "name": "redis", "image": "redis:3.2.11", "labels": { "io.kubernetes.pod.namespace": "default" ... } } } }
You can define a set of configuration templates to be applied when the condition matches an event. Templates define a condition to match on autodiscover events, together with the list of configurations to launch when this condition happens.
Conditions match events from the provider. Providers use the same format for Conditions that processors use.
Configuration templates can contain variables from the autodiscover event. They can be accessed under the data
namespace.
For example, with the example event, "${data.port}
" resolves to 6379
.
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: docker labels.dedot: true templates: - condition: contains: docker.container.image: redis config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379"
This configuration launches a redis
module for all containers running an image with redis
in the name.
labels.dedot
defaults to be true
for docker autodiscover, which means dots in docker labels are replaced with _ by default.
Also Metricbeat autodiscover supports leveraging Secrets keystore in order to retrieve sensitive data like passwords. Here is an example of how a configuration using keystore would look like:
metricbeat.autodiscover: providers: - type: docker labels.dedot: true templates: - condition: contains: docker.container.image: redis config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379" password: "${REDIS_PASSWORD}"
where REDIS_PASSWORD
is a key stored in local keystore of Metricbeat.
Kubernetes
editThe Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services to start, update, and stop.
The kubernetes
autodiscover provider has the following configuration settings:
-
node
- (Optional) Specify the node to scope metricbeat to in case it cannot be accurately detected, as when running metricbeat in host network mode.
-
namespace
- (Optional) Select the namespace from which to collect the metadata. If it is not set, the processor collects metadata from all namespaces. It is unset by default. The namespace configuration only applies to kubernetes resources that are namespace scoped.
-
cleanup_timeout
- (Optional) Specify the time of inactivity before stopping the running configuration for a container, disabled by default.
-
kube_config
- (Optional) Use given config file as configuration for Kubernetes client. If kube_config is not set, KUBECONFIG environment variable will be checked and if not present it will fall back to InCluster.
-
kube_client_options
- (Optional) Additional options can be configured for Kubernetes client. Currently client QPS and burst are supported, if not set Kubernetes client’s default QPS and burst will be used. Example:
kube_client_options: qps: 5 burst: 10
-
resource
-
(Optional) Select the resource to do discovery on. Currently supported
Kubernetes resources are
pod
,service
andnode
. If not configuredresource
defaults topod
. -
scope
-
(Optional) Specify at what level autodiscover needs to be done at.
scope
can either takenode
orcluster
as values.node
scope allows discovery of resources in the specified node.cluster
scope allows cluster wide discovery. Onlypod
andnode
resources can be discovered at node scope. -
add_resource_metadata
-
(Optional) Specify filters and configration for the extra metadata, that will be added to the event. Configuration parameters:
-
node
ornamespace
: Specify labels and annotations filters for the extra metadata coming from node and namespace. By default all labels are included while annotations are not. To change default behaviourinclude_labels
,exclude_labels
andinclude_annotations
can be defined. Those settings are useful when storing labels and annotations that require special handling to avoid overloading the storage output. Note: wildcards are not supported for those settings. The enrichment ofnode
ornamespace
metadata can be individually disabled by settingenabled: false
. -
deployment
: If resource ispod
and it is created from adeployment
, by default the deployment name is added, this can be disabled by settingdeployment: false
. -
cronjob
: If resource ispod
and it is created from acronjob
, by default the cronjob name is added, this can be disabled by settingcronjob: false
.Example:
-
add_resource_metadata: namespace: include_labels: ["namespacelabel1"] node: include_labels: ["nodelabel2"] include_annotations: ["nodeannotation1"] deployment: false cronjob: false
-
unique
-
(Optional) Defaults to
false
. Marking an autodiscover provider as unique results into making the provider to enable the provided templates only when it will gain the leader lease. This setting can only be combined withcluster
scope. Whenunique
is enabled enabled,resource
andadd_resource_metadata
settings are not taken into account. -
leader_lease
-
(Optional) Defaults to
metricbeat-cluster-leader
. This will be name of the lock lease. One can monitor the status of the lease withkubectl describe lease beats-cluster-leader
. Different Beats that refer to the same leader lease will be competitors in holding the lease and only one will be elected as leader each time.
The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can contain variables from the autodiscover event. They can be accessed under data namespace.
These are the fields available within config templating. The kubernetes.*
fields will be available on each emitted event.
Generic fields:
edit- host
- port (if exposed)
- kubernetes.labels
- kubernetes.annotations
Pod specific:
edit- kubernetes.container.id
- kubernetes.container.image
- kubernetes.container.name
- kubernetes.namespace
- kubernetes.node.name
- kubernetes.pod.name
- kubernetes.pod.uid
Node specific:
edit- kubernetes.node.name
- kubernetes.node.uid
Service specific:
edit- kubernetes.namespace
- kubernetes.service.name
- kubernetes.service.uid
- kubernetes.annotations
If the include_annotations
config is added to the provider config, then the list of annotations present in the config
are added to the event.
If the include_labels
config is added to the provider config, then the list of labels present in the config
will be added to the event.
If the exclude_labels
config is added to the provider config, then the list of labels present in the config
will be excluded from the event.
if the labels.dedot
config is set to be true
in the provider config, then .
in labels will be replaced with _
.
By default it is true
.
if the annotations.dedot
config is set to be true
in the provider config, then .
in annotations will be replaced
with _
. By default it is true
.
For example:
{ "host": "172.17.0.21", "port": 9090, "kubernetes": { "container": { "id": "bb3a50625c01b16a88aa224779c39262a9ad14264c3034669a50cd9a90af1527", "image": "prom/prometheus", "name": "prometheus" }, "labels": { "project": "prometheus", ... }, "namespace": "default", "node": { "name": "minikube" }, "pod": { "name": "prometheus-2657348378-k1pnh" } }, }
Example:
metricbeat.autodiscover: providers: - type: kubernetes scope: cluster node: ${NODE_NAME} unique: true identifier: leader-election-metricbeat templates: - config: - module: kubernetes hosts: ["kube-state-metrics:8080"] period: 10s add_metadata: true metricsets: - state_node
The above configuration when deployed on one or more Metribceat instances will enable state_node
metricset only for the Metricbeat instance that will gain the leader lease/lock. With this deployment
strategy we can ensure that cluster-wide metricsets are only enabled by one Beat instance when
deploying a Beat as DaemonSet.
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: kubernetes include_annotations: ["prometheus.io.scrape"] templates: - condition: contains: kubernetes.annotations.prometheus.io/scrape: "true" config: - module: prometheus metricsets: ["collector"] hosts: "${data.host}:${data.port}"
This configuration launches a prometheus
module for all containers of pods annotated prometheus.io/scrape=true
.
Manually Defining Ports with Kubernetes
editDeclare exposed ports in your pod spec if possible. Otherwise, you will need to use
multiple templates with complex filtering rules. The {port} variable will not be
present, and you will need to hardcode ports. Example: {data.host}:1234
When ports are not declared, Autodiscover generates a config using your provided template once per pod, and once per container. These generated configs are de-duplicated after they are generated. If the generated configs for multiple containers are identical, they will be merged into one config.
Pods share an identical host. If only the {data.host}
variable is interpolated,
then one config will be generated per host. The configs will be identical.
After they are de-duplicated, only one will be used.
In order to target one specific exposed port {data.host}:{data.ports.web}
can be used
in template config, where web
is the name of the exposed container port.
Metricbeat Autodiscover Secret Management
editLocal Keystore
editMetricbeat autodiscover supports leveraging Secrets keystore in order to retrieve sensitive data like passwords. Here is an example of how a configuration using keystore would look like:
metricbeat.autodiscover: providers: - type: kubernetes templates: - condition: contains: kubernetes.labels.app: "redis" config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379" password: "${REDIS_PASSWORD}"
where REDIS_PASSWORD
is a key stored in local keystore of Metricbeat.
Kubernetes Secrets
editMetricbeat autodiscover supports leveraging Kubernetes secrets
in order to retrieve sensitive data like passwords. In order to enable this feature add the following section
in Metricbeat’s ClusterRole
rules:
- apiGroups: [""] resources: - secrets verbs: ["get"]
The above rule will give permission to Metricbeat Pod to access Kubernetes Secrets API.
This means that anyone who have access to Metricbeat Pod (kubectl exec
for example) will be able to access
Kubernetes Secrets API and get a specific secret no matter which namespace it belongs to.
This option should be carefully considered, specially when used with hints.
One option to give permissions only for one namespace, and not cluster-scoped, is to use a specific Role for a targeted namespace so as to better control access:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: marketing-team name: secret-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["secrets"] verbs: ["get"]
One can find more info about Role and ClusterRole in the official Kubernetes documentation.
Here is an example of how a configuration using Kubernetes secrets would look like:
metricbeat.autodiscover: providers: - type: kubernetes templates: - condition: contains: kubernetes.labels.app: "redis" config: - module: redis metricsets: ["info", "keyspace"] hosts: "${data.host}:6379" password: "${kubernetes.default.somesecret.value}"
where kubernetes.default.somesecret.value
specifies a key stored as Kubernetes secret as following:
-
Kubernetes Namespace:
default
-
Kubernetes Secret Name:
somesecret
-
Secret Data Key:
value
This secret can be created in a Kubernetes environment using the following command:
cat << EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: somesecret type: Opaque data: value: $(echo -n "passpass" | base64) EOF
Note that Pods can only consume secrets that belong to the same Kubernetes namespace. For instance if Pod my-redis
is running under staging
namespace, it cannot access a secret under testing
namespace for example kubernetes.testing.xxx.yyy
.
Jolokia
editThe Jolokia autodiscover provider uses Jolokia Discovery to find agents running in your host or your network.
The configuration of this provider consists in a set of network interfaces, as
well as a set of templates as in other providers. The network interfaces will be
the ones used for discovery probes, each item of interfaces
has these settings:
-
name
-
the name of the interface (e.g.
br0
), it can contain a wildcard as suffix to apply the same settings to multiple network interfaces of the same type (e.g.br*
). -
interval
- time between probes (defaults to 10s)
-
grace_period
- time since the last reply to consider an instance stopped (defaults to 30s)
-
probe_timeout
- max time to wait for responses since a probe is sent (defaults to 1s)
Jolokia Discovery mechanism is supported by any Jolokia agent since version 1.2.0, it is enabled by default when Jolokia is included in the application as a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents. In any case, this feature is controlled with two properties:
-
discoveryEnabled
, to enable the feature -
discoveryAgentUrl
, if set, this is the URL announced by the agent when being discovered, setting this parameter implicitly enables the feature
There are multiple ways of setting these properties, and they can vary from application to application, please refer to the documentation of your application to find the more suitable way to set them in your case.
Jolokia Discovery is based on UDP multicast requests. Agents join the multicast group 239.192.48.84, port 24884, and discovery is done by sending queries to this group. You have to take into account that UDP traffic between Metricbeat and the Jolokia agents has to be allowed. Also notice that this multicast address is in the 239.0.0.0/8 range, that is reserved for private use within an organization, so it can only be used in private networks.
These are the available fields during within config templating. The jolokia.*
fields will be available on each emitted event.
- jolokia.agent.id
- jolokia.agent.version
- jolokia.secured
- jolokia.server.product
- jolokia.server.vendor
- jolokia.server.version
- jolokia.url
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: jolokia interfaces: - name: br* interval: 5s grace_period: 10s - name: en* templates: - condition: contains: jolokia.server.product: "tomcat" config: - module: jolokia metricsets: ["jmx"] hosts: "${data.jolokia.url}" namespace: test jmx.mappings: - mbean: "java.lang:type=Runtime" attributes: - attr: Uptime field: uptime
This configuration starts a jolokia module that collects the uptime of each
tomcat
instance discovered. Discovery probes are sent using all interfaces
starting with br
and en
, for the br
interfaces the interval
and
grace_period
is reduced to 5 and 10 seconds respectively.
Amazon EC2s
editThis functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
The Amazon EC2 autodiscover provider discovers EC2 instances. This is useful for users to launch Metricbeat modules to monitor services running on AWS EC2 instances.
For example, you can use this provider to gather MySQL metrics from MySQL
servers running on EC2 instances that have a specific tag, service: mysql
.
This provider will load AWS credentials using the standard AWS environment variables and shared credentials files
see Best Practices for Managing AWS Access Keys
for more information. If you do not wish to use these, you may explicitly set the access_key_id
and
secret_access_key
variables.
These are the available fields during within config templating.
The aws.ec2.*
fields and cloud.*
fields will be available on each emitted event.
- cloud.availability_zone
- cloud.instance.id
- cloud.machine.type
- cloud.provider
- cloud.region
- aws.ec2.architecture
- aws.ec2.image.id
- aws.ec2.kernel.id
- aws.ec2.monitoring.state
- aws.ec2.private.dns_name
- aws.ec2.private.ip
- aws.ec2.public.dns_name
- aws.ec2.public.ip
- aws.ec2.root_device_name
- aws.ec2.state.code
- aws.ec2.state.name
- aws.ec2.subnet.id
- aws.ec2.tags
- aws.ec2.vpc.id
Metricbeat supports templates for modules:
metricbeat.autodiscover: providers: - type: aws_ec2 period: 1m credential_profile_name: elastic-beats templates: - condition: equals: aws.ec2.tags.service: "mysql" config: - module: mysql metricsets: ["status", "galera_status"] period: 10s hosts: ["root:password@tcp(${data.aws.ec2.public.ip}:3306)/"] username: root password: password
This autodiscover provider takes our standard AWS credentials options.
With this configuration, mysql
metricbeat module will be launched for all EC2
instances that have service: mysql
as a tag.
This autodiscover provider takes our standard AWS credentials options.
On this page