- Logstash Reference: other versions:
- Logstash Introduction
- Getting Started with Logstash
- How Logstash Works
- Setting Up and Running Logstash
- Logstash Directory Layout
- Logstash Configuration Files
- logstash.yml
- Secrets keystore for secure settings
- Running Logstash from the Command Line
- Running Logstash as a Service on Debian or RPM
- Running Logstash on Docker
- Configuring Logstash for Docker
- Running Logstash on Windows
- Logging
- Shutting Down Logstash
- Upgrading Logstash
- Creating a Logstash pipeline
- Secure your connection
- Advanced Logstash Configurations
- Logstash-to-Logstash communication
- Managing Logstash
- Using Logstash with Elastic Integrations (Beta)
- Working with Logstash Modules
- Working with Filebeat Modules
- Working with Winlogbeat Modules
- Queues and data resiliency
- Transforming Data
- Deploying and Scaling Logstash
- Performance Tuning
- Monitoring Logstash with Elastic Agent
- Monitoring Logstash (legacy)
- Monitoring Logstash with APIs
- Working with plugins
- Integration plugins
- Input plugins
- azure_event_hubs
- beats
- cloudwatch
- couchdb_changes
- dead_letter_queue
- elastic_agent
- elastic_serverless_forwarder
- elasticsearch
- exec
- file
- ganglia
- gelf
- generator
- github
- google_cloud_storage
- google_pubsub
- graphite
- heartbeat
- http
- http_poller
- imap
- irc
- java_generator
- java_stdin
- jdbc
- jms
- jmx
- kafka
- kinesis
- logstash
- log4j
- lumberjack
- meetup
- pipe
- puppet_facter
- rabbitmq
- redis
- relp
- rss
- s3
- s3-sns-sqs
- salesforce
- snmp
- snmptrap
- sqlite
- sqs
- stdin
- stomp
- syslog
- tcp
- udp
- unix
- varnishlog
- websocket
- wmi
- xmpp
- Output plugins
- boundary
- circonus
- cloudwatch
- csv
- datadog
- datadog_metrics
- dynatrace
- elastic_app_search
- elastic_workplace_search
- elasticsearch
- exec
- file
- ganglia
- gelf
- google_bigquery
- google_cloud_storage
- google_pubsub
- graphite
- graphtastic
- http
- influxdb
- irc
- java_stdout
- juggernaut
- kafka
- librato
- logstash
- loggly
- lumberjack
- metriccatcher
- mongodb
- nagios
- nagios_nsca
- opentsdb
- pagerduty
- pipe
- rabbitmq
- redis
- redmine
- riak
- riemann
- s3
- sink
- sns
- solr_http
- sqs
- statsd
- stdout
- stomp
- syslog
- tcp
- timber
- udp
- webhdfs
- websocket
- xmpp
- zabbix
- Filter plugins
- age
- aggregate
- alter
- bytes
- cidr
- cipher
- clone
- csv
- date
- de_dot
- dissect
- dns
- drop
- elapsed
- elastic_integration
- elasticsearch
- environment
- extractnumbers
- fingerprint
- geoip
- grok
- http
- i18n
- java_uuid
- jdbc_static
- jdbc_streaming
- json
- json_encode
- kv
- memcached
- metricize
- metrics
- mutate
- prune
- range
- ruby
- sleep
- split
- syslog_pri
- threats_classifier
- throttle
- tld
- translate
- truncate
- urldecode
- useragent
- uuid
- wurfl_device_detection
- xml
- Codec plugins
- Tips and best practices
- Troubleshooting
- Contributing to Logstash
- How to write a Logstash input plugin
- How to write a Logstash codec plugin
- How to write a Logstash filter plugin
- How to write a Logstash output plugin
- Logstash Plugins Community Maintainer Guide
- Document your plugin
- Publish your plugin to RubyGems.org
- List your plugin
- Contributing a patch to a Logstash plugin
- Extending Logstash core
- Contributing a Java Plugin
- Breaking changes
- Release Notes
- Logstash 8.11.4 Release Notes
- Logstash 8.11.3 Release Notes
- Logstash 8.11.2 Release Notes
- Logstash 8.11.1 Release Notes
- Logstash 8.11.0 Release Notes
- Logstash 8.10.4 Release Notes
- Logstash 8.10.3 Release Notes
- Logstash 8.10.2 Release Notes
- Logstash 8.10.1 Release Notes
- Logstash 8.10.0 Release Notes
- Logstash 8.9.2 Release Notes
- Logstash 8.9.1 Release Notes
- Logstash 8.9.0 Release Notes
- Logstash 8.8.2 Release Notes
- Logstash 8.8.1 Release Notes
- Logstash 8.8.0 Release Notes
- Logstash 8.7.1 Release Notes
- Logstash 8.7.0 Release Notes
- Logstash 8.6.2 Release Notes
- Logstash 8.6.1 Release Notes
- Logstash 8.6.0 Release Notes
- Logstash 8.5.3 Release Notes
- Logstash 8.5.2 Release Notes
- Logstash 8.5.1 Release Notes
- Logstash 8.5.0 Release Notes
- Logstash 8.4.2 Release Notes
- Logstash 8.4.1 Release Notes
- Logstash 8.4.0 Release Notes
- Logstash 8.3.3 Release Notes
- Logstash 8.3.2 Release Notes
- Logstash 8.3.1 Release Notes
- Logstash 8.3.0 Release Notes
- Logstash 8.2.3 Release Notes
- Logstash 8.2.2 Release Notes
- Logstash 8.2.1 Release Notes
- Logstash 8.2.0 Release Notes
- Logstash 8.1.3 Release Notes
- Logstash 8.1.2 Release Notes
- Logstash 8.1.1 Release Notes
- Logstash 8.1.0 Release Notes
- Logstash 8.0.1 Release Notes
- Logstash 8.0.0 Release Notes
- Logstash 8.0.0-rc2 Release Notes
- Logstash 8.0.0-rc1 Release Notes
- Logstash 8.0.0-beta1 Release Notes
- Logstash 8.0.0-alpha2 Release Notes
- Logstash 8.0.0-alpha1 Release Notes
Troubleshooting plugins
editTroubleshooting plugins
editPlugin tracing
editWhen you troubleshoot Logstash deployments, you might check the node stats API to look for plugins that are taking too long to process data, dropping data, or never received the data in the first place. After you identify a plugin whose metrics indicate a potential issue, it’s time to identify which plugin this is, and where it is declared in the configuration files.
While you can define an "id" for each plugin to facilitate this discovery, naming each plugin is not practical, especially for very large pipelines containing hundreds of plugins.
You can use the information from the Logstash APIs to fetch link an auto generated ID of a plugin to its declaration (starting with 7.6.0).
Here’s how:
Browse stats API and find a plugin you want to investigate
editExample: "Give me any filter whose in/out events counters don’t match"
❯ curl -s localhost:9600/_node/stats | jq '.pipelines.main.plugins.filters[] | select(.events.in!=.events.out)'
{ "id": "75afda0f03a5af46279c4cba9408ca87664b9c988bf477e2a2cca535e59e856f", "events": { "in": 1, "out": 0, "duration_in_millis": 5 }, "name": "drop" }
Find the plugin declaration in the pipeline graph
editTake the id from step 1 and use it to find the plugin in the pipeline graph:
❯ curl -s localhost:9600/_node/pipelines?graph=true | jq '.pipelines.main.graph.graph.vertices[] | select(.id=="75afda0f03a5af46279c4cba9408ca87664b9c988bf477e2a2cca535e59e856f")'
{ "config_name": "drop", "plugin_type": "filter", "meta": { "source": { "protocol": "file", "id": "/private/tmp/logstash-7.9.1/cfg", "line": 10, "column": 5 } }, "id": "75afda0f03a5af46279c4cba9408ca87664b9c988bf477e2a2cca535e59e856f", "explicit_id": false, "type": "plugin" }
Lookup the plugin’s definition in the source files
editHere’s a simple script to do the lookup.
❯ cat /private/tmp/logstash-7.9.1/cfg | ruby -e 'line = 10; $stdin.read.split("\n").each_with_index {|l, i| puts "#{i+1}: #{l}" if (i+1).between?(line-1, line + 5) }' 9: } else if [message] == "d" { 10: drop {} 11: } else if [message] == "e" { 12: drop {} 13: } else if [message] == "f" { 14: drop {} 15: } else if [message] == "g" {
Or, you can open the file and go to the line.
On this page