- Logstash Reference: other versions:
- Logstash Introduction
- Getting Started with Logstash
- How Logstash Works
- Setting Up and Running Logstash
- Logstash Directory Layout
- Logstash Configuration Files
- logstash.yml
- Secrets keystore for secure settings
- Running Logstash from the Command Line
- Running Logstash as a Service on Debian or RPM
- Running Logstash on Docker
- Configuring Logstash for Docker
- Running Logstash on Windows
- Logging
- Shutting Down Logstash
- Installing X-Pack
- Setting Up X-Pack
- Upgrading Logstash
- Configuring Logstash
- Structure of a Config File
- Accessing Event Data and Fields in the Configuration
- Using Environment Variables in the Configuration
- Logstash Configuration Examples
- Multiple Pipelines
- Pipeline-to-Pipeline Communication (Beta)
- Reloading the Config File
- Managing Multiline Events
- Glob Pattern Support
- Converting Ingest Node Pipelines
- Logstash-to-Logstash Communication
- Centralized Pipeline Management
- X-Pack monitoring
- X-Pack security
- X-Pack Settings
- Managing Logstash
- Working with Logstash Modules
- Working with Filebeat Modules
- Data Resiliency
- Transforming Data
- Deploying and Scaling Logstash
- Performance Tuning
- Monitoring Logstash
- Monitoring APIs
- Working with plugins
- Input plugins
- azure_event_hubs
- beats
- cloudwatch
- couchdb_changes
- dead_letter_queue
- elasticsearch
- exec
- file
- ganglia
- gelf
- generator
- github
- google_cloud_storage
- google_pubsub
- graphite
- heartbeat
- http
- http_poller
- imap
- irc
- jdbc
- jms
- jmx
- kafka
- kinesis
- log4j
- lumberjack
- meetup
- pipe
- puppet_facter
- rabbitmq
- redis
- relp
- rss
- s3
- salesforce
- snmp
- snmptrap
- sqlite
- sqs
- stdin
- stomp
- syslog
- tcp
- udp
- unix
- varnishlog
- websocket
- wmi
- xmpp
- Output plugins
- boundary
- circonus
- cloudwatch
- csv
- datadog
- datadog_metrics
- elastic_app_search
- elasticsearch
- exec
- file
- ganglia
- gelf
- google_bigquery
- google_pubsub
- graphite
- graphtastic
- http
- influxdb
- irc
- juggernaut
- kafka
- librato
- loggly
- lumberjack
- metriccatcher
- mongodb
- nagios
- nagios_nsca
- opentsdb
- pagerduty
- pipe
- rabbitmq
- redis
- redmine
- riak
- riemann
- s3
- sns
- solr_http
- sqs
- statsd
- stdout
- stomp
- syslog
- tcp
- timber
- udp
- webhdfs
- websocket
- xmpp
- zabbix
- Filter plugins
- aggregate
- alter
- cidr
- cipher
- clone
- csv
- date
- de_dot
- dissect
- dns
- drop
- elapsed
- elasticsearch
- environment
- extractnumbers
- fingerprint
- geoip
- grok
- http
- i18n
- jdbc_static
- jdbc_streaming
- json
- json_encode
- kv
- memcached
- metricize
- metrics
- mutate
- prune
- range
- ruby
- sleep
- split
- syslog_pri
- threats_classifier
- throttle
- tld
- translate
- truncate
- urldecode
- useragent
- uuid
- xml
- Codec plugins
- Tips and Best Practices
- Troubleshooting Common Problems
- Contributing to Logstash
- How to write a Logstash input plugin
- How to write a Logstash codec plugin
- How to write a Logstash filter plugin
- How to write a Logstash output plugin
- Documenting your plugin
- Contributing a Patch to a Logstash Plugin
- Logstash Plugins Community Maintainer Guide
- Submitting your plugin to RubyGems.org and the logstash-plugins repository
- Contributing a Java Plugin
- Glossary of Terms
- Breaking Changes
- Release Notes
- Logstash 6.8.23 Release Notes
- Logstash 6.8.22 Release Notes
- Logstash 6.8.21 Release Notes
- Logstash 6.8.20 Release Notes
- Logstash 6.8.19 Release Notes
- Logstash 6.8.18 Release Notes
- Logstash 6.8.17 Release Notes
- Logstash 6.8.16 Release Notes
- Logstash 6.8.15 Release Notes
- Logstash 6.8.14 Release Notes
- Logstash 6.8.13 Release Notes
- Logstash 6.8.12 Release Notes
- Logstash 6.8.11 Release Notes
- Logstash 6.8.10 Release Notes
- Logstash 6.8.9 Release Notes
- Logstash 6.8.8 Release Notes
- Logstash 6.8.7 Release Notes
- Logstash 6.8.6 Release Notes
- Logstash 6.8.5 Release Notes
- Logstash 6.8.4 Release Notes
- Logstash 6.8.3 Release Notes
- Logstash 6.8.2 Release Notes
- Logstash 6.8.1 Release Notes
- Logstash 6.8.0 Release Notes
- Logstash 6.7.2 Release Notes
- Logstash 6.7.1 Release Notes
- Logstash 6.7.0 Release Notes
- Logstash 6.6.2 Release Notes
- Logstash 6.6.1 Release Notes
- Logstash 6.6.0 Release Notes
- Logstash 6.5.4 Release Notes
- Logstash 6.5.3 Release Notes
- Logstash 6.5.2 Release Notes
- Logstash 6.5.1 Release Notes
- Logstash 6.5.0 Release Notes
- Logstash 6.4.3 Release Notes
- Logstash 6.4.2 Release Notes
- Logstash 6.4.1 Release Notes
- Logstash 6.4.0 Release Notes
- Logstash 6.3.2 Release Notes
- Logstash 6.3.1 Release Notes
- Logstash 6.3.0 Release Notes
- Logstash 6.2.4 Release Notes
- Logstash 6.2.3 Release Notes
- Logstash 6.2.2 Release Notes
- Logstash 6.2.1 Release Notes
- Logstash 6.2.0 Release Notes
- Logstash 6.1.3 Release Notes
- Logstash 6.1.2 Release Notes
- Logstash 6.1.1 Release Notes
- Logstash 6.1.0 Release Notes
Avro codec plugin
editAvro codec plugin
edit- Plugin version: v3.3.1
- Released on: 2021-12-07
- Changelog
For other versions, see the Versioned plugin docs.
Installation
editFor plugins not bundled by default, it is easy to install by running bin/logstash-plugin install logstash-codec-avro
. See Working with plugins for more details.
Getting Help
editFor questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.
Description
editRead serialized Avro records as Logstash events
This plugin is used to serialize Logstash events as Avro datums, as well as deserializing Avro datums into Logstash events.
Event Metadata and the Elastic Common Schema (ECS)
editThe plugin behaves the same regardless of ECS compatibility, except adding the original message to [event][original]
.
Encoding
editThis codec is for serializing individual Logstash events as Avro datums that are Avro binary blobs. It does not encode Logstash events into an Avro file.
Decoding
editThis codec is for deserializing individual Avro records. It is not for reading Avro files. Avro files have a unique format that must be handled upon input.
Partial deserialization
Avro format is known to support partial deserialization of arbitrary fields,
providing a schema containing a subset of the schema which was used to serialize
the data.
This codec doesn’t support partial deserialization of arbitrary fields.
Partial deserialization might work only when providing a schema which contains
the first N
fields of the schema used to serialize the data (and
in the same order).
Usage
editExample usage with Kafka input.
input { kafka { codec => avro { schema_uri => "/tmp/schema.avsc" } } } filter { ... } output { ... }
Avro Codec Configuration Options
editSetting | Input type | Required |
---|---|---|
No |
||
Yes |
||
No |
||
No |
ecs_compatibility
edit- Value type is string
-
Supported values are:
-
disabled
: Avro data added at root level -
v1
,v8
: Elastic Common Schema compliant behavior ([event][original]
is also added)
-
Controls this plugin’s compatibility with the Elastic Common Schema (ECS).
schema_uri
edit- This is a required setting.
- Value type is string
- There is no default value for this setting.
schema path to fetch the schema from. This can be a http or file scheme URI example:
-
http -
http://example.com/schema.avsc
-
file -
/path/to/schema.avsc
tag_on_failure
edit- Value type is boolean
-
Default value is
false
tag events with _avroparsefailure
when decode fails
target
edit- Value type is string
- There is no default value for this setting.
- This is only relevant when decode data into an event
Define the target field for placing the values. If this setting is not set, the Avro data will be stored at the root (top level) of the event.
Example
input { kafka { codec => avro { schema_uri => "/tmp/schema.avsc" target => "[document]" } } }
On this page