« Elastic Agent to Elasticsearch with Kafka as middleware message queue Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue »
Elastic Docs ›Elastic Ingest Reference Architectures [8.16] ›Elastic Agent to Elasticsearch with Kafka as middleware message queue

Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue

edit

Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue

edit
Image showing connections between Elastic Agent and Elasticsearch using a Kafka messaging queue
Ingest model

Control path: Elastic Agent to Fleet to Elasticsearch
Data path: Elastic Agent to Logstash to Kafka to Logstash to Elasticsearch: Kafka as middleware message queue.

Logstash reads data from Kafka and routes it to Elasticsearch clusters (and/or other destinations)

Use when
You are standardizing on Kafka as middleware message queue between Elastic Agent and Elasticsearch
Notes
The transformation from raw data to Elastic Common Schema (ECS) and any other enrichment can be handled by Logstash as described in Elastic Agent to Logstash (for enrichment) to Elasticsearch.

Resources

edit

Info on Elastic Agent and agent integrations:

Info on Logstash and Logstash Kafka plugins:

Info on Elasticsearch:

« Elastic Agent to Elasticsearch with Kafka as middleware message queue Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue »