IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

›

Ingest architectures

edit

Ingest architectures

edit

We offer a variety of ingest architectures to serve a wide range of use cases and network configurations.

To ingest data into Elasticsearch, use the simplest option that meets your needs and satisfies your use case. For many users and use cases, the simplest approach is ingesting data with Elastic Agent and sending it to Elasticsearch. Elastic Agent and Elastic Agent integrations are available for many popular platforms and services, and are a good place to start.

You can host Elasticsearch on your own hardware or send your data to Elasticsearch on Elastic Cloud. For most users, Elastic Agent writing directly to Elasticsearch on Elastic Cloud provides the easiest and fastest time to value. Our hosted Elasticsearch Service is available on AWS, GCP, and Azure, and you can try it for free.

Decision tree

Data ingestion pipeline with decision tree

Ingest architecture	Use when
Elastic Agent to Elasticsearch	An Elastic Agent integration is available for your data source: Software components with Elastic Agent installed Software components using APIs for data collection
Elastic Agent to Logstash to Elasticsearch	You need additional capabilities from Logstash: enrichment between Elastic Agent and Elasticsearch persistent queue (PQ) buffering to accommodate network issues and downstream unavailability proxying in cases where Elastic Agents have network restrictions for connecting outside of the Elastic Agent network data needs to be routed to multiple Elasticsearch clusters and other destinations depending on the content
Elastic Agent to proxy to Elasticsearch	Agents have network restrictions that prevent connecting outside of the Elastic Agent network Note that Logstash as proxy is one option
Elastic Agent to Elasticsearch with Kafka as middleware message queue	Kafka is your middleware message queue: Kafka ES sink connector to write from Kafka to Elasticsearch Logstash to read from Kafka and route to Elasticsearch
Logstash to Elasticsearch	You need to collect data from a source that Elastic Agent can’t read (such as databases, AWS Kinesis). Check out the Logstash input plugins.

Elastic Agent to Elasticsearch »