Configure the internal queue

Filebeat uses an internal queue to store events before publishing them. The queue is responsible for buffering and combining events into batches that can be consumed by the outputs. The outputs will use bulk operations to send a batch of events in one transaction.

You can configure the type and behavior of the internal queue by setting options in the queue section of the filebeat.yml config file.

Example configuration:

queue.mem:
  events: 4096

The memory queue keeps all events in memory. It is the only queue type supported right now. By default no flush interval is configured. All events published to this queue will be directly consumed by the outputs. The output’s bulk_max_size setting limits the number of events being processed at once.

The memory queue waits for the output to acknowledge or drop events. If the queue is full, no new events can be inserted into the memeory queue. Only after the signal from the output will the queue free up space for more events to be accepted.

To enforce spooling in the queue, set the flush.min_events and flush.timeout options.

This sample configuration forwards events to the output if 512 events are available or the oldest available event is already waiting for 5s in the queue:

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

You can specify the following options in the queue.mem section of the filebeat.yml config file:

Number of events the queue can store.

The default value is 4096 events.

Minimum number of events required for publishing. If this value is set to 0, the output can start publishing events without additional waiting times. Otherwise the output has to wait for more events to become available.

The default value is 0.

Maximum wait time for flush.min_events to be fulfilled. If set to 0s, events will be immediately available for consumption.

The default values is 0s.