From the course: Apache Kafka Essential Training: Building Scalable Applications (2021)

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Batching message consumption

Batching message consumption

From the course: Apache Kafka Essential Training: Building Scalable Applications (2021)

Start my 1-month free trial

Batching message consumption

- [Instructor] In this video, let's explore how consumers can receive messages in batches and process them. How do consumers receive messages from Kafka. Consumers Poll Kafka periodically for new messages. The poll interval is configurable through the poll method. Each consumer is allocated a set of partitions to work with. Consumers poll corresponding partition leaders for new messages. When consumers poll, a batch of messages are returned from Kafka when new messages are available. The bigger the batch sizes and higher the poll interval, the lower or the round trips and request overloads. However, such large sizes and poll intervals, directly impact the latency of processing also. It is required to choose the right parameters based on the use case to balance between round-trips and latency. How do we configure this round-trip behavior? Here are some key parameters. The first one is the poll interval, which is the…

Contents