From the course: Apache Kafka Essential Training: Getting Started (2021)

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Intro to partitions

Intro to partitions

From the course: Apache Kafka Essential Training: Getting Started (2021)

Start my 1-month free trial

Intro to partitions

- [Instructor] In this chapter, we will deep dive into partitions and how they control the ingestion, storage, and consumption of messages. As discussed previously, each topic in Kafka can contain multiple partitions. A topic can have 1-n partitions. The number of partitions are set up during topic creation. The maximum number of partitions per cluster and per topic varies by the specific version of Kafka. Partitions allow Kafka to scale by parallelizing ingestion, storage, and consumption of messages. It provides horizontal scalability. However, creating too many partitions may result in increased memory usage and file handles. Each partition has separate physical log files which of course will rollover as they reach configured sizes. A given message in Kafka is taught in only 1 partition. Each partition is assigned a broker process, known as its Leader broker. In order to write to a specific partition, the message needs…

Contents