From the course: Apache Kafka Essential Training: Getting Started (2021)

Message queues

From the course: Apache Kafka Essential Training: Getting Started (2021)

Start my 1-month free trial

Message queues

“

[Instructor]- Before we dive into Apache Kafka it's important to understand the publish subscribe pattern and the benefits it provides. Let's review publish subscribe with message queues in this video. When we build distributed services, we have multiple running process instances that communicate with each other over the network to exchange messages. Let's start with the simple two process network where a sender process communicates but the receiver process to exchange data. In this case the sender needs to know the address of the receiver. In addition, both the sender and the receiver needs to agree upon a common data protocol and farmer. This creates a strong static binding between the two processes. Now suppose we have three receivers each responsible for a different type of processing based on the data received from the sender. In this case, the sender needs to know about all the three receivers. Addition of new receivers require changes on the sender side to configure the new receiver. Again, this creates a strong coupling between the senders and receivers. What happens when there are multiple senders, and multiple receivers exchanging the same type of data between them? The management becomes a lot more cumbersome here but each sender should know about each receiver. Add fault tolerance into the picture and it will get more complex. A clean and scalable solution here is to use a message queue. Each sender needs to only know about the message queue and simply publish the messages to the queue. A receiver becomes a subscriber and subscribes to the message queue. When a new message appears in the queue the subscriber is notified, who then proceeds to pull the message and use it. In this case each of the publishers and subscribers only need to know about the message queue. They are unaware of other publishers and subscribers using the same queue. This is the publish subscribe pattern also called pubsub. This pattern has many advantages. To begin with, the publishers and subscribers are decoupled from each other. This results in easy management of the setup where publishers and subscribers can be added and removed without any changes on others in the network. It then allows to scale the publishers and subscribers easily provided the message queue can handle the load. Message queues also provide Back-pressure handling. If the publishers generate data in spikes the message queues can act as a buffer zone to cache data until the subscribers catch up and process them. Message queues can also provide reliability through persistent queue data and tracking consumption of data. Apache Kafka is such a technology that provides these advantages. Let's learn more about it in the next video.

Contents