From the course: Stream Processing Design Patterns with Spark

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Alerts and thresholds: Pattern

Alerts and thresholds: Pattern - Apache Spark Tutorial

From the course: Stream Processing Design Patterns with Spark

Start my 1-month free trial

Alerts and thresholds: Pattern

- [Instructor] I will discuss the goal for the alerts and thresholds use case and create a pipeline designed for the same in this video. What is the use case we will be solving here for alerts and thresholds? We have an input Kafka topic that contains real-time exception logs collected from various services across the network. They are flowing into Kafka in real time. Each message is of the CSV format. It contains the timestamp of when the exception occurred, the level of the error, whether it is critical, high, or elevated. Then it has the error code and, finally, an error message. There are two goals for this use case. First, every time a critical exception happens, send the exception to a special critical alerts Kafka topic. Second, when the same error code occurs more than two times in a 10-second interval, send an alert to a high-volume alert Kafka topic. How would the pipeline designed for this use case look like? We…

Contents