From the course: Stream Processing Design Patterns with Spark
Unlock the full course today
Join today to access over 22,400 courses taught by industry experts or purchase this course individually.
Alerts and thresholds: Pattern - Apache Spark Tutorial
From the course: Stream Processing Design Patterns with Spark
Alerts and thresholds: Pattern
- [Instructor] I will discuss the goal for the alerts and thresholds use case and create a pipeline designed for the same in this video. What is the use case we will be solving here for alerts and thresholds? We have an input Kafka topic that contains real-time exception logs collected from various services across the network. They are flowing into Kafka in real time. Each message is of the CSV format. It contains the timestamp of when the exception occurred, the level of the error, whether it is critical, high, or elevated. Then it has the error code and, finally, an error message. There are two goals for this use case. First, every time a critical exception happens, send the exception to a special critical alerts Kafka topic. Second, when the same error code occurs more than two times in a 10-second interval, send an alert to a high-volume alert Kafka topic. How would the pipeline designed for this use case look like? We…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.