From the course: Stream Processing Design Patterns with Spark
Unlock the full course today
Join today to access over 22,400 courses taught by industry experts or purchase this course individually.
Leaderboards: Use case design - Apache Spark Tutorial
From the course: Stream Processing Design Patterns with Spark
Leaderboards: Use case design
- [Instructor] In this video, I will discuss the example use case for which we will build a leaderboard in this chapter. I will also explore the design for this example. The example use case for leaderboards is to build a gaming leaderboard for players. It shows the players who have the current top scores. The scores are continuously updated, and the score increments are published to Kafka. So the input is a Kafka topic that contains score increments by players. It's a key value input where the key represents the player. The value represents the score increment for the player. The goal is to maintain a player leaderboard based on the current total scores. What is the pipeline design for this use case? The input is a Kafka topic called streaming.leaderboards.input. Each message received would be a key value path. This needs to be converted to a data frame for further processing. The next stage is to update the leaderboard…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.