From the course: Stream Processing Patterns in Apache Flink
Unlock the full course today
Join today to access over 22,400 courses taught by industry experts or purchase this course individually.
Streaming analytics: Pipeline implementation - Flink Tutorial
From the course: Stream Processing Patterns in Apache Flink
Streaming analytics: Pipeline implementation
- [Facilitator] Having looked at the helper classes for the Streaming Analytics Pattern. Let's now explore the main pipeline for Streaming Analytics. The code for this implementation is in the Streaming Analytics class. We initialize the stream execution environment for Flink using the getExecutionEnvironment function. This sets up an embedded Flink execution pipeline. We also print the parallelism for the current environment. Typically in the embedded mode this is equal to the number of CPU codes. Each code will run a parallel task and update the MariaDB database independently. We (mumbles) create a MariaDB tracker in a separate thread. This tracker would print summaries of orders in the database every five seconds. We now start the Kafka Order Data Generator in a separate thread. This will start publishing order records at random intervals to the Kafka topic, streaming.orders.input. In order to consume data, we set up a…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.