From the course: Stream Processing Patterns in Apache Flink

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Streaming analytics: Pipeline implementation

Streaming analytics: Pipeline implementation - Flink Tutorial

From the course: Stream Processing Patterns in Apache Flink

Start my 1-month free trial

Streaming analytics: Pipeline implementation

- [Facilitator] Having looked at the helper classes for the Streaming Analytics Pattern. Let's now explore the main pipeline for Streaming Analytics. The code for this implementation is in the Streaming Analytics class. We initialize the stream execution environment for Flink using the getExecutionEnvironment function. This sets up an embedded Flink execution pipeline. We also print the parallelism for the current environment. Typically in the embedded mode this is equal to the number of CPU codes. Each code will run a parallel task and update the MariaDB database independently. We (mumbles) create a MariaDB tracker in a separate thread. This tracker would print summaries of orders in the database every five seconds. We now start the Kafka Order Data Generator in a separate thread. This will start publishing order records at random intervals to the Kafka topic, streaming.orders.input. In order to consume data, we set up a…

Contents