From the course: Azure Spark Databricks Essential Training

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Use a Spark Streaming notebook

Use a Spark Streaming notebook

From the course: Azure Spark Databricks Essential Training

Start my 1-month free trial

Use a Spark Streaming notebook

“

- [Instructor] In this example notebook, we're going to take a look at working with data in two types of input methods. The first method is batched, or regular input, and in the second method, we're going to take a look at the new structured streaming API. This is available for Python DataFrames. So as we did in previous examples, what I've done is I've just imported this. And you can see, here it is. And then of course, you'll remember that to interact with a cluster, we're going to attach it. So this is a relatively new API, and it's been very popular with my customers. The idea is that you can do complex distributed data processing in your cluster on both batch and streaming data. It does require, as it says here in command one, that you attach it to a Spark 2.x or greater cluster, as we've done. So, we're going to work with some new type of data. We're not going to use the diamond dataset anymore, we're going to use the event dataset. So let's take a look at that. Now, you'll…

Contents