From the course: Azure Spark Databricks Essential Training
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Azure Databricks for data warehousing
From the course: Azure Spark Databricks Essential Training
Azure Databricks for data warehousing
- [Instructor] For our first reference use case let's look at data warehouses and databricks. Typically data warehouses need data that is prepared for loading and there's some sort of processing. And we also could be looking at a migration. So let's start with loading and processing. One possible reference architecture could be the one shown here where we're loading data both using the streaming pattern and the batch pattern and we're using something like Kafka to load via a stream and Data Lake Store to load via a batch. Of course there's other components that we could be using. The role of Azure Databricks in this scenerio would be to perform the extract transform and load or ETL for the incoming data and of course Databricks is running on Spark which is very, very easy to parallelize and runs in memory so any sort of data cleaning, transforming, aggregating, that sort of activity can be done much, much more rapidly given the volume of data that you may be working with. And then the…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.