From the course: Azure Spark Databricks Essential Training

Unlock the full course today

Join today to access over 22,400 courses taught by industry experts or purchase this course individually.

Azure Databricks pipeline considerations

Azure Databricks pipeline considerations

From the course: Azure Spark Databricks Essential Training

Start my 1-month free trial

Azure Databricks pipeline considerations

- [Instructor] In this section, we're going to take a look at Databricks Azure in architectural pipelines, and we've talked about pipelines in several different contexts, and what I'm really going for here is end-to-end from when you receive the data to when you present the data and to show you some examples in terms of reference architectures of how this is best presented. So the pipelines will cover both cleansing and processing. We'll also highlight some enterprise features that are available through the integration with Azure security monitoring features. We'll talk about storage services that are commonly used, such as Azure Blob Store and Azure Data Lake and visualization tools such as Power VI. Also cost control is an important component because there are a number of services and the integration to Azure can help you to manage that. So pipeline characteristics come in several phases. The first phase is to preprocess the data, to clean it, compress it, partition it, so on, so…

Contents