From the course: Azure Spark Databricks Essential Training

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Understand MLflow

Understand MLflow

From the course: Azure Spark Databricks Essential Training

Start my 1-month free trial

Understand MLflow

“

- [Instructor] So the complexity of these pipelines is really one of the biggest hindrances from taking machine learning models and complex computation into value for the business. So a typical Azure-hosted complex pipeline will look something like this. On the left side you'll see the incoming data, business apps, custom apps, sensors and devices, then the models in the center, and on the right is the intelligence, so the predictive apps, the operational reports, and the analytical dashboards. So the key phases are in Ingest, Store, Prep and Train, Model and Serve. So Databricks, you can see, is in the Prep and Train phase in this particular high-level reference architecture, but it often is part of the Model and Serve phases as well, in my experience. So in Ingest, we have a hot pipeline here with Kafka, or the Azure Event Hub or IoT Hub bringing in data in a streaming fashion. And then for storage, this pipeline is showing Blobs on the Data Lake. Prep and Train is Databricks using…

Contents