From the course: Azure Spark Databricks Essential Training

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Understand ML Pipelines API

Understand ML Pipelines API

From the course: Azure Spark Databricks Essential Training

Start my 1-month free trial

Understand ML Pipelines API

“

- [Instructor] As we work with a product, I have found that a lot of things are called ML and I wanted to clarify terms. So some are associated to Spark and some are Databricks. So Spark has what are called MLlib, that's an API, to simply machine learning on Spark. So it contains algorithms and pipelines as you see below, so MLlib algorithms called ML algorithms and those are algorithms for classification, regression, typical machine learning problems. So they would be something like decision trees for example. Spark MLlib pipelines, which we're going to be exploring next, is an API to create and tune models and it supports hyper-parameter optimization and many other features to work with machine learning models. So in the Spark world we have MLlib, MLlib algorithms, which are called ML algorithms, and ML pipelines. In the Databricks world, we have the Databricks runtime ML which we just saw in the previous movie. That is a cluster type which supports third-party machine learning…

Contents