From the course: Azure Spark Databricks Essential Training

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Use distributed ML training

Use distributed ML training

From the course: Azure Spark Databricks Essential Training

Start my 1-month free trial

Use distributed ML training

“

- [Instructor] We've been looking at relatively common use cases on ML pipelines for Spark with manual pipelines and Spark ML pipelines, and I think it's fun to look at what for most of us will be the future, and this is around distributed TensorFlow, which is a deep neural network, the most popular library, open sourced by Google, with a new library called Horovod and this is open sourced from Uber. So, to understand the class of problems that this is designed to solve you really want to put your future hat on. Uber is using this to process all of their massive amounts of data from their worldwide distributed driver network and companies like Google and Facebook are using it as well. So, this is really at the highest volumes of data, the most complex compute, but as I've mentioned, I've been working in the world of genomics and I'm starting to applications from some of these companies be applicable in other domains, such as genomics. So, we're going to take a look. Now, one thing…

Contents