From the course: Apache Spark Deep Learning Essential Training

Setting up a Databricks environment - Spark DataFrames Tutorial

From the course: Apache Spark Deep Learning Essential Training

Start my 1-month free trial

Setting up a Databricks environment

“

- [Instructor] So head over to Databricks and log in and let's head over to the Cluster and select Create Cluster and give it a name. Let's call it DL, for deep learning, and select Create Cluster. It takes a couple of seconds to create this cluster and once the cluster has been created, the state moves from pending to running. So hover over the cluster and select Libraries. You now want to install the relevant libraries. So select Install New and from the Library Source select Maven, select Search Packages, and wait for the list to populate. Sometimes the list doesn't come up immediately, so what I do is I close the window and select Search Packages again. So I now enter, deep learning. So that's deep learning and I can see that option is available to me. And we're going to be using release number 1.4.0. So select that and select the option using that link and now click on Install to install this package. We need to install a couple of other libraries, so head back to Install New and this time select PyPi as a source and enter TensorFlow in the packages. So, TensorFlow equals equals and we want to install version 1.12.0. Select Install. Head back to Install New. Select PyPi and this time you want to install Keras. So that's Keras equals equals and we want to install Keras version 2.2.4. Select Install. Now TensorFlow and Keras are deep learning frameworks. What we also need is the H5Py package, which is a Python interface to the HDF5 binary data format. So I select Install New. Select PyPi and type H5Py and I want to install version 2.7.0. And the last library we need to install is the wrapt and this is used for creating function wrappers and decorator functions. So select Install New, PyPi, and wrapT or wrapt and select Install. And once all five packages have been installed, we're now in a position to run deep learning in our Apache Spark environment.

Contents