From the course: Introduction to Spark SQL and DataFrames

Unlock the full course today

Join today to access over 22,500 courses taught by industry experts or purchase this course individually.

Save data from DataFrames

Save data from DataFrames

From the course: Introduction to Spark SQL and DataFrames

Start my 1-month free trial

Save data from DataFrames

“

- [Instructor] Now sometimes after we've been working with DataFrames, and creating new DataFrames, and running calculations, and doing sampling and so forth, we might want to save our results out. It's very simple way to do that. Now, here, I've started with a DataFrame already loaded, df1 has our location temperature data. If I want to save that out, I can simply specify the name of the DataFrame, that's df1, and then use the write method and specify the csv method within that, and then specify a name for what I'd like to save. In this case, it's df1.csv. And I'll execute that. So what that's done is it has saved the DataFrame out to disk using the csv format. So let's take a look at the directory. So, to execute a shell command from within the Jupyter notebook, you can use the exclamation mark, and that basically tells Jupyter notebook, everything that's going to follow is basically a shell command. So I can do…

Contents