From the course: Data Engineering Foundations

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Loading data into a DB

Loading data into a DB

From the course: Data Engineering Foundations

Start my 1-month free trial

Loading data into a DB

“

- [Tutor] So far we have extracted and transformed the data. And it is time to load this transform data back to the database in a new table. Before that, you're first going to wrap these extraction and transformation code snippets into separate functions, so that automation becomes easy and these function would be reusable as well. So here you can see I have first graded the spark session as we did earlier, then we moved on to actually define or wrap the movie extraction or the user data extraction code snippets into functions. So I've created this first function to extract movie table data which I've named, extract movies to df. So here I'm doing the same thing, all I've done is simply define a function and I'm returning the movies on the score data frame. Similarly for the users table as well, I have named it extract users to df and I am returning the user_dataframe that I have just read using the spark session.…

Contents