From the course: Introduction to Spark SQL and DataFrames

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

Filter data with DataFrame API

Filter data with DataFrame API

From the course: Introduction to Spark SQL and DataFrames

Start my 1-month free trial

Filter data with DataFrame API

“

- [Instructor] Now let's take a look at how we can use the Dataframe API to filter some of the rows in our Dataframe. I uploaded the location temp csv file and again that's time series data that has a location ID and a temperature measurement at a particular point and time. Now we have a number of different locations. So, for example, if we wanted to see all of a particular location, we could specify a Dataframe name in this case df1, and we want to apply the filter command or filter method. And what we want to do is we want to specify that the Dataframe 1 location ID column should be equal to, let's call it loc0, location 0. And that'll return some results and let's make sure we show those results. And what you'll notice here is we got 20 rows of location 0 but how many are there? Let's take a look at how we can get a count for the results of a filter. So let's filter Dataframe 1 using the filter method.…

Contents