From the course: Faster pandas
Join today to access over 22,600 courses taught by industry experts or purchase this course individually.
Understanding eval and query
- [Tutor] For this video, you will need to install the Num expression library. You can do it with python -m pip install numexpr This technique works only on large datasets. We load the small one and duplicate it. So ipython and we're going to import pandas as pd and import sqlite3 and we'll find the connection as sqlite3.connect to our logs database and we're going to tell sqlite to detect_types equal sqlite3.PARSE_DECLTYPES. And then we're going to say that the DataFrame is pd.read_sql, select everything from logs using the connection and the length of the df is 10,000. So now we're going to duplicate it. So we're going through the df equal pd.concat, of the DataFrame time 1000, and we can check the length of the DataFrame. This is how to read. So what I usually do is len of df and I'm telling Python to format it with commas. We see we have 10 million rows now, and now let's say that we want to get all the rows where the method is get and the status code is bigger or equal to 400…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.