From the course: Faster pandas

Unlock the full course today

Join today to access over 22,600 courses taught by industry experts or purchase this course individually.

The limitations of appending

The limitations of appending

From the course: Faster pandas

Start my 1-month free trial

The limitations of appending

“

- [Instructor] There will be cases, when you'll want to build a data frame dynamically. The first instinct is to use DataFrame.append method. Let's have a look. We have a file with several log files, we'd like to load it into a data frame. Here's the code, to parse_line, we have a parse_time which parses a timestamp, which is a string into a datetime, and a function called parse_line, which parses the line with origin, time, method, path, status code and size. And now let's use, let us ignore this one, and now let's use it. So ipython and since the logs are compressed, I'm going to use lzma to load the logs and with lzma.open of the logs and I want to read them in textual format as fp and I'm going to do lines = fp.read lines, load the whole file into memory because we're going to iterate over it several times. In real life you'll probably do it one line at a time. And let's have a look, we have 1000 lines and now we can run our code parse_log.py and we will import pandas as pd. We…

Contents