From the course: Web Scraping with Python
Join today to access over 22,500 courses taught by industry experts or purchase this course individually.
Scrapy settings file - Python Tutorial
From the course: Web Scraping with Python
Scrapy settings file
- [Instructor] As is usually the case when it comes to programming, scraping provides the user with multiple ways to do things. SO in the last section, we looked at an example where Scrapy would write CSV data to file. We also limited the number of pages it scraped with this command line option. But the command line options are hard to remember. Sometimes you just want to load some presets. So remember, in the last section we used the command CLOSPIDER_PAGECOUNT equals 10. The dash s here actually stands for settings. These are one time settings that you can pass in via the command line, but if you want to make these applicable for every command, without having to write them each time, take a look at settings.py. We can write whatever we want in here. There's a bunch of great stuff. I would recommend reading through the comments and getting an idea of things you can control from this file. And it's pretty…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.