Driving your web-scraper - Conclusion 💭

Driving your web-scraper - Conclusion 💭
Photo by Kenny Eliason / Unsplash
👨🏼‍💻
This is an excerpt from my long form article Web-scraping requires you to think: How to construct your web-scraper ethically and make it more human.
You can find the full article (with more context) now on Medium

To make use of the vehicle, we need to execute it in a main script (vehicle.py).

I would recommend that your main script also documents with which pages you interacted and when, to keep track of your progress. Consider making interaction with the program convenient, e.g., by Editing parameters in a text-file, as done here with the file cities.txt, which would contain the names of the cities where we want to search for houses.

This post is for subscribers only

Already have an account? Sign in.

Subscribe to ds-econ

Sign up now to get access to the library of members-only issues.
Jamie Larson
Subscribe