Driving your web-scraper - Conclusion π
π¨πΌβπ»
This is an excerpt from my long form article Web-scraping requires you to think: How to construct your web-scraper ethically and make it more human.
You can find the full article (with more context) now on Medium
You can find the full article (with more context) now on Medium
To make use of the vehicle, we need to execute it in a main script (vehicle.py
).
I would recommend that your main script also documents with which pages you interacted and when, to keep track of your progress. Consider making interaction with the program convenient, e.g., by Editing parameters in a text-file, as done here with the file cities.txt
, which would contain the names of the cities where we want to search for houses.
This post is for subscribers only
Already have an account? Sign in.