Drawing up your web-scraper π·ββοΈ
π¨πΌβπ»
This is an excerpt from my long form article Web-scraping requires you to think: How to construct your web-scraper ethically and make it more human.
You can find the full article (with more context) now on Medium
You can find the full article (with more context) now on Medium
Before we start writing the code for our web-vehicle, we need to start at the drawing board and plan it out. Good code starts on a piece of paper.
First, use the website for a while and figure out its behaviours. Are there, e.g., sub-pages that take a while to load? Is there content that appears only after you scroll down? How many entries are there per page? Your vehicle might need to be able to navigate these types of quirks, consider what it has to be able to do and what not.
This post is for subscribers only
Already have an account? Sign in.