Drawing up your web-scraper πŸ‘·β€β™‚οΈ

Drawing up your web-scraper πŸ‘·β€β™‚οΈ
Photo by Kenny Eliason / Unsplash
This is an excerpt from my long form article Web-scraping requires you to think: How to construct your web-scraper ethically and make it more human.
You can find the full article (with more context) now on Medium

Before we start writing the code for our web-vehicle, we need to start at the drawing board and plan it out. Good code starts on a piece of paper.

First, use the website for a while and figure out its behaviours. Are there, e.g., sub-pages that take a while to load? Is there content that appears only after you scroll down? How many entries are there per page? Your vehicle might need to be able to navigate these types of quirks, consider what it has to be able to do and what not.

This post is for subscribers only

Already have an account? Sign in.

Subscribe to ds-econ

Sign up now to get access to the library of members-only issues.
Jamie Larson