Humanise your web-scraper 🕊️
👨🏼💻
This is an excerpt from my long form article Web-scraping requires you to think: How to construct your web-scraper ethically and make it more human.
You can find the full article (with more context) now on Medium
You can find the full article (with more context) now on Medium
When programming a web-scraper, let your code sleep. A lot.
Think about where you need to sleep (e.g., in everyiteration of a for-loop). Consider randomly selecting the exact sleep time for every iteration, as this is closer to a human interaction than waiting exactly 5 seconds every time.
This post is for subscribers only
Already have an account? Sign in.