Scaling scrapers to scrap 10 millions items
Web scraping is an interesting job, finding out imperfections or loop holes in a site that can be used to scrap the website without rate limits. With current webstack framework like react or vue, it gets quite difficult to work with just simple requests module as it demands rendering of…