<![CDATA[Blog]]>https://blog.shivy.co.in/https://blog.shivy.co.in/favicon.pngBloghttps://blog.shivy.co.in/Ghost 3.19Wed, 22 Jul 2026 16:10:36 GMT60<![CDATA[Scaling scrapers to scrap 10 millions items]]>Web scraping is an interesting job, finding out imperfections or loop holes in a site that can be used to scrap the website without rate limits. With current webstack framework like react or vue, it gets quite difficult to work with just simple requests module as it demands rendering of

]]>https://blog.shivy.co.in/scraping-10-million-items-with-scaling/61ba2d7752f2e05312d6a76bSun, 26 Dec 2021 19:16:37 GMTWeb scraping is an interesting job, finding out imperfections or loop holes in a site that can be used to scrap the website without rate limits. With current webstack framework like react or vue, it gets quite difficult to work with just simple requests module as it demands rendering of javascript to load dom content. It usually ends up with either using selenium to scrap the page content or mess around the network tab to look for any open APIs that can be used.

But what's next step, after building a scraper? Deployment!
Issue with deployment is most of the scrapers I have seen or worked on aren't scalable after some point. Sure they can scrap 10k records from facebook or weibo without getting blocked but that's it. Scaling them vertically is not an option since it binds to the same IP address of machine which can lead to 419 error (too many requests error) or direct block.

I have heard of many solutions where the links are divided in batches and executed on different machines. While that works, issue is if a machine goes down or I have an extra machine to add, I have to start over again.

tl;dr: Celery is the solution

Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.
- Celery's homepage

In one of the recent projects, I had to scrap website with around 338 search pages, and each page containing 100 items and for each item there are around 30-500 comments.
Idea is dividing the scraping of single item into basic celery tasks that can be sent over to queue and processed as atomic functions in different machines.

A trigger (job adder) script will make a search query in main website and send out the search result pages link to the SCRAP_SEARCH_PAGES worker i.e., 

1. domain.com/?q=iphone&page=1
2. domain.com/?q=iphone&page=2
3. domain.com/?q=iphone&page=3

SCRAP_SEARCH_PAGES will scrap all products links from the passed search page url, and send scraped product links to next worker SCRAP_PRODUCT_INFO.


SCRAP_PRODUCT_INFO will scrap item's details like title, description, seller etc. and send the generated item json to INSERT_INTO_DB worker.

And at the same time it will send the item to SCRAP_ITEM_COMMENTS worker, to process the comments as well.

SCRAP_ITEM_COMMENTS will scrap comments from the item's detail page and send them as json to INSERT_INTO_DB worker.

Instead of linking each worker with database, I used one db worker cause on scaling out other workers (scraper workers), it will also increase number of db connections hence increasing the load on database server creating a bottleneck.
I have measured single db worker running on t3.micro instance and mongo database server running on t3.medium and it was able to reach around ~420 entries per second, which was good enough for the project.

Picking up resources

I have used rabbitmq instead of going with redis for celery queue, while redis comes with easy plug an play config I don't think it could have handled 10+ million queue items without creating any latency issue for 35+ workers running at once. I went with rabbitmq for performance reasons and after adding memory swap and resizing allocated in-memory block for queues it worked without any issues on single t3.nano instance.

For the workers, it depends on what kind of task is being done. Like if it's image processing or uploading to s3 bucket, I'd prefer picking medium or large instance with 2-4 workers. If it's making simple http requests using requests module and bs4 parsing, micro instances work well. I picked both micro and large instances with aws autoscaling option to scale out instances depending on the size of job queues.

For easier and faster deployment, I built docker image and pushed everything to aws ecr. I wrote a bash script and put it in user-data so on every start of machine, it will pull the latest image and start worker with given env for queue, listening for queue passed in the $QUEUE env using the below command:

celery -A proj worker -Q $QUEUE

Conclusion

There are not many solutions to distributed scraping in the market. I have tested crawlab and scrapyd but they aren't very suitable for large jobs. While this solution is complicated to setup but it promises that no jobs are lost in case of system crash or restarting workers. With this setup, the entire site was scraped in less than a day. Only limit was how much spot instances can be allocated.

What we didn't do

  • Kubernetes setup for better CI/CD
  • Automatic scaling based on average cluster load or queue's size
  • Cluster monitoring using grafana to get proper utilization metrics of instances
  • Centralized logs if in case tasks were failing
]]><![CDATA[Shivy won the Microsoft Code for Future hackathon!]]>https://blog.shivy.co.in/shivy-won-microsoft-hackathon/5ee0b0c76b006f6addc6e5ffThu, 19 Nov 2020 21:01:55 GMTHello everyone, Shivy won the Microsoft Code for Future hackathon!

A while ago, we took part in Microsoft Code for future open source hackathon as Road Busters and in this post we will share our experience with you.

In general Microsoft Code for future hackathon was all about Smart Traffic Management system for Quick Commute and carbon reduction. We had to develop a system to smartly control the traffic for quick commutation and to reduce the carbon emission using Azure service and Open Source technologies.

Existing System

Traffic control system in Indian cities is not smart.

Shivy won the Microsoft Code for Future hackathon!
  • They don’t adapt to the dynamic situation of the continuously changing traffic.
  • Most common traffic systems use a pre-set timer which turns the traffic signal red/green after a specified delay.
  • The vehicles have to wait for a long time span even if the traffic density is very less which leads to increase in Carbon emission rate and increase the density of traffic.

Our Idea

Now to control the traffic smartly and reduce the carbon emission rate, we came up with an idea of passing the vehicles as soon as they reach the junction.

Shivy won the Microsoft Code for Future hackathon!

In other words, suppose a vehicle takes 2 minutes to reach crossroad B from crossroad A, then crossroad B shouldn’t turn green for vehicles that are coming from the direction of crossroad A for 2 minutes.

And in between those 2 minutes, crossroad B could be green for vehicles that are not coming from the direction of crossroad A.

In order to make it possible. We would need to know about the distance between crossroad A and crossroad B. With that we will able to find the average time a vehicle will take to reach crossroad B from crossroad A with average speed of 30 - 40 km/hr. We would also need the location of all traffic junctions.

With all that data we will able to build a system which would be able to control the traffic smartly and will reduce the carbon emission rate without any external hardware need.

In Technical Terms

Shivy won the Microsoft Code for Future hackathon!

If we assume roads as pipelines carrying water and each junction as valve which opens/closes to control the flow of the water. We can map logic to the roads such that only X amount of vehicles are passing through given location at a given point of time.

Implementing the idea

Firstly, we collected all the traffic junction location data and an area of Ahmedabad (a city in India) from Openstreet map.

We had to be creative, because we couldn't find all traffic junction locations.Cause this project was just to showcase how our idea will smartly control the traffic and how it will reduce carbon emission rate. Thus, manually we had to add traffic junction locations at each crossroad.

Then we used that data within our own GUI built on top of javascript to help us simulate how traffic will be affected with the new implementation.

It also helped in tuning the delay values to get least possible waiting time at each junction where traffic light is present.

Just like we implemented our idea for dummy data, we can implement it into the real world scenario and reduce the emission rate of carbon.

Challenges faced while building the system

  • The biggest challenge we faced was to find and collect valid data (all the traffic lights position in a city) and somehow integrate them in UI such that cars also follow that rule. We didn’t get the data we were looking for, hence we created another tool and filled up the traffic light positions manually.

    • Shivy won the Microsoft Code for Future hackathon! Shivy won the Microsoft Code for Future hackathon!

  • The other challenge was the UI itself like how to teach cars when to move or stop when there is green light or red light. For solving this challenge, we checked the distance of each car if it's near any traffic light
    • if yes, then check if the nearest traffic light is red
      • if yes, then stop the car
      • else go ahead

Deployment

Here comes the deployment of our smart traffic simulation system. We used heroku for the deployment. As we were using Express.js server and it was a small-scale project without much processing, memory consumption.

Feel free to check out our project here.

Technologies used and why?

Python with AioHTTP

We used AioHTTP library to get the route data. We used it instead of requests for making asynchronous request, non-blocking I/O calls, and for the fast results.

Mappa library

We used Mappa library in order to work with the static map. It allow us to overlay a canvas on top of a tile map.

p5.js library

We used p5.js for creating graphic within the web page.

Leaflet.js library

We used Leaflet.js for creating markers/cars in the map.

Route finding library

We used Pyroute routing library to find the shortest path between two crossroads.

OpenStreetMap

We used OpenStreetMap for getting the map/tiles of an area of Ahmedabad city.

Overpass Turbo

Overpass turbo (overpass-turbo.eu) is a web based data mining tool for OpenStreetMap. We used it to get distance between crossroads and for getting the location of the crossroads.

Some Q/A

  • How will this project make the earth more greener?
    • We are trying to reduce the waiting time at each traffic junction without any extra hardware or software by adjusting the timers of the city to take into account the traffic amount from their adjacent junctions.
    • With less waiting time there’ll be less wastage of fuel.
  • Will it require any extra hardware like other adaptive solutions?
    • No, it won’t require any extra hardware for the first phase.
    • We can calculate value of traffic light of each junction in the city on one serve and can change them in the city accordingly.
    • For next phase, we can add microcontroller to change the values dynamically when server updates the value from its end.
  • Can it also adapt to rush hours and weekend traffic automatically?
    • At phase one it can’t, but once we add microcontroller to automatically update time from server it can handle rush hours and weekend traffic based on previously collected dataset.
  • Will it also help when there’s sudden traffic change?
    • No, it can’t help in such situation.
    • We need to add crowd detection module to detect crown at each junction and trigger traffic lights.

AM aka Ajay's Experience in his own words -

It was all in a rush when Endo called me and suggested that we should register in Microsoft’s Code for Future hackathon on 10th of March (last day of registration).

There was this idea that I had been pondering over whenever I stopped at a red light at a traffic junction; like if somehow my timings were correct along with my vehicle’s speed. I could reach home without encountering any red light or in the least amount of them.

I occasionally did find at some point the perfect timing, if I left my office by 19:00 and hit the road by 19:15 I was able to avoid most of the red lights that I encountered on other days.

With this hackathon, we got a chance to work on this and even though we didn’t exactly complete it, we know that it is possible to build a system without the need of any extra hardware or image recognition cameras to regulate traffic with existing technology if we tweak timings accordingly.

We hope you found this post interesting and helpful.

Link to Our Project

Thank you so much for reading this post.

]]>