Scalable do-it-yourself scraping – How to build and run scrapers on a large scale

Large scale scrapingBusinesses that don’t rely on data have a meager chance of success in a data-driven world.

One of the best sources of data is the data available publicly online on various websites and to get this data you have to employ the technique called Web Scraping or Data Scraping.

You can use full-service professionals such as ScrapeHero to do all this for you or if you feel brave enough, you can tackle this yourself.

The purpose of this article is to walk you through some of the things you need to do and the issues you need to be cognizant of when you do decide to do it yourself.

When you decide to do this yourself, you will most likely be hiring a few developers who know how to build scrapers and setting up some servers and related infrastructure to run these scrapers without interruption and integrating the data you extract into your business process.

Building and maintaining a large number of web scrapers is a very complex project that like any major project requires planning, personnel, tools, budget and skills.

Here are the high-level steps involved in this process and we will go through each of these in detail in this article.

  1. Building scrapers
  2. Running web scrapers at scale
  3. Getting past anti-scraping techniques – IP Rotation, Proxies
  4. Data Validation and Quality Control
  5. Ongoing Maintenance

Building Scrapers

The first thing to do is to build the scrapers. You can use one of the many web scraping tools and frameworks.

It would be wise to choose an open-source web scraping framework for building your scrapers – like Scrapy (Python), PySpider (Python) or Puppeteer(Javascript). You won’t run into the risk of your developer(s) disappearing one day, leaving no one to maintain your scrapers. You will not lock yourself into the ecosystem of a proprietary tool, having no way to move hundreds of scrapers into another tool if they shut down. This has already happened in our industry with a high profile company, read more about that here – Demise of Kimono Labs and the fate of their users.

The choice of tools or frameworks should depend on a few factors depending upon the website(s) that you plan to scrape. Some tools are better that others at handling complexity of the sites. Generally speaking, more flexible the tool, more the learning curve and conversely the easy to use tools may not handle complex sites or complex logic.

What is a complex website?

Any website that is built using some advanced JavaScript frameworks such as React or Angular is usually complex if you are extracting a lot of data from it. You will need a real web browser such as Puppeteer or Selenium to scrape the data. Alternatively you can check and  reverse engineer the REST API of the website if it exists.

What is a complex logic?

Here are some examples of complex logic:

  • Scrape some details from a listing website, search with the listing name in another site, combine this data and save it to a database
  • Take a list of keywords, perform a search in google maps for each keyword, go to the result – extract contact details, repeat the same process on yelp and a few other websites, then finally combine all this data

Should you use a visual web scraping tool?

Visual web scraping tools are pretty good at extracting data from simple websites but they are not great at performing complex tasks or handle complex websites. We are yet to find an open source visual web scraping tool that can handle complex logic.

Visual tools are easy to get started with, but once you hit a wall, there isn’t much you can do. We recommend that you use visual tools for extracting data from websites which are not too complicated or if your scraping logic is complex. If it is a bit complex, you are better off building a scraper from scratch using a programming language like Python.

Which programming language is the best for building web scrapers?

Another common question we usually hear from our readers at ScrapeHero is – Which programming language should we use for building web scrapers?

We recommend that you use Python. The most popular web scraping framework, Scrapy is built using Python. It also has the most number of web scraping frameworks and is excellent for parsing and processing data.

You could use tools like Selenium with Python for scraping most of the modern websites built using Javascript frameworks like React, Angular or VueJS. There is also a massive community of Python developers if you are in need of better talent.

Running web scrapers at scale

There is a massive difference between writing and running one scraper that scrapes 100 pages to a large scale distributed scraping infrastructure that can scrape thousands of websites or millions of pages a day.

Did you know?

ScrapeHero’s infrastructure can scrape about 3000 pages per second, but we never use it all on a single website. Be nice to websites you scrape without putting a heavy load on their servers and CDNs

Distributed Web Scraping

For scraping millions of pages daily, you will need a few servers and a way to distribute your scrapers across them and have them communicate with each other. Here are some components you would need to make this happen –

URL Queue and a Data Queue – using Message Broker like Redis, RabbitMQ or Kafka to distribute URLs and data across the scrapers that are running in different servers. You can design scrapers to read URLs from a queue in the broker, scrape them and put the extracted data into another queue and feed newly discovered URLs into the URL Queue. Another process would read from the data queue and write it to a database while the scraper is running. You could skip this step and write directly to the database from the scraper if you are not writing a lot of data.

You would also need to some process manager to make sure your scraper scripts restart automatically, if they are killed for some reason while parsing data.

Frameworks like Scrapy – Redis, and PySpider lets you skip a lot of the above tasks.

Scheduling scrapers for periodic data collection

If you need to refresh data periodically, you can either run it manually or automate it using some tool. If you are using Scrapy, scrapyd + cron can schedule the spiders for you, and it will update the data the way you need it.  PySpider also has a similar interface to do this.

Databases to store a large number of records

Once you have this massive data trove, you need a place to save it. We would suggest using a NoSQL database like MongoDB, Cassandra or HBase to store this data, depending upon the frequency and speed of scraping.

You can then extract this data from this database/datastore and integrate it with your business process. But before you do that, you should set up some reliable Quality Assurance tests for your data (more on that later)

Getting Past Anti Scraping Tools – IP Rotation, Proxies

Large scale scraping comes with a multitude of problems and anti-scraping tools and techniques are one of the biggest. There is a considerable number of Bot Mitigation & Screen Scraping Protection Tools aka anti-scraping tools like Akamai, Distill Networks, Shield Square, Perimeter X, etc. that block your scrapers from accessing websites usually by an IP Ban. 

If any of the websites you need have any IP based blocking involved, your servers’ IP address will be blacklisted in no time and the site will not respond to requests from your servers or show you a captcha, leaving you with very few options after getting blacklisted.

As we are talking about millions of requests. You might need some of these

  • Rotate the requests you make through 1000+ private proxies if you are not dealing with a captcha or anti-scraping service
  • Make requests through a provider with 100,000+ geo proxies that are not completely blacklisted when dealing with most anti-scraping solutions
  • Or time and resources to reverse engineer and bypass the anti-scraping solutions

Data Validation and Quality Control

The data you scrape is only as good as its quality. To ensure the data that you scraped inaccurate and complete, you need to run a variety of Quality assurance tests on it right after it is scraped. You need to validate every field of every record before you save it or before you process it for your use.

Having a set of Tests for the integrity of the data is essential. You can automate some of it by using Regular Expressions, to check if the data follows a predefined pattern and if it doesn’t then generate some alerts so that it can be manually inspected.

There are tools like Pandas, Cerebrus, Schema, etc built on Python, to help you validate the data record by record.

If the scraper is part of a data pipeline, you can set up multiple stages in the pipeline to verify the integrity and quality of the data using ETL Tools.

Maintenance

Scrapers

Every website will change its structure now and then, and so should your scrapers. Scrapers usually need adjustments every few months or even weeks, as a minor change in the target website affecting the fields you scrape might either give you incomplete data or even crash the scraper, depending on the logic of your scraper.

You need a mechanism to alert you if a lot the data you scrape is invalid or empty all of a sudden, so that you may check the scrapers and the website for changes that triggered it. You need to fix the scraper once it is broken either manually or by building some sophisticated logic to repair itself quickly to prevent disruptions in your data pipeline.

Database and Storage

If you are scraping a lot of data – you will need a lot of storage to store the data and it is best to plan this up front. Small amount of data can be stored in flat files or spreadsheets but when the data becomes too large for a spreadsheet to handle, you need to evaluate storage options including Cloud Storage and Cloud Hosted Databases (S3, Azure SQL, Azure Postgres, Redshift, Aurora, RDS, DynamoDB, Redis) or Relational Databases (MySQL, Oracle, SQL Server) or NoSQL databases (MongoDB, Cassandra etc).

Depending upon the size of data, you need to clean up your database of outdated and invalid data to save space and money. You might also want to scale up your systems if you still need the old data. Sharding and Replication of databases can be of help.

You need to make sure that you do not keep anything that is not of use after the scraping is complete.

Know when to ask for help

As you might have realized by now, this whole process is expensive and time-consuming, and you need to be ready to take on this challenge.

You also need to know when to stop and ask for help. ScrapeHero has been doing all this and more for many years now, so let us know if you need any help.

Or you can just worry about what to do with the data, we deliver to you. Interested ?

Turn the Internet into meaningful, structured and usable data



Please DO NOT contact us for any help with our Tutorials and Code using this form or by calling us, instead please add a comment to the bottom of the tutorial page for help

Posted in:   Scraping Tips, Web Scraping Tutorials

Responses

Comments or Questions?

Turn the Internet into meaningful, structured and usable data