Web scraping is a task that has to be performed responsibly so that it is does not have a detrimental effect on the sites being scraped. Web Crawlers can retrieve data much quicker, in greater depth than humans, so bad scraping practices can have some impact on the performance of the site.
If a crawler performs multiple requests per second and downloads large files, an under-powered server would have a hard time keeping up with requests from multiple crawlers. Since web crawlers, scrapers or spiders (words used interchangeably) don’t really drive human website traffic and seemingly affect the performance of the site, some site administrators do not like spiders and try to block their access. Most websites may not have anti scraping mechanisms since it would affect the user experience, but some sites do block scraping because they do not believe in open data access.
In this article, we will talk about how websites detect and block spiders and techniques to overcome those barriers.
How can websites detect web scraping?
Websites can use different mechanisms to detect a scraper/spider from a normal user. Some of these methods are enumerated below:
- Unusual traffic/high download rate especially from a single client/or IP address within a short time span.
- Repetitive tasks performed on the website – based on an assumption that a human user won’t perform the same repetitive tasks all the time.
- Detection through honeypots – these honeypots are usually links which aren’t visible to a normal user but only to a spider. When a scraper/spider tries to access the link, the alarms are tripped.
How to easily know if a site doesn’t support data scraping?
Check the robots.txt file, which is usually in the root directory of a website e.g http://example.com/robots.txt.
If it contains lines like the ones shown below, it means the site doesn’t like and want scraping.
However, since most sites want to be on Google (arguably the largest scraper of websites globally ;-)) they do allow access to bots and spiders.
This line is for preventing well-behaved bots or the bots which respect robots.txt.
What happens when you get banned ?
There are two simple ways to ban a webspider/webscraper – either by banning all accesses from a particular IP or by banning all accesses that use a specific id to access the server (most browsers and web spiders identify themselves whenever they request a page by user agents. Chrome browser for example uses
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36
The banning can be temporary or permanent.
Temporary blocks can last minutes or hours.
Permanent bans go against the open nature of the Internet but some sites resort to this “scorch the internet” measure.
How do you find out if a website has blocked you ?
If any of the following signs appear on the site that you are crawling, it is usually a sign of being blocked or banned.
- CAPTCHA pages
- Unusual content delivery delays
- Frequent response with HTTP 404, 301 or 50x errors
Frequent appearance of these HTTP status codes is also indication of blocking
- 301 Moved Temporarily
- 401 Unauthorized
- 403 Forbidden
- 404 Not Found
- 408 Request Timeout
- 429 Too Many Requests
- 503 Service Unavailable
A comprehensive list of HTTP return codes (successes and failures) can be found here. It will be worth your time to read through these codes and be familiar with them.
Web Crawling Best Practices
Basic Rule: “Be Nice”
An overarching rule to keep in mind for any kind of web scraping is
BE GOOD AND FOLLOW A WEBSITE’S CRAWLING POLICIES
Here are some of the best practices you can follow to overcome the detection.
1. Make the crawling slower, do not slam the server, treat websites nicely
Use auto throttling mechanisms which will automatically throttle the crawling speed based on the load on both the spider and the website that you are crawling. Adjust the spider to an optimum crawling speed after a few trial runs and do this periodically because the environment does change over time.
The faster you crawl, the worse it is for everyone.
Put some random programmatic sleep calls in between requests, add some delays after crawling a small number of pages and choose the lowest number of concurrent requests possible.
These techniques make the spider look like a human and are generally good for everyone.
2. Disguise your requests by rotating IPs and Proxy Services
A server can easily detect a bot by checking the requests from a single IP address, so if you use different IP addresses for making requests to a server, the detection becomes harder. Create a pool of IPs that you can use and use random ones for each request.
There are several methods can be used to change your outgoing IP.
Services such as VPNs, shared proxies and TOR can help. In addition, various commercial providers also provide services for automatic IP rotation.
This technique also distributes the load across various exit points.
Some websites have blocked well known IP ranges such as AWS completely to prevent using the massive Amazon AWS IPs. Such protectionist policies are obviously counter to the open nature of the Internet.
3. User Agent Rotation and Spoofing
Every request made from a web browser contains a user-agent header and using the same user-agent consistently leads to the detection of a bot.
User Agent rotation and spoofing is the best solution for this.
Spoof the User Agent by creating a list of user agents and picking a random one for each request.
Websites do not want to block genuine users so you should try to look like one. Set your user-agent to a common web browser instead of using the default user-agent (such as wget/version or urllib/version). You could even pretend to be the Google Bot: Googlebot/2.1 if you want to have some fun! (http://www.google.com/bot.html)
You can check your user-agent string here:
A user-agent string listing to get you started can be found here:
4. Beware of Honey Pot Traps
Some websites install honeypots to detect web spiders. These honeypots usually are links that normal user can’t see but a spider can.
When following links always take care that the link has proper visibility with no nofollow tag. Some honeypot links to detect spiders will be have the CSS style display:none or will be color disguised to blend in with the page’s background color.
This detection is obviously not easy and requires a significant amount of programming work to accomplish properly, as a result this technique is not widely used on either side – the server side or the bot or scraper side.
5. Do not follow the same crawling pattern
Only robots follow the same crawling pattern because unless specified otherwise, programmed bots follow a logic which is usually very specific.
Sites that have intelligent anti-crawling mechanisms can easily detect spiders from finding pattern in their actions. Humans generally will not perform repetitive tasks.
Incorporate some random clicks on the page, mouse movements and random actions that will make a spider looks like a human.
6. Always respect the robots.txt file
Web spiders should ideally follow rules in a robots.txt file for the website being scraped. The robots.txt file specifies rules for good behavior, such as how frequently bots are allowed to request pages, what pages are allowed to be scraped and which areas are off limits for scraping.
However, some websites are extremely liberal with letting Google scrape their websites but not allowing any other bots access. This goes against the open nature of the Internet, but the websites owners are well within their rights to resort to such behavior.
[Update July 11 2017: Added new item below]
7. Use a headless browser
The solution for this is to use headless browsers (browsers that aren’t really visual on a desktop but are fully functional otherwise). Selenium, PhantomJS, and the latest entrant – Google’s own headless chrome are some options to explore further.
Keep in mind that headless browsers like Selenium and Puppeteer use a lot of resources (RAM, CPU, Bandwidth etc) in comparison to script-based approaches. You can check out these web scraping tutorials on using headless browsers:
All these ideas above provide a starting point for you to build your own solutions or refine your existing solution. If you have any ideas or suggestions, please join the discussion in the comments section.
Thank you for reading.
Or you can ignore everything above, and just get the data delivered to you as a service. Interested ?
Turn the Internet into meaningful, structured and usable data