Comparison and Use Cases of popular python frameworks and libraries used for webs scraping like - Scrapy,Urllib, Requests, Selenium, Beautifulsoup and LXML
Open Source has fueled a massive part of the technology boom we are all experiencing. Even in the world of web scraping, open source web scraping tools play a large part to help gather data from the internet. We will walk through open source web scraping frameworks and tools that are great for crawling, scraping the web, and parsing out the data.
Here is a comparison chart showing the important features of all the best open source web scraper frameworks and tools that we will go through in this post:
Features/Tools | GitHub Stars | GitHub Forks | GitHub Open Issues | Last Updated | Documentation | License |
---|---|---|---|---|---|---|
Puppeteer | 84.5k | 9.1k | 289 | September 2023 | Excellent | Apache-2.0 |
Scrapy | 48.4k | 10.1k | 478 | September 2023 | Excellent | BSD-3-Clause |
Selenium | 27.7k | 7.8k | 180 | September 2023 | Good | Apache-2.0 |
PySpider | 16k | 3.7k | 272 | August 2020 | Good | Apache-2.0 |
Crawlee | 9k | 400 | 79 | June 2020 | September 2023 | Apache-2.0 |
NodeCrawler | 6.5k | 913 | 35 | December 2022 | Good | MIT |
MechanicalSoup | 4.4k | 397 | 28 | July 2023 | Average | MIT |
Heritrix | 2.5k | 754 | 35 | May 2020 | Good | Apache-2.0 |
Apache Nutch | 2.7k | 1.2k | – | August 2023 | Excellent | Apache-2.0 |
StormCrawler | 819 | 248 | 44 | September 2023 | Good | Apache-2.0 |
Note: Data as on September 2023
These are the best Open Source web scraper tools available in each language or platform :
Puppeteer
Puppeteer is a Node library which provides a powerful but simple API that allows you to control Google’s headless Chrome browser. A headless browser means you have a browser that can send and receive requests but has no GUI. It works in the background, performing actions as instructed by an API. You can truly simulate the user experience, typing where they type and clicking where they click.
The best case to use Puppeteer for web scraping is if the information you want is generated using a combination of API data and JavaScript code. A headless browser is a great tool for automated testing and server environments where you don’t need a visible UI shell. For example, you may want to run some tests against a real web page, create a PDF of it, or just inspect how the browser renders a URL. Puppeteer can also be used to take screenshots of web pages visible by default when you open a web browser. Puppeteer’s API is very similar to Selenium WebDriver, but works only with Google Chrome, while WebDriver works with most popular browsers. Puppeteer has a more active support than Selenium, so if you are working with Chrome, Puppeteer is your best option for web scraping.
Requires Version – Node v6.4.0, Node v7.6.0 or greater
Available Selectors – CSS
Available Data Formats – JSON
Pros
- With its full-featured API, it covers a majority of use cases
- The best option for scraping JavaScript websites on Chrome
Cons
- Only available for Chrome
- Supports only JSON format
Scrapy

Scrapy is an open source web scraping framework in Python used to build web scrapers. It gives you all the tools you need to efficiently extract data from websites, process them as you want, and store them in your preferred structure and format. One of its main advantages is that it’s built on top of a Twisted asynchronous networking framework. If you have a large web scraping project and want to make it as efficient as possible with a lot of flexibility then you should definitely use Scrapy.
Scrapy is under BSD license.
Requires Version – Python 2.7, 3.4+
Available Selectors – CSS, XPath
Available Data Formats – CSV, JSON, XML
Pros
- Suitable for broad crawling
- Easy setup and detailed documentation
- Active community
Cons
- Since it is a full-fledged framework, it is not beginner friendly
- Does not handle JavaScript
Selenium WebDriver
When it comes to websites that use very complex and dynamic code, it’s better to have all the page content rendered using a browser first. Selenium WebDriver uses a real web browser to access the website, so it would like its activity wouldn’t look any different from a real person accessing information in the same way. When you load a page using Web Driver, the browser loads all the web resources and executes the javaScript on the page. At the same time, it stores all the cookies created by websites and sends complete HTTP headers as all browsers do. This makes it very hard to determine whether a real person accesses the website or if its a bot.
Although it’s mostly used for testing, WebDriver can be used for scraping dynamic web pages. It is the right solution if you want to test if a website works properly with various browsers or JavaScript-heavy websites. Using WebDriver makes web scraping easier, but the scraping process is much slower as compared to simple HTTP request to the web browser. When you are using the WebDriver, the browser waits until the whole page is loaded and then can you only access the elements. Selenium has a very large and active community which is great for beginners.
Requires Version – Python 2.7 and 3.5+ and provides bindings for languages JavaScript, Java, C, Ruby, and Python.
Available Selectors – CSS, XPath
Available Data Formats – Customizable
Pros
- Suitable for scraping heavy JavaScript websites
- Large and active community
- Detailed documentation, making it easy to grasp for beginners
Cons
- Hard to maintain when there are any changes in the website structure
- High CPU and memory usage
PySpider
PySpider is a web crawler written in Python. It supports JavaScript pages and has a distributed architecture. This way you can have multiple crawlers. PySpider can store the data on a backend of your choosing such as MongoDB, MySQL, Redis, etc. You can use RabbitMQ, Beanstalk, and Redis as message queues.
One of the advantages of PySpider its easy to use UI where you can edit scripts, monitor ongoing tasks and view results. If you are working with a website-based user interface, PySpider is the internet scrape to consider. It also supports AJAX heavy websites. To know more about PySpider, you can check out their documentation and or their community resources. It’s currently licensed under Apache License 2.0.
Requires Version – Python 2.6+, Python 3.3+
Available Selectors – CSS, XPath
Available Data Formats – CSV, JSON
Pros
- Facilitates more comfortable and faster scraping
- Powerful UI
Cons
- Difficult to deploy
Crawlee
Crawlee, is the successor to Apify SDK which is a Node.js library and positions itself as a universal web scraping library in JavaScript, with support for Puppeteer, Cheerio, and more. Crawlee has full TypeScript support, anti-blocking features, and a similar Apify SDK interface. It also retains all the web crawling and scraping-related tools.
Requirements – Crawlee requires Node.js 16 or higher
Available Selectors – CSS
Available Data Formats – JSON, JSONL, CSV, XML, Excel or HTML
Pros
- Runs on Node.js and it’s built in TypeScript to improve code completion
- Automatic scaling and proxy management
- Mimic browser headers and TLS fingerprints
Cons
- Single Actors (scrapers) occasionally break causing delays in data scraping
- The interface is a bit difficult to navigate, especially for new users
Installation
Add Crawlee to any Node.js project by running:
npm install crawlee
Best Use Case
If you need a better developer experience and powerful anti-blocking features.
NodeCrawler
NodeCrawler is a popular web crawler for NodeJS, making it a very fast crawling solution. If you prefer coding in JavaScript, or you are dealing with mostly a JavaScript project, NodeCrawler will be the most suitable web crawler to use. Its installation is pretty simple too. JSDOM and Cheerio (used for HTML parsing) use it for server-side rendering, with JSDOM being more robust.
Requires Version – Node v4.0.0 or greater
Available Selectors – CSS, XPath
Available Data Formats – CSV, JSON, XML
Pros
- Easy installation
Cons
- It has no Promise support
MechanicalSoup
MechanicalSoup is a Python library that is designed to simulate the behavior of a human using a web browser and built around the parsing library BeautifulSoup. If you need to scrape data from simple sites or if heavy scraping is not required, using MechanicalSoup is a simple and efficient method. MechanicalSoup automatically stores and sends cookies, follows redirects and can follow links and submit forms.
It’s best to use MechanicalSoup when interacting with a website that doesn’t provide a web service API, out of a browser. If the website provides a web service API, then you should use this API and you don’t need MechanicalSoup. If the website relies on JavaScript, then you probably need a fully-fledged browser, like Selenium. MechanicalSoup is licensed under MIT.
Requires Version – Python 3.0+
Available Selectors – CSS, XPath
Available Data Formats – CSV, JSON, XML
Pros
- Preferred for fairly simple websites
Cons
- Does not handle JavaScript
Apache Nutch

Requirements – Java 8
Available Selectors – XPath, CSS
Available Data Formats – JSON, CSV, XML
Pros
- Highly extensible and flexible system
- Open source web-search software, built on Lucene Java
- Dynamically scalable with Hadoop
Cons
- Difficult to setup
- Poor documentation
- Some operations take longer, as the size of crawler grows
Heritirix

Requires Versions – Java 5.0+
Available Selectors – XPath, CSS
Available Data Formats – ARC file
Pros
- Excellent user documentation and easy setup
- Mature and stable platform
- Good performance and decent support for distributed crawls
- Respects robot.txt
- Supports broad and focused crawls
Cons
- Not dynamically scalable
StormCrawler
StormCrawler is a library and collection of resources that developers can leverage to build their own crawlers. The framework is based on the stream processing framework Apache Storm and all operations occur at the same time such as – URLs being fetched, parsed, and indexed constantly – which makes the whole crawling process more efficient. It comes with modules for commonly used projects such as Apache Solr, Elasticsearch, MySQL, or Apache Tika and has a range of extensible functionalities to do data extraction with XPath, sitemaps, URL filtering or language identification.
Requirements – Apache Maven, Java 7
Available Selectors – XPath
Available Data Formats – JSON, CSV, XML
Pros
- Appropriate for large scale recursive crawls
- Suitable for low latency web crawling
Cons
- Does not support document deduplication
Wrapping Up
These are just some of the open source web scraping tools and frameworks you can use for your web scraping projects. If you have greater scraping requirements or would like to scrape on a much larger scale it’s better to use web scraping services.
If you aren’t proficient with programming or your needs are complex, or you need large volumes of data to be scraped, there are great web scraping services that will suit your requirements to make the job easier for you.
You can save time and get clean, structured data by trying us out instead – we are a full-service provider that doesn’t require the use of any tools and all you get is clean data without any hassles.
Turn the Internet into meaningful, structured and usable data
We can help with your data or automation needs
Responses