Python Frameworks and Libraries for Web Scraping

Web scraping is the easiest and fastest method to extract data from the internet. With so many web scraping tutorials and guides available out there on so many frameworks, programming languages, and libraries, it could be quite confusing to pick one for your web scraping needs. In this article, we will take a look at some of the top python web scraping frameworks and libraries.

  1. Scrapy – The complete framework
  2. Urllib
  3. Python Requests
  4. Selenium
  5. Beautifulsoup
  6. LXML

Scrapy – The complete web scraping framework

Scrapy is an open source web scraping framework written in Python which takes care of everything from downloading HTML if web pages to storing them in the form you want. For those of you who are familiar with Django, Scrapy is a lot similar to it. The requests we make on Scrapy are scheduled and processed asynchronously. This is because it is built on top of Twisted, an asynchronous framework.

What is asynchronous?

For those of you, who aren’t familiar, let’s take a case. Assume you have to make 100 phone calls. Now, what would you do? Sit down next to the phone and dial up the first number. Wait for the response, process the call and then move to next. This is how it is with conventional web scraping methods. With Scrapy you can dial up 40 numbers and process each call as and when it receives a response. No time is wasted waiting. This becomes extremely important when your scraping needs are large.


  • The CPU usage is a lot lesser
  • Consumes lesser memory
  • Extremely efficient in comparison
  • The well-designed architecture offers you both robustness and flexibility.
  • You can easily develop custom middle-ware or pipeline to add custom functionalities


  • Overkill for simple jobs
  • Might be difficult to install.
  • The learning curve is quite steep.
  • Not very beginner-friendly, since it is a full-fledged framework


To install Scrapy using conda run:

conda install c condaforge scrapy

Alternatively, if you are more familiar with the installation from PyPI, you can install using pip as :

pip install Scrapy

Note that sometimes this may require solving compilation issues for some Scrapy dependencies depending on your operating system, so be sure to check the Platform specific installation notes.

Best Use Case

Scrapy is best if you need to build a real spider or web-crawler for large web scraping needs, instead of just scraping a few pages here and there. It can offer extensibility and flexibility to your project. 


As the official docs say, urllib is a python web scraping library with several modules for working with URLs (Uniform Resource Locators). It also offers a slightly more complex interface for handling common situations – like basic authentication, encoding, cookies, proxies, and so on. These are provided by objects called handlers and openers.

  • urllib.request for opening and reading URLs
  • urllib.error containing the exceptions raised by urllib.request
  • urllib.parse for parsing URLs
  • urllib.robotparser for parsing robots.txt files


  • Included in python standard library
  • It defines functions and classes to help with URL actions (basic and digest authentication, redirections, cookies, etc)


  • Unlike Requests, while using urllib you will need to use the method urllib.encode() to encode the parameters before passing them
  • Complicated when compared to Python Requests


Urllib is already included with your python installation, you don’t need to install it.

Best Use Case

  • If you need advanced control over the requests you make

Requests – HTTP for humans

Requests is a python library. It is the perfect example how beautiful an API can be with the right level of abstraction. The Requests library allows you to send organic, grass-fed requests, without the need for manual labor.


  • Easier and shorter codes than Urllib.
  • Thread-safe.
  • Multipart File Uploads & Connection Timeouts
  • Elegant Key/Value Cookies & Sessions with Cookie Persistence
  • Automatic Decompression
  • Basic/Digest Authentication
  • Browser-style SSL Verification
  • Keep-Alive & Connection Pooling
  • Good Documentation
  • No need to manually add query strings to your URLs
  • Supports the entire restful API, i.e., all its methods – PUT, GET, DELETE, POST.


  • If your web page has javascript hiding or loading content, then requests might not be the way to go.


You can install requests using conda as :

conda install -c anaconda requests

and pip using:

pip install requests.

Best Use Case

  • If you are a beginner, and your scraping task is simple and contains no javascript elements

Selenium – The Automator

Selenium is a library that automates browsers based on Java. But you can access it via the Python Package Selenium. Though primarily used as a tool for writing automated tests for web applications, it has come to some heavy use for pages that have javascript on them.


  • Beginner-friendly
  • You get the real browser to see what’s going on ( unless you are on a headless mode )
  • Mimics human behavior while browsing, including clicks, selection, filling text box and scrolling
  • Renders a full webpage and shows HTML rendered via XHR or Javascript


  • Very slow
  • Heavy memory use
  • High CPU usage


To install this package with conda run:

conda install -c conda-forge selenium 

Using pip, you can install by running the below on your terminal

pip install selenium

But you will need to install  Selenium Web Driver or geckodriver for Firefox browser interface. Failure of the same results in errors. See more of the installation instructions here.

Best Use Case

  •  When you need to scrape sites with data tucked away by JavaScript.

The Parsers

  1. Beautiful Soup
  2. LXML

Now that we have got our required HTML content, the job is to go through them and get the data. As well explained here, regular expressions can be used to extract data from an HTML document. It is almost never the best way to create maintainable code. With a regex, you are simply parsing text with no structure and thus you are more likely to come across errors. Why bother so when HTML in itself is presenting text in a given structure. Instead of a text match and regex, we could just parse. If writing maintainable code, it’s best to use parsers.

What are Parsers?

A parser is simply a program that can extract data from HTML and XML documents. They parse the structure into memory and felicitates the use of selectors (either CSS or XPath) to easily extract the data. The advantage here is that parsers can automatically correct “bad” HTML (unclosed tags, badly quoted attributes, invalid HTML etc) and allow us to get the data we need. The disadvantage is that it requires more processor work in most cases, but as ever it’s a trade-off and tends to be a worthwhile one.


Beautiful Soup (BS4) is a python library that can parse data. As we already know, the content we are parsing might belong to various encodings. BS4 automatically detects it. BS4 creates a parse tree which helps you navigate a parsed document easily and find what you need.


  • Easier to write a BS4 snippet than LXML.
  • Small learning curve, easy to learn.
  • Quite robust.
  • Handles malformed markup well.
  • Excellent support for encoding detection


  • If the default parser chosen for you is incorrect, they may incorrectly parse results without warnings, which can lead to disastrous results.
  • Projects built using BS4 might not be flexible in terms of extensibility.
  • You need to import multiprocessing  to make it run quicker


To install this package with conda run:
conda install -c anaconda beautifulsoup4 

Using pip, you can install using

pip install beautifulsoup4

Best Use Case

  • When you are a beginner to web scraping.
  • If you need to handle messy documents, choose Beautiful Soup.


Lxml is a high-performance, production-quality HTML and XML parsing library.


  • It is the best performing parser. As shown here
  • Most feature-rich python library for the same.


  • The official documentation isn’t that friendly, so beginners are better off starting somewhere else.


With conda run, you can install using,

To install this package with conda run:
conda install -c anaconda lxml 

You can install using LXML directly using pip,

sudo apt-get install python3-lxml

Best Use Case

  • If you need speed, go for lxml.

Our recommendations

If your web scraping needs are simple, then any of the above tools might be easy to pick up and implement. For smaller data requirements there are free web scraping tools you could try out that do not need much coding skills and are cost-effective. We have a post on free web scraping tools that can help you decide.

But when you have a large amount of data that needs to be scraped consistently, especially over pages that might even change it’s structure and links, doing it on your own might be too much of an effort.

You might want to look at Scalable do-it-yourself scraping & How to build and run scrapers on a large scale

If you are looking for some professional help with scraping complex websites, let us know by filling up the form below.

Tell us about your complex web scraping projects

Turn the Internet into meaningful, structured and usable data

Please DO NOT contact us for any help with our Tutorials and Code using this form or by calling us, instead please add a comment to the bottom of the tutorial page for help


Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.

Posted in:   Developers, Scraping Tips, Web Scraping Tools


ScrapeHero October 4, 2018


Comments are closed.

Turn the Internet into meaningful, structured and usable data   

ScrapeHero Logo

Can we help you get some data?