How To Scrape Competitor Prices from using Python and LXML

Prices on products can fluctuate indiscriminately. Frequently monitoring pricing data can help you adjust your prices and infer which products are popular and in demand. Web scraping is a profitable technique that can help with collecting information for businesses and pricing intelligence.

For this article, we will show you how to extract the names and prices of products in all categories by a brand available on eBay. Scraping data from at regular intervals can be useful to check the details of products and compare them with your competitor sites.

Below is a screenshot of the data we will be extracting.


You could also scrape details such as the number of products sold or the ratings given by consumers, but for now, we will keep it simple and scrape these.

Scraping Logic

  1. First, construct the URL for the search results from eBay. Since we will be monitoring prices by their brand, here is the one for Apple-
  2. Download HTML of the search result page using Python Requests.
  3. Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths. We have predefined the XPaths for the details we need in the code.
  4. Save the data to a CSV file. In this article we are only scraping the product’s name, price, and URL from the first page of results, so a CSV file should be enough to fit in all the data. If you would like to extract details in bulk, a JSON file is more preferable. You can read about choosing your data format, just to be sure.


Install Python 3 and Pip

Here is a guide to install Python 3 in Linux –

Mac Users can follow this guide –

Windows Users go here –


For this web scraping tutorial using Python 3, we will need some packages for downloading and parsing the HTML. Below are the package requirements:

The Code

You can download the link at if the embed above does not work.

If you would like the code in Python 2.7, you can check out the link at

Running the Scraper

We have named the script If you type in the script name in terminal or command prompt with a -h


The brand argument represents any brand available on eBay. You can type in a brand that eBay currently has on their site such as- Samsung, Canon, Dell, etc. The script must be run with the argument for brand. As an example, to find all of the products Apple currently has on eBay, we would run the scraper like this. 

python3 apple

This will create a CSV file named apple-ebay-scraped-data.csv that will be in the same folder as the script. Here are some of the data extracted from eBay in a CSV file from the command above.


You can download the code at

We would love to know how this scraper worked for you. Let us know in the comments below.

Known Limitations

This code should be able to scrape the details of most brands available on eBay. If you want to scrape the details of thousands of products for each brand and check the prices of products periodically (on an hourly basis), then you should read  Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping

Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.

If you need some professional help with scraping complex websites you can fill up the form below.

Do you need some prices monitored ?

We help business monitor prices across e-Commerce websites by collecting data


2 comments on “How To Scrape Competitor Prices from using Python and LXML


i m a newbie in python trying to install it steps by steps, however i m getting error msg like below

/usr/local/lib/python3.6/site-packages/requests/ RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn’t match a supported version!
Traceback (most recent call last):
File “”, line 4, in
import unicodecsv as csv
ModuleNotFoundError: No module named ‘unicodecsv’

grateful if you can give some advice, thanks!

Join the conversation

Turn websites into meaningful and structured data through our web data extraction service