How To Scrape eBay using Python and LXML

In this article, we will show you how to scrape eBay and extract data such as prices and names of products in all categories by a brand. Web scraping eBay can help with collecting information for eBay keyword monitoring, price monitoring, brand monitoring and pricing intelligence. Scraping eBay listings at regular intervals can be useful to check the details of products and compare them with your competitor sites.

Here are the steps on how to scrape eBay

1. Construct the URL for the search results to scrape eBay.
   Example – https://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=iphone+7&_blrs=recall_filtering
2. Download HTML of the search result page using Python Requests.
3. Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths.
4. Save the scraped eBay product data to a CSV file.

Below is a screenshot of the data to extract using our eBay scraping tutorial.

You could also scrape details such as the number of products sold or the ratings given by consumers, but for now, we will keep this eBay scraper simple and extract these.

Requirements for eBay Scraping

Install Python 3 and Pip

Here is a guide to install Python 3 in Linux – http://docs.python-guide.org/en/latest/starting/install3/linux/

Mac Users can follow this guide – http://docs.python-guide.org/en/latest/starting/install3/osx/

Windows Users go here – https://www.scrapehero.com/how-to-install-python3-in-windows-10/

Packages

For this eBay scraping tutorial using Python 3, we will need some packages for downloading and parsing the HTML. Below are the package requirements:

• PIP to install the following packages in Python (https://pip.pypa.io/en/stable/installing/ )
• Python Requests, to make requests and download the HTML content of the pages ( http://docs.python-requests.org/en/master/user/install/).
• Python LXML, for parsing the HTML Tree Structure using Xpaths ( Learn how to install that here – http://lxml.de/installation.html )

The Code

Since we will be monitoring prices by their brand, here is the one for Apple – https://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=iphone+7&_blrs=recall_filtering

https://gist.github.com/scrapehero/2a1be61eb28cfa577e379e2b69b31c90

You can download the link at https://gist.github.com/scrapehero/2a1be61eb28cfa577e379e2b69b31c90 if the embed above does not work.

If you would like the code in Python 2.7, you can check out the link at https://gist.github.com/scrapehero/352052515da5e2511d5a31a5eb2786da

Running the Python eBay Scraper

We have named the script ebay_scraper.py. If you type in the script name in terminal or command prompt with a -h

usage: ebay_scraper.py [-h] brand

positional arguments:

  brand       Brand Name

optional arguments:

  -h, --help  show this help message and exit

The brand argument represents any brand available on eBay. You can type in a brand that eBay currently has on its site such as – Samsung, Canon, Dell, etc. The script must be run with the argument for brand. As an example, to find all of the products Apple currently has on eBay, we would run the scraper like this. 

python3 ebay_scraper.py apple

In this article we are only scraping the product’s name, price, and URL from the first page of results, so a CSV file should be enough to fit in all the data. If you would like to extract details in bulk, a JSON file is more preferable. You can read about choosing your data format, just to be sure.

This will create a CSV file named apple-ebay-scraped-data.csv that will be in the same folder as the script. Here are some of the data extracted from eBay in a CSV file from the command above.

You can download the code at https://gist.github.com/scrapehero/2a1be61eb28cfa577e379e2b69b31c90

We would love to know how this scraper worked for you. Let us know in the comments below.

Why Scrape eBay?

eBay scraping using the code above can be useful for a number of reason. Here are a few reasons on how web scraping eBay can be useful –

1. eBay Keyword monitoring – You can easily monitor eBay for a specific keywords using this tutorial.
2. Brand monitoring – You can replace the search term in this tutorial to include a brand name and easily monitor which brand products are being sold more often on eBay.
3. Price monitoring – eBay is one of the largest marketplaces in the world, scraping eBay for price and comparing with the Amazon scraper and Walmart scraper pricing data can help create an efficient price monitoring system.

Known Limitations to eBay Scraping

This code should be able to scrape eBay prices and details of most brands available. If you want to scrape and extract the details of thousands of products for each brand and check the competitor prices of products periodically (on an hourly basis), then you should read  Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping

If you need some professional help with scraping complex websites you can fill up the form below.