Web Scraping Job Posts from Glassdoor Using Python and LXML

Aggregating job postings from the web is difficult as its time consuming to manually extract data from websites. Web scraping is the best source for job data feeds if you are looking for jobs in a city or within a specific salary range.

This tutorial is about extracting details of jobs listings based on a particular job name and location. You can scrape the estimated salary, job ratings, or go a bit further and scrape the jobs based on the number of miles from a particular city. With web scraping, can find job listings over a certain time period and identify when job postings are listed and removed to make an analysis on jobs that are trending.

In this tutorial, we will scrape Glassdoor.com, one of the fastest growing job recruiting sites. The scraper will extract the data fields for a particular job name in a location given.

Here is the list of fields that we will be extracting:

  1. Job Name
  2. Company
  3. State
  4. City
  5. Salary
  6. URL

Here is a screenshot of the data we will be extracting


Scraping Logic

  1. First, construct the URL for the search results from Glassdoor. Since we will be extracting listings by their job name and location, here are the listings to find Android developer in Boston, Massachusetts-https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=Android+Developer&sc.keyword=Android+Developer&locT=C&locId=1154532&jobType=
  2. Download HTML of the search result page using Python Requests.
  3. Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths. We have predefined the XPaths for the details we need in the code.
  4. Save the data to a CSV file. In this article, we are only scraping the job name, company, location and estimated salary from the first page of results, so a CSV file should be enough to fit in all the data. If you would like to extract details in bulk, a JSON file is more preferable. You can read about choosing your data format, just to be sure.


Install Python 3 and Pip

Here is a guide to install Python 3 in Linux – http://docs.python-guide.org/en/latest/starting/install3/linux/

Mac Users can follow this guide – http://docs.python-guide.org/en/latest/starting/install3/osx/

Windows Users go here – https://www.scrapehero.com/how-to-install-python3-in-windows-10/


For this web scraping tutorial using Python 3, we will need some packages for downloading and parsing the HTML. Below are the package requirements:

The Code

 If the embed above does not work you can download the link at https://gist.github.com/scrapehero/352286d0f9dee87990cd45c3f979e7cb 

Running the Scraper

The name of the script is glassdoor.py. If you type in the script name in terminal or command prompt with a -h

The argument “keyword” represents a keyword related to the job you are searching for and the argument “place” is used to find the desired job in a specific location. The example below shows how to run the script to find the list of Android developers in Boston:

As an example, to find the list of Android developers in Boston we would run the script like this:

This will create a CSV file named Android developer-Boston-job-results.csv that will be in the same folder as the script. Here is some of the data extracted from Glassdoor in a CSV file from the command above.

You can download the code at https://gist.github.com/scrapehero/352286d0f9dee87990cd45c3f979e7cb

Let us know in the comments how this scraper worked for you.

Known Limitations

This scraper should work for extracting most job listings on Glassdoor unless the website structure changes drastically. If you would like to scrape the details of thousands of pages at very short intervals, this scraper is probably not going to work for you. You should read  Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping.

If you need professional help with scraping complex websites, contact us by filling up the form below.

You can also get data delivered to you, as a Service from us. Interested?

Turn websites into meaningful and structured data through our web data extraction service

Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.

2 comments on “Web Scraping Job Posts from Glassdoor Using Python and LXML

Piotr Zieliński

Thanks for a great article! I am currently working on a project where I also need a date of posting a job offer. Could you advise me how can I scrape it ? Regards

Join the conversation

Turn websites into meaningful and structured data through our web data extraction service