Aggregating job listings from the web is difficult as its time consuming to manually extract data from websites. Web scraping is the best source for job data feeds if you are looking for jobs in a city or within a specific salary range.
This tutorial is about extracting details of jobs listings based on a particular job name and location. You can scrape the estimated salary, job ratings, or go a bit further and scrape the jobs based on the number of miles from a particular city. With web scraping, can find job listings over a certain time period and identify when job postings are listed and removed to make an analysis on jobs that are trending.
In this tutorial, we will scrape Glassdoor.com, one of the fastest growing job recruiting sites. The scraper will extract the data fields for a particular job name in a location given.
Here are the list of fields that we will be extracting:
- Job Name
Here is a screenshot of the data we will be extracting
- First, construct the URL for the search results from Glassdoor. Since we will be extracting listings by their job name and location, here are the listings to find Android developer in Boston, Massachusetts-https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=Android+Developer&sc.keyword=Android+Developer&locT=C&locId=1154532&jobType=
- Download HTML of the search result page using Python Requests.
- Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths. We have predefined the XPaths for the details we need in the code.
- Save the data to a CSV file. In this article, we are only scraping the job name, company, location and estimated salary from the first page of results, so a CSV file should be enough to fit in all the data. If you would like to extract details in bulk, a JSON file is more preferable. You can read about choosing your data format, just to be sure.
For this web scraping tutorial using Python, we will need some packages for downloading and parsing the HTML. Below are the package requirements.
- Python 2.7 ( https://www.python.org/downloads/ )
- PIP to install the following packages in Python (https://pip.pypa.io/en/stable/installing/ )
- Python Requests, to make requests and download the HTML content of the pages ( http://docs.python-requests.org/en/master/user/install/).
- Python LXML, for parsing the HTML Tree Structure using Xpaths ( Learn how to install that here – http://lxml.de/installation.html )
If the embed above does not work you can download the link here.
Running the Scraper
The name of the script is glassdoor.py. If you type in the script name in terminal or command prompt with a -h
usage: 1934_glassdoor.py [-h] keyword place positional arguments: keyword job name place job location optional arguments: -h, --help show this help message and exit
The argument “keyword” represents a keyword related to the job you are searching for and the argument “place” is used to find the desired job in a specific location. The example below shows how to run the script to find the list of Android developers in Boston:
As an example, to find the list of Android developers in Boston we would run the script like this:
python filename.py "Android developer" "Boston"
This will create a CSV file named Android developer-Boston-job-results.csv that will be in the same folder as the script. Here is some of the data extracted from Glassdoor in a CSV file from the command above.
You can download the code at https://gist.github.com/scrapehero/352286d0f9dee87990cd45c3f979e7cb
Let us know in the comments how this scraper worked for you.
This scraper should work for extracting most job listings on Glassdoor unless the website structure changes drastically. If you would like to scrape the details of thousands of pages at very short intervals, this scraper is probably not going to work for you. You should read Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping.
If you need professional help with scraping complex websites, contact us by filling up the form below.
You can also get data delivered to you, as a Service from us. Interested?