How to Scrape Flight Schedules and Prices from using Python and LXML

Gathering data regarding flights is a mammoth task when done manually. There are hundreds of thousands of combinations of airports, routes, timings and ever changing prices. Ticket prices tend to vary daily (or even hourly), and there are a large number of flights available per day. Web Scraping is one of the solutions to keep track of this data. You could scrape data for any combinations of airports, timings, and flights and use this information to monitor and analyze ticket prices, pricing trends, cheap routes etc.

This tutorial is about scraping airline prices and schedules. You could scrape ticket prices, flight schedules, or even go a bit deeper and get flight prices per kilometer for each airline and plot amazing graphs.

In this tutorial, we will scrape, a leading travel booking website to extract details on flights. Our scraper will extract the flight schedules and prices for a source and destination pair.

Here is a list of fields that we will be extracting:

  1. Arrival Airport
  2. Arrival Time
  3. Departure Airport
  4. Departure Time
  5. Plane Name
  6. Airline
  7. Flight Duration
  8. Plane Code
  9. Ticket Price
  10. No of Stops


Below is a screenshot of some of the data we will be extracting


Scraping Logic

  1. Construct the URL of the search results from Expedia- Here is one for the available flights listed from New York to Miami –,%20NY%20(NYC-All%20Airports),to:Miami,%20Florida,departure:04/01/2017TANYT&passengers=children:0,adults:1,seniors:0,infantinlap:Y&mode=search
  2. Download HTML of the search result page using Python Requests.
  3. Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths. We have predefined the XPaths for the details we need in the code.
  4. Save the data to a JSON file. You can later modify this to write to a database.


For this web scraping tutorial using Python 3, we will need some packages for downloading and parsing the HTML. Below are the package requirements.

Install Python 3 and Pip

Here is a guide to install Python 3 in Linux –

Mac Users can follow this guide –

Windows Users go here –

Install Packages

The Code

The code is self-explanatory.

If the embed above doesn’t work, you can download the code from the link here

If you would like the code in Python 2, you can check out the link here.

Running The Scraper

Assume the script is named If you type in the script name in command prompt or terminal along with a -h


The arguments source and destination are the airport codes for the source and destination airports. The date argument should be in the format MM/DD/YYYY.

As an example, to find the flights listed from New York to Miami we would put the arguments like this:

This will create a JSON output file called nyc-mia-flight-results.json that will be in the same folder as the script. 

The output file will look similar to this:


You can download the code at 

Let us know in the comments how this scraper worked for you.

Known Limitations

This scraper should work for extracting most flight details available on Expedia unless the website structure changes drastically. If you would like to scrape the details of thousands of pages at very short intervals, this scraper is probably not going to work for you. You should read  Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping.

If you need professional help with scraping complex websites, contact us by filling up the form below.

Tell us about your complex web scraping projects

Turn websites into meaningful and structured data through our web data extraction service

Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.

4 comments on “How to Scrape Flight Schedules and Prices from using Python and LXML

Kevin Smith

This was an exciting tutorial as I am new to python, however I received the error below after running the code:

Traceback (most recent call last):
File “”, line 100, in
scraped_data = parse(source,destination,date)
File “”, line 24, in parse
departure_location_airport = flight_data[‘legs’][i][‘departureLocation’][‘airportLongName’]
KeyError: ‘airportLongName’

I’m researching what this error message means, but if you all happen to know what the issue is, I would love to know myself! Cheers!


    Hi, I’m having exactly the same problem. Did you solve it?


    Replace ‘airportLongName’ with ‘airportCity’.


I received the following errors:
File “C:\Users\user1\PycharmProjects\Scraping_Flight1\”, line 95, in
scraped_data = parse(source, destination, date)
File “C:\Users\user1\PycharmProjects\Scraping_Flight1\”, line 16, in parse
parser = html.fromstring(response.text, headers=headers, verify=False)
File “C:\Users\user1\ScrapeFlight_1\lib\site-packages\lxml\html\”, line 876, in fro
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
File “C:\Users\user1\ScrapeFlight_1\lib\site-packages\lxml\html\”, line 762, in doc
value = etree.fromstring(html, parser, **kw)
File “src\lxml\etree.pyx”, line 3215, in lxml.etree.fromstring (src\lxml\etree.c:80983)
TypeError: fromstring() got an unexpected keyword argument ‘headers’

Can you help me with that?

Join the conversation

Turn websites into meaningful and structured data through our web data extraction service