How to Scrape Store Locations from Walmart.com using Python 3

In this tutorial, we will show you how to extract the details of store locations available at Walmart.com, one of the largest retail stores in the U.S.

We’ll search Walmart.com for store locations based on a zip code and extract the following data:

1. Store Name
2. Store ID
3. Distance (in miles) from a given zip code
4. Address
5. Zip Code
6. City
7. Phone Number

Take a look at the screenshot below annotated with the data we will be extracting from Walmart.

You can extract more details from the store detail page such as timings, days open, departments, and services. But for now, we’ll keep it simple and stick to these fields.

Rendering the Data

Open any browser (we are using Chrome) and go to the link https://www.walmart.com/store/finder?location=20005&distance=50. This is the link to search for Walmart store locations listed for the zip code ‘20005’ and within a radial distance of 50 miles.

Right-click on any link on the page and choose – Inspect Element. The browser will open a toolbar and show the HTML Content of the Web Page, formatted nicely. Click  Clear   on the Network panel to clear all requests from the Request table.

Then enter the desired zip code you will need and click on the Search option. Let’s choose the zip code ‘20005’ for now. Select XHR and click on the request:

stores?singleLineAddr=20005&servicceTypes=pharmancydistance=50.

You’ll see the Request URL-

https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=20005&serviceTypes=pharmacy&distance=50.

Open the link in a new tab and you’ll see the data in the raw form. To view the data in JSON format, download the extension JSON formatter.

Building The Scraper

We will use Python 3 for this tutorial. The code won’t run if you are using Python 2.7. You need Python 3 and PIP installed on your computer.

Most UNIX operating systems like Linux and Mac OS comes with Python pre-installed. But, not all the Linux Operating Systems ship with Python 3 by default.

To check your python version open a terminal (in Linux and Mac OS) or Command Prompt (on Windows) and type

python --version

and press enter. If the output looks something like Python 3.x.x, you have Python 3 installed. If it says Python 2.x.x you have Python 2. If it prints an error, you don’t probably have python installed.

If you don’t have Python 3, install it first.

Install Python 3 and Pip

Here is a guide to install Python 3 in Linux – http://docs.python-guide.org/en/latest/starting/install3/linux/

Mac Users can follow this guide – http://docs.python-guide.org/en/latest/starting/install3/osx/

Windows Users go here – https://www.scrapehero.com/how-to-install-python3-in-windows-10/

Install Packages

• Python Requests, to make requests and download the HTML content of the pages (http://docs.python-requests.org/en/master/user/install/).
• UnicodeCSV for handling Unicode characters in the output file. Install it using pip install unicodecsv.

The Code

import csv
import requests
import json
import argparse
import traceback

def locate_stores(zip_code):
    """
    Function to locate walmart stores
    """
    url = "https://www.walmart.com/store/finder/electrode/api/stores?singleLineAddr=%s&serviceTypes=pharmacy&distance=50"%(zip_code)
    headers = { 'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                'accept-encoding':'gzip, deflate, br',
                'accept-language':'en-GB,en;q=0.9,en-US;q=0.8,ml;q=0.7',
                'cache-control':'max-age=0',
                'upgrade-insecure-requests':'1',
                'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36'
    }
    stores = []
    print("retrieving stores")
    for retry in range(10):
        try:
            get_store = requests.get(url, headers=headers, verify=False)
            store_response = get_store.json()
            stores_data = store_response.get('payload',{}).get("storesData",{}).get("stores",[])
            
            if not stores_data:
                print("no stores found near %s"%(zip_code))
                return []
            print("processing store details")
            #iterating through all stores
            for store in stores_data:
                store_id = store.get('id')
                display_name = store.get("displayName")
                address = store.get("address").get("address")
                postal_code = store.get("address").get("postalCode")
                city = store.get("address").get("city")
                phone = store.get("phone")
                distance = store.get("distance")

                data = {
                        "name":display_name,
                        "distance":distance,
                        "address":address,
                        "zip_code":postal_code,
                        "city":city,
                        "store_id":store_id,
                        "phone":phone,
                }
                stores.append(data)
            return stores
        except:
            print(trackback.format_exc())
    
    return []   

if __name__=="__main__":
    
    argparser = argparse.ArgumentParser()
    argparser.add_argument('zip_code',help = 'zip code to search')
    args = argparser.parse_args()
    zip_code = args.zip_code
    scraped_data = locate_stores(zip_code)
    
    if scraped_data:
        print ("Writing scraped data to %s_stores.csv"%(zip_code))
        with open('%s_stores.csv'%(zip_code),'wb') as csvfile:
            fieldnames = ["name","store_id","distance","address","zip_code","city","phone"]
            writer = csv.DictWriter(csvfile,fieldnames = fieldnames,quoting=csv.QUOTE_ALL)
            writer.writeheader()
            for data in scraped_data:
                writer.writerow(data)

Execute the full code with the script name followed by the zip code:

python3 walmart_store_locator.py zipcode

As an example, to find all the Walmart stores in and near Boston, Massachusetts we would put the argument as 20005 for zip code:

python3 walmart_store_locator.py 20005

You should get a file called 20005_stores.csv which will be in the same folder as the script. The output file will look similar to this.

You can download the code at https://github.com/scrapehero/walmart_store_locator.  Let us know in the comments how this scraper worked for you.

Known Limitations

This code should work for extracting details of Walmart stores for all zip codes available on Walmart.com. If you want to learn how to scrape the details of thousands of pages you should read Scalable do-it-yourself scraping – How to build and run scrapers on a large scale and How to prevent getting blacklisted while scraping.

If you need professional help with scraping complex websites, contact us by filling up the form below

Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.