Google Shopping, formerly known as Google Product Search, Google Products, and Froogle is an online platform by Google that facilitates efficient online shopping experiences for consumers as well as businesses. It streamlines the process of finding a wide array of products, and one can even scrape Google Shopping for instant comparisons of product details such as price, availability, reviews, rating, and much more.
Google Shopping also offers a robust advertising solution, namely Product Listing Ads (PLAs), so businesses can showcase their products in Google search results. You might already be familiar with scraping Google Maps, Google Careers and Google Reviews. This article will be mainly focused on scraping Google Shopping results.
Why Scrape Google Shopping?
It becomes necessary to scrape Google Shopping as the conversion rate of Google Shopping is significantly high. Scraping large amounts of e-commerce data can assist enterprises in making informed decisions on marketing strategies and tracking competitors. Below are some of the prime reasons why you should scrape Google Shopping.
-
Market Research
You can scrape Google Shopping results data for market research as it provides a deep understanding of the current trends in the market. A detailed analysis of products can lead to a better product marketing strategy for any business. Tracking the competitors and updating with the trends also contributes to creating a better market for any product.
-
Price Monitoring
Google Shopping price monitoring allows businesses to monitor and compare their competitors’ product prices on different eCommerce sites. Google Shopping results are thus considered an efficient way for businesses to create a competitive pricing strategy, stay ahead of the competition, and lure more customers.
-
Product Research
Apart from Google Shopping price monitoring the obtained results can be made in use for product research. Product listings, performance of different products, sales data, and other product data can be tracked, and companies can determine their product placement in the market.
Google Shopping Page Structure
A Google Shopping Results Page generally includes product name, product image, price, product link, merchant name, reviews, and ratings. Understanding the Google Shopping page structure is crucial to creating a Google Shopping scraper. Distinct HTML elements contain all this information, and with a suitable web scraping tool or library, they can be extracted.
Search/Home Page
The Search/Home page lists some recent and trending products from different categories. Using the search bar, desired products can be searched and located. A typical Google Shopping Home page may be similar to the image given below. Note that, according to the users, product suggestions can vary.
Listing Page
The keyword that is used for searching in the search bar will land up on the listing page, where all the products based on keywords are listed. Usually, the listing page includes the following sections:
- Resultant Products: These are the search results that are listed according to the search keyword. Every listed product in this section contains the following data.
- Image Thumbnail: A small image icon of the product.
- Title: The title of the product
- Rating: The customer rating of the product.
- Rating Count: The number of customers that rated this product.
- Price: The sale price of this product.
- Short Specification: The major specifications of this product.
- Website: The actual website where the product is listed.
- Price Comparison: A hyperlink that leads to the price comparison of this product on different websites.
- Filter Bar: With the filter bar, you can sort or filter the search results according to the requirement. Different categories will have different filter options.
The image given below is a typical listing page that opens when searched for the keyword laptop.
Product Page
From the listing page you can click a specific product to open the product page. It contains more detailed information about the product, which is given below.
Image: The image of the product.
Title: The title of the product
Rating: The customer rating of the product.
Rating Count: The number of customers that rated this product.
Price Comparison Table: A table that helps in Google Shopping price monitoring and compares the product prices from different websites.
A typical product page is as shown below:
Steps To Scrape Google Shopping
To scrape Google Shopping, different tools can be used. In this tutorial, the Python-Requests module for handling HTTP requests and LXML for parsing and extracting data from the response are used in creating a Google Shopping scraper.
Run the following commands to scrape Google Shopping and complete the process:
-
Installing Dependencies
$ pip install requests
$ pip install lxml-
Importing the Required Packages
# For csv file handling
import csv
# For cleaning the data
import re
# For sending HTTP Request
import requests
# For parsing the response
from lxml import html-
Initializing proxies
If you are using proxies define it here. We are keeping it as empty dictionary
# Defining proxies
proxies = {
# "http": "http://:",
# "https": "http://:"
}-
Defining the Headers for the Requests
# Defining the HTTP headers for the request
headers = {
"authority": "www.google.com",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,a\
pplication/signed-exchange;v=b3;q=0.7",
"accept-language": "en-GB,en-US;q=0.9,en;q=0.8",
"referer": "https://shopping.google.com/",
"sec-ch-ua": '"Chromium";v="110", "Not A(Brand";v="24", "Google Chrome";v="110"',
"sec-ch-ua-arch": '"x86"',
"sec-ch-ua-bitness": '"64"',
"sec-ch-ua-full-version": '"110.0.5481.177"',
"sec-ch-ua-full-version-list": '"Chromium";v="110.0.5481.177", "Not A(Brand";v="24.0.0.0", "Google Chrome";v="\
110.0.5481.177"',
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-model": '""',
"sec-ch-ua-platform": '"Linux"',
"sec-ch-ua-platform-version": '"5.14.0"',
"sec-ch-ua-wow64": "?0",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "same-site",
"upgrade-insecure-requests": "1",
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari\
/537.36",
}-
Creating a Function To Save the Data to a CSV File
def save_as_csv(data: list):
"""
The function that used to save
the data as a csv file
Args:
data (list): data as list of dictionary
"""
# Column(field) names for the csv file
fieldnames = ["title", "image_url", "rating", "website", "item_price", "url"]
# Opening an csv file to save the data
with open("outputs.csv", mode="w", newline="") as file:
writer = csv.DictWriter(file, fieldnames=fieldnames)
# Writing the headers(Column name)
writer.writeheader()
# Iterating through each product data
for row in data:
# Writing the data in a row
writer.writerow(row)-
Creating a Function To Extract Product URLs From Listing Page
def parse_listing_page(input_search_term: str):
"""
This function will generate the listing page url from
the input search keyword, Send request to it and extract
product URLs from the response.
Args:
input_search_term (string): The seach keyword that we want
to search
Returns:
list: List of extracted product urls
"""
# Generating the listing page URL
url = f"https://www.google.com/search?q={input_search_term}&safe=off&tbm=shop"
# Sending request to the URL
response = requests.get(url=url, headers=headers)
# Parsing the response
tree = html.fromstring(response.content)
# Extracting product URLs
product_urls = tree.xpath('//a[@class="xCpuod"]/@href')
return product_urls-
Creating a Function To Extract the Data From Product Page
def parse_product_page(product_urls: list):
"""
This function will accept a list of input URLs,
iterate through each of them, Send request to each
URL, store the data in a list and return back
Args:
product_urls (list): List of product URLs
Returns:
list: data as list of dictionary
"""
# Initializing an empty list to save the extracted data
data = []
# Iterating through each product URL
for product_url in product_urls:
# Generating the product URL
url = f"https://www.google.com{product_url}"
# Sending the request
response = requests.get(url=url, headers=headers)
# Parsing the response
tree = html.fromstring(response.content)
# Extracting title and image url
title = tree.xpath("//span[contains(@class, 't__title')]/text()")
image_url = tree.xpath(
"//div[@id='sg-product__pdp-container']//img[contains(@alt, 'View product image #1')]/@src"
)
if not image_url:
image_url = tree.xpath(
"//div[@id='sg-product__pdp-container']//div[contains(@class, 'main-image')]/img/@src"
)
# Extracting the price comparison table
price_table = tree.xpath(
"//table[contains(@id, 'online-sellers-grid')]/tbody/tr"
)
# Extracting rating
rating = tree.xpath(
'//a[@href="#reviews"]/div[contains(@aria-label, "out of")]/@aria-label'
)
# Iterating through each row
for row in price_table:
# Extracting the website name
website = row.xpath("./td[1]//a/text()")
if not website:
continue
# Extracting the price on that website
item_price = row.xpath("./td[3]/span/text()")
item_price = item_price[0] if item_price else None
# Creating a dictionary of data and appending to the list
data_to_save = {
"title": title[0] if title else None,
"image_url": image_url[0] if image_url else None,
"rating": rating[0] if rating else None,
"website": website[0] if website else None,
"item_price": item_price,
"url": url,
}
data.append(data_to_save)
return data-
Combining the Functions
if __name__ == "__main__":
input_search_term = "laptop"
product_urls = parse_listing_page(input_search_term)
data = parse_product_page(product_urls)
save_as_csv(data)Put all the code together to see the output. Refer the following link to get the complete code.
https://github.com/scrapehero-code/Google-Shopping/blob/main/google_shopping.py
Wrapping Up
The Google Shopping scraper is a powerful tool that assists enterprises in gaining insight into products and trends in the market, thus leading to data-driven decisions for scaling businesses. Target markets, consumers, competitors, etc. can be studied from the extracted data.
ScrapeHero, a reputed enterprise-grade data scraping service provider, offers various services related to data scraping. Check out our scrapers for scraping various Google services. Let’s connect if what you search for is a reliable data scraping service provider for customized services ranging from data extraction to robotic process automation to alternative data.
