Web Scraping using ChatGPT: A Comprehensive Tutorial

Web Scraping has become an indispensable skill at a time when data is ever-expanding. But what is web scraping?
Web scraping is a process of extracting specific data points from web pages or HTML content. It involves sending HTTP or HTTPS requests to a web server and then sifting through the response to pluck out the desired data.
Meanwhile, ChatGPT is a large language model designed to generate responses based on user input. Its natural language processing capabilities make it adept at sending requests to web pages and parsing and organizing the data obtained. With enough programming knowledge, anyone can scrape any website using ChatGPT.
Now that you have an understanding of what web scraping and ChatGPT are, let’s learn how you can use ChatGPT for web scraping:

How to Automate Web Scraping Using ChatGPT

While ChatGPT itself is primarily designed for generating human-like responses and performing natural language processing tasks, it can be used alongside other tools and libraries to create web scraping solutions.
ChatGPT can be used to automate web scraping tasks such as

1. Input Processing
2. Generation of Scraping Instructions
3. Integration with Web Scraping Libraries
4. Error Handling and Edge Cases

Note – ChatGPT helps generate instructions and provide guidance, but actually implementing a web scraper requires more programming and knowledge of web scraping libraries. You can check out our resources for a more comprehensive understanding of web scraping.

Workflow for ChatGPT Web Scraping

Input Processing

ChatGPT can process the user inputs and understand the requirements for the web scraping process. For example, we can specify the URL to be scraped.

Generation of Scraping Instructions

ChatGPT can generate instructions and commands required for web scraping based on user inputs. The generated instruction/code snippets tell how to extract specific data from the website.

Integration with Web Scraping Libraries

The generated instructions can be combined with different types of scraping libraries, such as BeautifulSoup and Scrapy, etc., to implement an actual scraping process. The generated code/instructions can be used to extract specific data points from the HTML content, perform pagination/navigation, and save the data.

Error Handling and Edge Cases

ChatGPT can assist in error handling and addressing edge cases that may arise during web scraping. It can provide guidance on handling various scenarios, such as handling dynamic website content, avoiding anti-scraping measures, or dealing with different data formats.

How to Create a web scraper using chatGPT

The following prompt can be input to ChatGPT to generate a Python-based web scraper:

User Instruction to ChatGPT terminal

Build a Python-based scraper
1. Start URL - "https://scrapeme.live/shop/"
2. Navigate to each product page xpath='//li/a[contains(@class, "product__link")]' and collect the below data points:
price = '//p[@class="price"]/span/span/following-sibling::text()'
product title = '//h1[@class="product_title entry-title"]/text()'
description = '//div[contains(@class,"product-details__short-description")]/p/text()'
3. After extracting the details from the product page, do the pagination till 5 pages of the listing page
4. Utilize the Python requests module and the lxml parser to send HTTPS requests and extract the required data from the web pages.
5. Implement error handling by checking the response from the server or website. If the response status code is not 200, raise an exception indicating the failure and suggest a retry for that specific request.
6. All requests should be made through a proxy IP 192.168.1.10:8000 and with the following headers
headers = {
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
'accept-language': 'en-GB,en;q=0.9',
'cache-control': 'no-cache',
'pragma': 'no-cache',
'sec-ch-ua': '"Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Linux"',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'none',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
}

Output Instruction Generated by ChatGPT

import requests
from lxml import etree
start_url = "https://scrapeme.live/shop/"
proxy = "192.168.1.10:8000"
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-language': 'en-GB,en;q=0.9',
    'cache-control': 'no-cache',
    'pragma': 'no-cache',
    'sec-ch-ua': '"Google Chrome";v="113", "Chromium";v="113", "Not-A.Brand";v="24"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Linux"',
    'sec-fetch-dest': 'document',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-site': 'none',
    'sec-fetch-user': '?1',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
}
def scrape_product_page(url):
    response = requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers)
    if response.status_code != 200:
        raise Exception(f"Request failed with status code: {response.status_code}. Retry the request.")
    html = response.text
    parser = etree.HTMLParser()
    tree = etree.fromstring(html, parser)
    price = tree.xpath('//p[@class="price"]/span/span/following-sibling::text()')[0]
    title = tree.xpath('//h1[@class="product_title entry-title"]/text()')[0]
    description = tree.xpath('//div[contains(@class,"product-details__short-description")]/p/text()')[0]
    return {
        "price": price,
        "title": title,
        "description": description
    }
def scrape_listing_page(url):
    response = requests.get(url, proxies={"http": proxy, "https": proxy}, headers=headers)
    if response.status_code != 200:
        raise Exception(f"Request failed with status code: {response.status_code}. Retry the request.")
    html = response.text
    parser = etree.HTMLParser()
    tree = etree.fromstring(html, parser)
    product_links = tree.xpath('//li/a[contains(@class, "product__link")]/@href')
    for link in product_links:
        product_url = start_url + link
        product_data = scrape_product_page(product_url)
        print(product_data)  # You can modify this line to store the data in a database or file.
def scrape_multiple_listing_pages(start_url, num_pages):
    for page in range(1, num_pages + 1):
        listing_url = start_url + f"page/{page}/"
        try:
            scrape_listing_page(listing_url)
        except Exception as e:
            print(e)
            print(f"Retrying request for {listing_url}")
            scrape_listing_page(listing_url)
scrape_multiple_listing_pages(start_url, 5)

Limitations

ChatGPT is more than capable of helping you create a web scraper. However, it’s essential to be aware of the limitations involved. Below are some key points to bear in mind:

• Anti-scraping measures can pose challenges: Some websites use certain anti-scraping technologies, which help in preventing the web scraper from scraping the data from their websites. ChatGPT may not be aware of such kinds of blocking or ways to bypass it. This requires expertise in web scraping and related tools, which ChatGPT lacks.

If you’re curious, here’s how to scrape websites without getting blocked

• Maintenance and Scalability: As web scraping projects evolve, maintenance and scalability become challenging. ChatGPT can’t provide instruction to address these challenges, including handling large amounts of data.
• Difficulty in Handling Complex Scraping Scenarios: Web scraping can include handling complex data, dynamic content, maintaining sessions, etc., which is not readily available using ChatGPT.

A web scraping service provider can help you to overcome these challenges easily.

Use cases

While there are certain limitations to using ChatGPT for web scraping, there are many use cases where it can prove to be a valuable asset for automated web scraping. Some instances include:

• Automation: ChatGPT can automate repetitive tasks by generating code snippets or instructions based on user input, thus reducing manual effort.
• Rapid Prototyping: ChatGPT can rapidly build a prototype for web scraping with minimal functionality from initial instructions from the user. This prototype serves as a starting point for developing scrapers and checking the feasibility of said scraper.
• Error Handling and Troubleshooting: ChatGPT can help troubleshoot common problems or errors encountered during web scraping.

Wrapping up: ChatGPT Web Scraping Alternative

We have come a long way in this tutorial. By now, you have mastered the art of ChatGPT web scraping. Web scraping with NLP holds great potential, and the future for it looks bright as ever.
However, building and maintaining web scrapers can be challenging, especially when the structure of web pages changes or when dealing with large-scale data extraction.
If you’re looking for a more streamlined, speedy, and hassle-free solution, we have an alternative for you: ScrapeHero Cloud.
ScrapeHero Cloud offers pre-built crawlers and APIs for all your web scraping needs. The platform is known for its affordability, speed, and reliability. Its user-friendly interface allows users to start scraping without extensive technical knowledge by offering a no-code approach. ScrapeHero Cloud provides free options as well for those on a tight budget or looking to test the service.
Remember, the method you choose depends on your specific requirements and constraints. While ChatGPT is an incredible asset for those who want a programmatic approach, ScrapeHero Cloud caters to those looking for ease and convenience.