WebSocket Scraping: Extracting Real-Time Data from WebSockets

WebSockets offer a fast, efficient way for streaming real-time data across the web. By using WebSocket scraping, you can track cryptocurrency prices, stock prices, and live event updates. 

This guide discusses how to scrape WebSocket data using Python.

What Is WebSocket Scraping?

WebSocket scraping is the process of establishing a direct connection to a WebSocket server to intercept and extract real-time data streams.

Unlike traditional web scraping—which relies on sending repeated HTTP requests (GET or POST) to retrieve static HTML or REST API responses—WebSocket scraping uses persistent, full-duplex communication channels. 

Once a connection is established, the interaction differs significantly from standard request-response models:

1. Zero Polling Required: The server streams continuous updates directly to the client without requiring continuous HTTP polling. 
2. Low latency: Data frames are pushed instantaneously from the server the moment an event occurs.
3. Direct Payload parsing: The scraper maintains an active socket, listens for incoming data frames, and parses raw JSON or binary payloads on the fly.

How Do You Scrape Data From a WebSocket?

Scraping data from a WebSocket connection follows a slightly different approach than standard HTML parsing. Instead of requesting page elements, you establish a persistent connection, interact directly with the socket server, and listen for incoming data frames in real time.

Below is the step-by-step process to build a real-time WebSocket scraper using Python.

Finding a WebSocket Endpoint

The first step of WebSocket scraping is to locate the WebSocket URL.

You can get it using your browser’s developer tools:

1. Open the Network tab in the developer tools
2. Filter for Socket traffic
3. Reload the page to capture the WebSocket connection
4. Look for the WebSocket URL starting with wss:// and copy it

In this tutorial, the code extracts data from a WebSocket used by CryptoCompare.com. This WebSocket requires you to send a subscription message containing the names of cryptocurrencies for which you need real-time updates.

You can find this subscription message in the Messages section.

Installing the Required Python Packages

Now that you have the WebSocket URL and subscription message, install the necessary Python libraries to prepare the environment for WebSocket data scraping.

You need to install the websocket-client library to interact with WebSocket servers; install it using Python’s package manager pip.

Copypip install websocket-client

Besides this, the code also needs a few packages from the Python standard library, meaning you don’t need to install them:

1. json module to serialize dictionaries into JSON strings
2. time module to implement delays
3. datetime module to create UTC timestamps

Curious about Python packages for web scraping? Check out our article on Python web scraping frameworks.

Importing Necessary Packages

Start the code by importing the required packages.

Copyfrom websocket import create_connection
import json
import time
import datetime

This snippet imports the packages mentioned in the previous section. However, it only imports the create_connection module from websocket.

Establishing the WebSocket Connection

This part of the code for extracting real-time WebSocket data involves creating a connect() function to establish the connection.

You also need to define the headers, which simulate a browser connection.

Copy# WebSocket URL (replace with the URL you copied)
websocket_url = "wss://streamer.cryptocompare.com/v2?format=streamer"

# Define the headers for the WebSocket connection
headers = json.dumps({

	"Host": "streamer.cryptocompare.com",
	"User-Agent": "Mozilla/5.0...",
	"Accept": "*/*",
	"Sec-WebSocket-Version": "13",
	"Upgrade": "websocket"
})

def connect():

	"""Establish a WebSocket connection with automatic retry."""

	while True:
		try:
			ws = create_connection(websocket_url, headers=headers)
			print("Connected to WebSocket. Press CTRL+C to stop.")
			return ws

		except Exception as e:

			print(f"Connection failed: {e}. Retrying in 5 seconds...")

			time.sleep(5)

This code snippet:

1. Initializes two variables:

• websocket_url: This is the WebSocket endpoint that you copied from your browser’s developer tools.
• headers: The headers are necessary to simulate a real browser connection, allowing the WebSocket server to accept the connection.

2. Defines a connect() function to establish a connection to the WebSocket URL.

Sending the Subscription Message

Now you can use the connect() function to establish the connection and send the subscription message.

To do so, define a main() function for this workflow.

Copydef main():
    ws = None
    try:
        ws = connect()

        ws.send(json.dumps({
            "action": "SubAdd",
            "subs": [
                "5~CCCAGG~BTC~USD",
                "5~CCCAGG~ETH~USD",
                "5~CCCAGG~SOL~USD"
            ]
        }))

In this code snippet:

• create_connection(websocket_url, headers) establishes the connection to the WebSocket server.
• ws.send() sends the subscription message to the server. This message subscribes to real-time prices for Bitcoin (BTC), Ethereum (ETH), and Solana (SOL) prices in USD.

Receiving and Processing Data

Now that you’re connected and subscribed to the data stream, you need to listen for incoming messages. Each message will contain updates, such as the latest price for the subscribed currencies.

To handle this, the main() function receives the incoming messages using .recv() and processes them to extract the relevant data.

Copy       previous_price = 0

        # Create a new CSV file for this run with a UTC timestamp in the filename
        run_ts = datetime.datetime.now(datetime.timezone.utc).strftime('%Y%m%dT%H%M%SZ')
        csv_filename = f"CryptoPrices_{run_ts}.csv"
        print(f"Writing prices to {csv_filename}")
        with open(csv_filename, 'w') as f:
            f.write('Currency,Price(USD),Timestamp\n')

        while True:
            try:
                message = ws.recv()
                items = message.split('~')

                price_raw = items[5]
                currency = items[2]

                price = round(float(price_raw), 2)

                if previous_price != price and price != 0:
                    timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat().replace('+00:00', 'Z')
                    with open(csv_filename, 'a') as f:
                        f.write(f"{currency},{price},{timestamp}\n")
                    previous_price = price

            except IndexError:
                pass
            except ValueError:
                pass
            except Exception as e:
                print(f"Error: {e}. Reconnecting...")
                ws.close()
                ws = connect()
                ws.send(json.dumps({
                    "action": "SubAdd",
                    "subs": [
                        "5~CCCAGG~BTC~USD",
                        "5~CCCAGG~ETH~USD",
                        "5~CCCAGG~SOL~USD"
                    ]
                }))
    except KeyboardInterrupt:
        print("Streaming Stopped")
    finally:
        if ws is not None:
            try:
                ws.close()
            except Exception:
                pass

This code does the following:

• Getting the Message: ws.recv() waits for a message from the WebSocket server. Once a message is received, it’s processed.
• Splitting the message: Each WebSocket message contains multiple pieces of data separated by ~. The code splits the message to extract the relevant parts (e.g., price, currency).
• Price extraction: The price is located at index 5 in the message, and the currency is at index 2. The code converts the price to a float and rounds it to two decimal places.
• CSV writing: At startup, the code creates a new UTC-timestamped CSV file with Currency, Price(USD), and Timestamp columns. If the price has changed, it writes the price and the current UTC timestamp to the file.
• Reconnection: If the WebSocket connection drops, the script catches the error, closes the current connection, calls connect() to re-establish it, and sends the subscription message again.

Code Limitation

While the provided code for WebSocket scraping works for many real-time data scraping tasks, there are a few limitations:

1. Rate Limits: WebSocket servers may limit the number of requests or subscriptions. Be aware of these limits to avoid being blocked.
2. Data Volume: High-frequency data streams can result in large amounts of incoming data. Ensure your script can handle the volume efficiently.
3. Connection Stability: WebSocket connections can drop due to network issues. Implementing reconnection logic can mitigate this issue.
4. Message Format Changes: The structure of incoming data can change. If the WebSocket server updates the message format, you’ll need to modify your data parsing code.

Wrapping Up: Why Use a Web Scraping Service

Scraping real-time data from WebSockets involves establishing a connection to the WebSocket server, subscribing to data streams, and processing the incoming data to store it in a CSV file.

However, keep in mind the limitations, such as data volume handling, connection stability, and rate limits. If you’re dealing with larger-scale scraping or facing connection issues, it might be time-consuming to handle everything manually.

For more complex scraping tasks, a web scraping service like ScrapeHero can help automate the process, scale your scraping efforts, and handle WebSocket connections efficiently.

ScrapeHero is an enterprise-grade web scraping service. We can take care of the heavy lifting, allowing you to focus on analyzing the data.