Navigating the Variations: Scraping Data Behind Feature Flags

Feature flags make life more complicated for web scrapers. Because of them, the target website becomes highly dynamic, making it necessary to shift your scraping techniques. This article discusses scraping data behind feature flags using advanced scraping techniques.

Why Feature Flag Data Scraping Is Challenging

Feature flags make it easier to create dynamic and interactive web experiences. They enable developers to activate or deactivate specific features without requiring a new deployment.

These flags are conditional statements that control the execution of specific features. 

Web developers can create multiple versions of a webpage using them, which makes extracting the right content difficult because the scraper has to pull data from different versions from the same URL.

How Feature Flags Work

Feature flags are fundamentally conditional statements, typically if-then-else logic. They determine when specific code paths execute at runtime.

By evaluating these flags, websites can toggle specific features depending on:

• User Roles
• User Preferences
• Environment Variables
• Specific Dates or Times
• User Attributes
• Geographic Segments

Websites may evaluate feature flags on the server side or the client side. They prefer server-side evaluation because it provides a cleaner user interface, doesn’t expose feature names, and reduces client-side code size.

However, server-side evaluation of feature flags makes web scraping a challenging task. The client only gets the content that the feature flag permits, which means you don’t know the flag states and the logic behind them.

In contrast, it is much easier to scrape data behind feature flags if the website evaluates them on the client’s device. That’s because the code for all the versions is shipped to the client.

By analyzing the code, you can understand the logic and use browser-automation libraries to simulate various flag states and user interactions.

Go the hassle-free route with ScrapeHero

Why worry about expensive infrastructure, resource allocation and complex websites when ScrapeHero can scrape for you at a fraction of the cost?

Join Us

Scraping Data Behind Feature Flags: Techniques

The techniques for scraping feature-flags controlled data depend on whether the website evaluates the flag on the client side or server side.

Client-Side Evaluated Flags

Start by identifying how the website manages the client-side feature flags using the browser’s DevTools.

Browser Storage

Identify whether the website stores the flags in the browser’s storage:

1. Open DevTools
2. Go to the Application tab
3. Check key-value pairs in local storage, session storage, and cookies.

Network Traffic

Identify network traffic and look for API calls that fetch flags:

1. Open DevTools
2. Go to the Network tab
3. Filter for endpoints like /flags, /featureflags, etc.

Loaded JavaScript

Analyze the loaded JavaScript:

1. Open DevTools
2. Go to the Sources/Debugger tab
3. Click on the JavaScript files to see the code
4. Look for known SDKs or flag libraries (e.g., ld client) and the flag checks (e.g., if statements checking window.FEATURE_FLAGS)

After determining how the website stores the flag states, use one of these techniques to modify them:

Locally Stored Flags

Use add_init_script() to add an initial script that sets the flag state before the page loads. Here’s an example using Python Playwright:

Copy# assuming you have launched the Playwright browser
page.add_init_script("""
  window.localStorage.setItem('feature_flag','true');
""")
page.goto("https://example.com")

Cookie-Stored Flags

Use add_cookies() to modify the cookies before the page loads. The browser will send cookies with the requests.

Copy# Set cookie for feature flag
context = browser.new_context()
context.add_cookies([{
    'name': 'feature_flag',
    'value': 'true',
    'domain': 'example.com',
    'path': '/'
}])
page = context.new_page()
page.goto("https://example.com")

Flags Evaluated Using SDKs

Use add_init_script() to mock the flag evaluation before the page loads. For instance, if the object is ‘FEATURE_FLAGS’, you can use this code:

Copypage.add_init_script("""
  window.FEATURE_FLAGS = window.FEATURE_FLAGS || {};
  window.FEATURE_FLAGS['newFeature'] = true;
""")
page.goto("https://example.com")

API-Fetched Flags

After figuring out the correct API endpoint, use route() to mock a response that gets you the desired flag value.

Copydef handle_route(route):
    if "api/flagstates" in route.request.url:
        route.fulfill(
            status=200,
            content_type="application/json",
            body='{"newFeature": true}'
        )
    else:
        route.continue_()

page.route("**/api/flagstates", handle_route)
page.goto("https://example.com")

Server-Side Flags

Trial and error is the only way to find all the variations of a website when it evaluates flags on the server side.

You need to try possible combinations of user IDs, cookies, headers, etc., while making requests to the servers, so that they send different variations of the web page according to each combination.

This process is similar to finding the API endpoint when a website uses client-side evaluation, but because the logic for evaluating the flag is not visible, you cannot target specific flag states as easily.

Moreover, unlike client-side evaluation—which sends flags—the server-side evaluation only sends the rendered web page.

Scraping Data Behind Feature Flags: Challenges

Web scraping for feature-flags controlled data presents several challenges:

• Multiple feature flags create further complexities as you need to understand the purpose of each flag state through trial and error.
• Websites frequently change their HTML, requiring you to analyze the HTML code for the latest feature flag logic (in the case of client-side evaluation).
• Websites may restrict scraping using various techniques, like CAPTCHAs or IP blocking. You need to find ways to avoid these anti-scraping measures.
• Using browser automation libraries for extracting data behind feature flags is resource-intensive. This can contribute to considerable upfront costs, especially for large-scale projects.

Wrapping Up: Why Use a Web Scraping Service

It’s possible to scrape data behind feature flags. But you need to spend time understanding the logic behind the flags or simulating several user profiles to fetch all the data. This requires you to be proficient at coding.

Moreover, you also need to navigate the quagmire of the ethical and legal landscape, which requires considerable research.

However, if you just want data, a web scraping service will be much more efficient.

A web scraping service like ScrapeHero can take care of researching and understanding the possibilities of data collection from the web page variations. You are then free to focus on the analysis part.

ScrapeHero is a full-service web scraping service provider. We can build enterprise-grade scrapers and crawlers according to your needs. Contact ScrapeHero to stop worrying about data collection.