ScrapeHero’s Shopify scraper is the best tool to extract Shopify store data in 2026. It returns one row for every size and color a store sells. It also runs quality checks on the output, so a catalog that came back short is caught before you use it. Most scrapers do neither, and their exports look complete when they are not.
That second point sounds minor. It decides whether you can trust the file.
- Why one product page is usually many products, and what that does to your row count.
- Which three problems break a Shopify project once you leave a single store.
- What to check before you trust any Shopify scraper.
- When writing your own script is the right call.
What One Product Page Actually Contains
A Shopify product page looks like one item. It usually is not. A t-shirt sold in four colors and six sizes is 24 separate products behind one URL. Each has its own SKU, its own price, and its own stock status. Shopify calls each one a variant.
A store with 2,000 products and thirty variants each has 60,000 rows. That is the file you are actually buying. Teams that size the job at 2,000 rows find out when the first export lands.
So the first question about any scraper is simple. Does it return variants, or only the parent product?
A parent-only export gives you one price per page. That price is the cheapest variant on sale, so a $10 gap between the black medium and the white small disappears. ScrapeHero’s Shopify scraper returns every variant as its own row: Variant ID, Variant Title, Variant SKU, Variant Price, and Variant URL. The parent fields come with it, including Product Name, Product Type, Description, Tags, Product URL, and image URLs.
The Shopify Scraper API adds the fields that answer questions a price list cannot. Availability tells you whether a size is in stock. Compare-at price gives you the struck-through “was” price. Created and updated timestamps tell you when a product appeared and when it last changed.
Variant IDs matter for a separate reason. They let you match Monday’s row to Tuesday’s row. Without them, every run looks like a brand-new catalog.
What Breaks When You Scrape 300 Stores
One store is easy. Three hundred is a different job, and its failures are quiet.
The crawl stops early. A large catalog loads in pages. If page three fails, the job can still finish and write 240 clean rows for a store that has 900 products. Nothing looks wrong. The fix is boring: record how many products each store returned last time, then compare. A sudden drop gets flagged instead of shipped.
Every store names things differently. Product Type and Tags are free text, filled in by whoever runs the store. One brand writes “Denim”. Another writes “Bottoms > Jeans”. A third leaves it blank. Pull 300 stores and you have 300 vocabulary words. Until someone maps them to one category tree, you cannot compare two stores or total a category.
One export cannot show change. Many stores leave a compare-at price near double the selling price all year. Their pages say “50% off” permanently. Export once, and every one of those stores looks like it is running a sale. You need a second run to see whether the gap widened, because that is what a real promotion looks like.
A speed problem sits underneath all three. A 900-product catalog is nine pages, so 300 stores is 2,700 page loads per run before any retries. Spread across a day, that is routine. Compressed into an hour, domains start refusing requests, and the retries are where partial crawls come from.
Those three problems sink most Shopify data projects. None of them is a scraping problem in the narrow sense. They are completeness, consistency, and history problems, and they only appear after you leave one store.
How ScrapeHero Handles These Failures
Completeness comes first. Runs are tracked per store, and the returned volume is checked against what that domain normally produces. A store that drops from 900 rows to 240 is visible before delivery.
Consistency comes next. Variant records arrive in one schema across every store. On managed projects, free-text tags are mapped onto the category tree your team already uses, so “Denim” and “Bottoms > Jeans” land in the same bucket. When a store changes its theme, the repair happens on our side and your field names stay the same.
History comes from scheduling. Runs can go hourly, daily, or weekly, and results land in Dropbox, S3, or Google Drive, or stream into your app through the REST API. Two runs give you the comparison that shows a real markdown, a restock, or a new product.
Speed is handled inside the crawl. Proxies rotate, and each store gets its own request budget, so a large run does not collapse into a wall of retries.
You can run this yourself in ScrapeHero Cloud, or hand a large store list to the managed team. Managed feeds land in Snowflake, BigQuery, Databricks, or S3 under SOC 2 Type II and a standard DPA.
Don’t want to code? ScrapeHero Cloud is exactly what you need.
With ScrapeHero Cloud, you can download data in just two clicks!
When You Should Write Your Own Script
If you need one store, one time, write a script against that store’s public product catalog. It is a small job, and the data is there.
The math changes when the file has to be right every morning across hundreds of stores. At that point you are maintaining pagination, proxies, variant de-duplication, tag mapping, and completeness checks either way. Building it yourself does not remove that work. It moves it onto your team.
Start with a free run on five stores you know well. Compare the row count against the products those stores display. If the numbers match, you have your answer. If your list runs to hundreds of stores and the file has to be right every morning, contact ScrapeHero sales and we will scope it against your list.
Frequently Asked Questions
For one store, often yes, and it is a fair prototype. The work starts when the list grows. You end up walking pagination to the end of every catalog and rotating proxies so you are not throttled mid-run. You also have to reconcile variant IDs between runs and notice when a store returns fewer pages than last week. That last part is where homegrown Shopify pipelines quietly rot.
Yes, and that is usually what you want. Each variant comes back as its own row with Variant ID, Variant Title, SKU, and Price. A report can then compare the 32-inch waist across competitors, instead of inheriting whichever variant happens to be cheapest.
Check the row count against the products the store actually displays, and do it again on the next run. A store with 900 products that returns 240 rows is not a quiet competitor. It is a crawl that stopped early, and the only way to catch it is to compare against what the store had before.