We share some product monitoring scenarios and essential questions you need to ask yourself before you start monitoring products online
Price Impact of Access method
Type of Access: Direct vs Browse vs Search based
The method of product identification has a direct and substantial impact on the pricing so we would like to explain the difference clearly so that you can optimize your spend.
Direct Access
If we get direct access to the product page, you get to spend the least amount to get the data. We can get direct access with either
- Direct URLs to the products e.g you can provide us list of direct URLs to each product such as https://www.walmart.com/ip/Spalding-NBA-Varsity-27-5-Basketball/11032308
- Unique Data points that can help create a direct URL for each product, such as the ASIN for Amazon. eg you provide us an ASIN such as B0192CTMYG and we translate that to the direct product URL such as https://www.amazon.com/dp/B0192CTMYG
With this approach the number of products that you need monitored equals directly the pages we need to monitor and there are no surprises and variance as described in the subsequent methods.
Browse or Navigation based
In a Browse based method, you provide a category or sub-category or even a brand store within Amazon or a sellers virtual store on eBay or Amazon. You want all products or a limited number (say 100) products from that category or store to be monitored.
e.g. The input could be the URL for a Amazon seller i.e. their virtual store such as https://www.amazon.com/s?marketplaceID=ATVPDKIKX0DER&merchant=A19NVE4G6SOT2C)
or the input could be a category such as Washing machines on Home Depot https://www.homedepot.com/b/Appliances-Washers-Dryers-Washing-Machines/N-5yc1vZc3ov
It is very likely that the website will restrict the amount of products that it will show for each category or seller and the data will be limited to such limits in most cases. It is possible to get more than that limit to certain extent, but such methods will increase the page count (and cost to a significant and unpredictable extent)
Search based
In the Search based method, you can provide us any kind of input (i.e a “search term”) that can be searched on a website using the website’s search function. e.g. Search for “Microwave” on Home Depot https://www.homedepot.com/s/microwave or on Best Buy https://www.bestbuy.com/site/searchpage.jsp?st=microwave
Or it could be searching for “USB Cables” on Amazon such as https://www.amazon.com/s/ref=nb_sb_noss_2?url=search-alias%3Daps&field-keywords=usb+cables
As you can see, the search results will show many results (more than 100,000 results in case of USB cables on Amazon) so we need you to let us know how many results we need to monitor keeping in mind that the results will change constantly
Now let’s say you provide us a (supposedly unique) Model number such as “52-3024” and we search that on Amazon.com. Even with a specific manufacturer’s model number we get 200+ results and not one (as probably expected)!
We recommend that we rely on the Search algorithms for each website and assume that the first product they show on the search results is the right product and get that data. This method is the most cost effective (may not be accurate) and will result in two pages checked for each search term – i.e the search results page such as https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=52-3024 and then the product page such as https://www.amazon.com/Crayola-Crayons-24-Count-52-3024/dp/B00CO4QAWW/
The ability of finding products using the “search terms” such as SKUs or MPNs, UPCs at each website depends on whether the website stores the SKU, MPN, UPC etc and allows searching for the SKU, MPN, UPC.
Contrary to popular belief that is not the case for every website and every search term
