Why *not* scrape yourself

Before you get all kinds of ideas about what the topic of this article means – please look at the context – We are talking about Web Scraping here ! This post will talk about reason why not to do this yourself and why to call in a professional (wink wink – use ScrapeHero) You see some data that is publicly available and you start thinking about the possibilities of what you could do with it. A brand new use, maybe the idea for a new startup, maybe a way to improve your existing business, reduce costs, get a leg … Continue reading Why *not* scrape yourself

Webscraping using Python without using large frameworks like Scrapy

If you need publicly available data from scraping the Internet, before creating a webscraper, it is best to check if this data is already available from public data sources or APIs. Check the site’s FAQ section or Google for their API endpoints and public data. Even if their API endpoints are available you have to create some parser for fetching and structuring the data according to your needs.   Scrapy is a well established framework for scraping, but it is also a very heavy framework. For smaller jobs, it may be overkill and for extremely large jobs it is very … Continue reading Webscraping using Python without using large frameworks like Scrapy