How to Scrape Twitter Without Coding

Accessing data from social media feeds can be useful in conducting sentiments analysis and understanding user behavior towards a particular event, product, or statement. With the right infrastructure, you can scrape twitter for keywords or based on a time frame. This tutorial shows you how to scrape tweet data from Twitter’s advanced search for free using the Twitter Scraper available on ScrapeHero Cloud and help you scrape Twitter data easily without any coding.

Here are the steps to scrape Twitter Data:

  1. Create a ScrapeHero Cloud account and select the Twitter Crawler.
  2. Input the Twitter Advanced search URLs and filters to be scraped.
  3. Setup and run the Twitter scraper.
  4. Download the scraped tweet data from Twitter (CSV, JSON, XML).

The ScrapeHero Cloud has pre-built scrapers that in addition to gathering social media data from the web, can Scrape Google, Scrape Job data, Scrape Real Estate Data and more. The tool is easy to use and does not require any coding skills to run, it also provides a free plan to test the speed, accuracy, and quality of the data before signing up for a paid plan. These scrapers are pre-built and cloud-based, you need not worry about selecting the fields to be scraped nor download any software in order to run the scraper. The scraper can run from any browser and can deliver the data directly to Dropbox.

If you don't like or want to code, ScrapeHero Cloud is just right for you!

Skip the hassle of installing software, programming and maintaining the code. Download this data using ScrapeHero cloud within seconds.

Get Started for Free
Deploy to ScrapeHero Cloud

The crawler scrapes the data without logging in, so the actual number of pages crawled might differ in ScrapeHero Cloud.

Data Fields to Extract

These are the data fields we can extract using the Twitter Crawler based on the input URLs.

  1. Handle
  2. Content
  3. Name
  4. Replies
  5. Retweets
  6. Favorite
  7. Date
  8. Hashtag
  9. URL

Step 1: Create an account

First, we will create an account in ScrapeHero Marketplace. To sign up go to the link – https://cloud.scrapehero.com/accounts/login/ and create an account with your email address.

Step 2: Input the Details for the Twitter Scraper

There are two ways you can provide input URL for the Twitter crawler in two ways:

– Getting the input URL from Twitters Advanced Search

Twitter Advanced Search lets you find historical tweets that you can filter based on parameters like Words, People, and Dates. In order to scrape historical tweet data, use the advanced search in Twitter by going to this URL

https://twitter.com/search-advanced?lang=en

and filter the data based on your needs. For now, we will do a search for all tweets which has the text “tesla” and was made between October 1 to October 5, 2018. Copy the search result URL. Our link looks like this:

https://twitter.com/search?l=&q=tesla%20since%3A2018-10-01%20until%3A2018-10-05&src=typd&lang=en

– Providing a Hashtag or Twitter Profile as an input URL

You can provide the URL of a Twitter Profile like this:
https://twitter.com/NatGeo

Or based on a search hashtag like this:
https://twitter.com/hashtag/ElonMusk?src=hashtag_click&f=live

Step 3: Setting up Twitter Scraper and Running it

This advanced Twitter scraper allows you to input filters based on which you would like to scrape tweets from Twitter. Choose the date filter to limit the tweets from a certain time and the number of tweets to collect. If do not want any original or referenced quotes and hashtags, you have the option to exclude them. After you have input all your URLs and filters click on ‘Continue’.

The Twitter crawler page will open up and you will see the option to gather the data. Once you have click it, the scraper will start scraping tweets from Twitter.

After the scrape is complete the ‘Status’ of the crawler will change from ‘Started’ to ‘Finished’. Click on ‘View Data’ to view the scraped Twitter data. 

Step 4: Download Twitter Data

You can see all the scraped tweets on this page. To download the scraped tweet data click on ‘Download Data’.

A drop down to select a data format will appear. You can choose between CSV, JSON and XML formats. After clicking on the data format option, a file will soon be downloaded with all the scraped Twitter data.

You can get data delivered to Dropbox if you integrate the crawler account to your Dropbox account. You also have the option to schedule the data if you want to scrape twitter data on a timely basis. 

Go to the tab ‘Schedule’ in the table and click on the button ‘Add Schedule’. There are the options to choose the date, time and time zone along with the options to repeat the run as often as you want – hourly, weekly or daily. 

If you don't like or want to code, ScrapeHero Cloud is just right for you!

Skip the hassle of installing software, programming and maintaining the code. Download this data using ScrapeHero cloud within seconds.

Get Started for Free
Deploy to ScrapeHero Cloud

The crawler scrapes the data without logging in, so the actual number of pages crawled might differ in ScrapeHero Cloud.

We can help with your data or automation needs

Turn the Internet into meaningful, structured and usable data



Please DO NOT contact us for any help with our Tutorials and Code using this form or by calling us, instead please add a comment to the bottom of the tutorial page for help

Disclaimer: Any code provided in our tutorials is for illustration and learning purposes only. We are not responsible for how it is used and assume no liability for any detrimental usage of the source code. The mere presence of this code on our site does not imply that we encourage scraping or scrape the websites referenced in the code and accompanying tutorial. The tutorials only help illustrate the technique of programming web scrapers for popular internet websites. We are not obligated to provide any support for the code, however, if you add your questions in the comments section, we may periodically address them.

Responses

Meaghan April 3, 2019

What is the significance of the value used in the request interval field?

Reply

k T April 18, 2019

this use to work but no longer returns any results when the process has completed. Would you be able to assist?

Reply

    ScrapeHero April 19, 2019

    You must be getting blocked by Twitter. We just tested this again, and seems to be working fine. Would you mind sharing the link to the adavanced search results you used.

    Reply

      Kristian Pedersen April 23, 2019

      We are experiencing the same problem. Once the scraping tool have completed there is no data available and we cant generate a csv file. We are able to extract “top posts” for a given day, but only get around 90 observations.

      We are using this url “https://twitter.com/search?f=tweets&vertical=news&q=brexit%20since%3A2016-06-15%20until%3A2016-06-20&l=en&src=typd”

      Do you have any idea of why this may be?

      Reply

        ScrapeHero April 29, 2019

        Could you also try using the advanced search for the same keyword IFB and limiting it between a shorter date range.

        Reply

          K T April 29, 2019

          Older version worked with the URL previously provided for searches with the latest results.

          I tried the advanced search with the latest version of the extension, it worked. Seems that it came down to a user error at the end of the day.

          Thanks, ScrapeHero !! You guys rock


          ScrapeHero April 30, 2019

          Glad to be of help 😉


Kristian Pedersen April 23, 2019

We are experiencing the same issue. Once the scraping tool have completed, we don’t get any data. Have this become a common problem?

Reply

    ScrapeHero April 23, 2019

    Hey Kristian, This might be a problem with the latest version of the webscraper extension. We haven’t been able to reproduce the problem yet. Would you mind sharing the link to the advanced search results you used.

    Reply

    Marc Dunnink May 21, 2019

    Any ideas?

    Reply

      ScrapeHero May 22, 2019

      Please refer to the comment by K T on April 29, 2019 above. This seems to be a problem with the latest version of web scraper extension.

      Reply

        Marc Dunnink May 22, 2019

        Hi ScrapeHero

        Thanks for getting back to me.
        I had used the latest version of Web Scrapper a few months ago using the advanced twitter search, and worked really well. It seems twitter has changed its interface a bit since then though, the actual search platform is different and the search results differ somewhat. I’m wondering if this perhaps has something to do with it and maybe the JSON from github needs to me updated accordingly.
        In any case, thanks for the assistance, ill also try figure out how to get an older version of the scrapper and see if that works, still not sure how KT got the new version to work with the advanced search.

        cheers,
        M

        Reply

JR May 26, 2019

same here. could it be the amount of data? when I scrape for shorter timeframes and feeds with less content, I get the data, but if longer/more intense feeds, doesn’t work.

Reply

Jagos Radovic May 26, 2019

same problem

Reply

lauren ansell July 16, 2019

Once again having a problem retrieving the data after a scrap. Your example is performing the scrape (I can see the page automatically scrolling) but when I change to scrape the tweets I need, the new page is not even scrolling. But neither are collecting any data and the csv file is empty. Twitter have just launched an update, could this be the issue?

Reply

    ScrapeHero July 19, 2019

    Lauren,
    Please look at the error messages to see what’s going on.
    When we can, we will check the site and see if there is some change.

    Reply

      lauren July 23, 2019

      There is no error message, it appears to preform the scrape but no data is captured.

      Reply

Bryan July 24, 2019

Hi, I could get data for time frames of up to a month. Any longer, it wouldnt display the data. Any workaround for this? Thanks.

Reply

nostromo July 29, 2019

seems the json code doesn’t work with the new twitter layout/design.

Reply

    Fellow Student August 19, 2019

    You can download the firefox/chrome extension GoodTwitter to force the old layout on twitter again. Then the code works again!!

    Reply

      catherineannebrown September 28, 2019

      Thank you!

      Reply

Lauren H November 20, 2019

Hello, i have attempted to scrape twitter data over the period of 9 months but only ended up extracting 100 tweets from one day, why might this be? thanks

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Turn the Internet into meaningful, structured and usable data   

ScrapeHero Logo

Can we help you get some data?