Web Scraping Tutorial for Beginners – Part 3 – Navigating and Extracting Data

upvotes = soup.find('div',attrs={'class':'score unvoted'}).text

upvotes

Selecting Original Poster

Original poster or the username of the person who submitted the link is inside an `<a>` tag that looks like

<a href="https://www.reddit.com/user/SmartassRemarks" class="author may-blank id-t2_as4uv">SmartassRemarks</a>

Now you might be tempted to use the classes as is for the selection criteria. `id-t2_as4uv` looks like a randomly generated value. Let’s go to another comments page ( https://www.reddit.com/r/news/comments/7ud052/san_francisco_plans_to_wipe_out_thousands_of_pot/ ) and see if the classes remain the same.

<a href="https://www.reddit.com/user/Kobobzane" class="author may-blank id-t2_1389xh">Kobobzane</a>

Nope, it does. The `id-t2_` has another random value. Let’s try selecting `<a>` tags with `author` as a class. First, let’s check for multiple matches.

len(soup.find_all('a',attrs={'class':'author'}))

`len()` tells you how many items are present in the list. There are 209 – Not a unique selection criteria. Our current criteria selects every username present in the page, we just need the one of the person who submitted the link.

This calls for more isolation. Let’s go a few levels above the `a` tag and find a way to isolate this a from the rest.

All of the details for the link is enclosed in a `div` with an ID `siteTable`. Let’s go ahead and grab it, then extract the rest from the isolated HTML.

main_post = soup.find('div',attrs={'id':'siteTable'})

Let’s check if there are multiple matches here as well. Note that we are using `main_post` instead of `soup` here. We isolated the HTML into `main_post` and we only need to look inside it.

len(main_post.find_all('a',attrs={'class':'author'}))

Great. Just one. We might as well modify the other fields above to just look inside the main_post.

title = main_post.find('a',attrs={'class':'title'}).text
upvotes = main_post.find('div',attrs={'class':'score unvoted'}).text
original_poster = main_post.find('a',attrs={'class':'author'}).text

print(title)
print(upvotes)
print(original_poster)

Selecting Number of Comments

Let’s stick to extracting from `main_post`. Number of comments is in

<a href="/r/news/comments/7ud052/san_francisco_plans_to_wipe_out_thousands_of_pot/" 
data-inbound-url="/r/news/comments/7ud052/san_francisco_plans_to_wipe_out_thousands_of_pot/?utm_content=comments&amp;utm_medium=front&amp;utm_source=reddit&amp;
utm_name=news" data-href-url="/r/news/comments/7ud052/san_francisco_plans_to_wipe_out_thousands_of_pot/" data-event-action="comments" class="bylink comments 
may-blank" rel="nofollow">1825 comments</a>

We can select this by using the `class` or the `data-event-action` attributes. We won’t run into multiple matches are we are in the isolated part of the HTML

comments_count = main_post.find('a',attrs={'class':'bylink comments may-blank'}).text
print(comments_count)

Selecting comments

We have so many comments, and then there are details we need to extract from each comment. So, lets first isolate the comments section to prevent matches with the main post. They are in a div with class as `commentarea`.

comment_area = soup.find('div',attrs={'class':'commentarea'})

Let’s find all comments. If you inspect a single comment and move a few tags upwards, you can see that each comment is in a dive with classes `entry unvoted`. We can use the same to grab and isolate each comment.

comments = comment_area.find_all('div', attrs={'class':'entry unvoted'})
len(comments)

Alright, we have grabbed 379 comments. You can print the variable to see each of them. Lets loop through each comment, extract the commenter, comment text and the permalink to the comment. S0 we’ll skip the inspection and selection as we have been over it for a while.

extracted_comments = []
for comment in comments: 
    commenter = comment.find('a',attrs={'class':'author'}).text
    comment_text = comment.find('div',attrs={'class':'md'}).text
    permalink = comment.find('a',attrs={'class':'bylink'})['href']
    extracted_comments.append({'commenter':commenter,'commenter_profile':commenter_profile,'comment_text':comment_text,'permalink':permalink})

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-24-fc1ffc67af14> in <module>()
      4     comment_text = comment.find('div',attrs={'class':'md'}).text
      5     permalink = comment.find('a',attrs={'class':'bylink'})['href']
----> 6extracted_comments.append({'commenter':commenter,'commenter_profile':commenter_profile,'comment_text':comment_text,'permalink':permalink})

NameError: name 'commenter_profile' is not defined

Now that’s bad. Let’s find out why we got this error. Print the comment variable and look at the HTML.

commment

We are getting this because there is a load more comments box after many comments. For that, we need to filter those out when we look for comments. If you inspect the HTML of any comment, you can see it has a form element in it. So let’s filter out anything that doesn’t have a form element in it. Let’s rewrite the code block above to look like this:

extracted_comments = []
for comment in comments: 
    if comment.find('form'):
        commenter = comment.find('a',attrs={'class':'author'}).text
        comment_text = comment.find('div',attrs={'class':'md'}).text
        permalink = comment.find('a',attrs={'class':'bylink'})['href']
        extracted_comments.append({'commenter':commenter,'comment_text':comment_text,'permalink':permalink})

`extracted_comments` should now contain a list of dicts like

{'comment_text': 'TL;DR Two immune stimulating agents injected into the tumors of mice eliminated all traces of cancer as well as distant, untreated metastases. 87 of 90 mice were cured of the cancer.\nAlthough the cancer returned in three mice, they again regressed after a second treatment.\nThe researchers saw similar results in mice bearing breast, colon, and melanoma tumors. Treating the first tumor that arose often prevented the occurrence of future tumors.\nOne agent is already approved for human use and the other has been tested in other unrelated clinical trials. A clinical trial was launched in January to test the effect of the treatment in patients with lymphoma.\n',
  'commenter': 'Robobvious',
  'permalink': 'https://www.reddit.com/r/worldnews/comments/7ug4pb/cancer_vaccine_eliminates_tumors_in_mice_90_of_90/dtk7tie/'}

Permalink of Post

The only field left is the permalink of the post, which we will get directly from the URL in the response object.

permalink = request.full_url

Putting it all together into a function

We’ll make a function that receives a comment page URL as an input and then parse and extract the data from it. This will return a dict with the data.

def parse_comment_page(page_url):
    #Adding a User-Agent String in the request to prevent getting blocked while scraping 
    user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
    request = urllib.request.Request(page_url,headers={'User-Agent': user_agent})
    html = urllib.request.urlopen(request).read()
    soup = BeautifulSoup(html,'html.parser')
    main_post = soup.find('div',attrs={'id':'siteTable'})
    title = main_post.find('a',attrs={'class':'title'}).text
    upvotes = main_post.find('div',attrs={'class':'score unvoted'}).text
    original_poster = main_post.find('a',attrs={'class':'author'}).text
    comments_count = main_post.find('a',attrs={'class':'bylink comments may-blank'}).text
    comment_area = soup.find('div',attrs={'class':'commentarea'})
    comments = comment_area.find_all('div', attrs={'class':'entry unvoted'})
    extracted_comments = []
    for comment in comments: 
        if comment.find('form'):
            #We are now looking for any element with a class of author in the comment, instead of just looking for a tags. 
            #We noticed some comments whose authors have deleted their account shows up with a span tag instead of an a 
            commenter = comment.find(attrs={'class':'author'}).text
            comment_text = comment.find('div',attrs={'class':'md'}).text.strip()
            permalink = comment.find('a',attrs={'class':'bylink'})['href']
            extracted_comments.append({'commenter':commenter,'comment_text':comment_text,'permalink':permalink})
    #Lets put the data in dict 
    post_data = {
        'title':title,
        'no_of_upvotes':upvotes,
        'poster':original_poster,
        'no_of_comments':comments_count,
        'comments':extracted_comments
    }
    return post_data

Let’s test this function out with another URL

parse_comment_page('https://www.reddit.com/r/aww/comments/87bj4l/dad_get_the_ball_plz/')

So our parser for the comments page is ready. Let’s get back to navigating.

Navigating Pages

Let’s call this function in the for loop, because we used to pull the comment page URL. For now, we’ll stick to extracting data from 10 links.

import urllib.request
from bs4 import BeautifulSoup
import json
from time import sleep 
url = "https://www.reddit.com/top/"
#Adding a User-Agent String in the request to prevent getting blocked while scraping 
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
request = urllib.request.Request(url,headers={'User-Agent': user_agent})
html = urllib.request.urlopen(request).read()
soup = BeautifulSoup(html,'html.parser')
#First lets get the HTML of the table called site Table where all the links are displayed
main_table = soup.find("div",attrs={'id':'siteTable'})
comment_a_tags = main_table.find_all('a',attrs={'class':'bylink comments may-blank'})
extracted_data = []
#Remove[:10] to scrape all links 
for a_tag in comment_a_tags[:10]:
    url = a_tag['href']
    if not url.startswith('http'):
        url = "https://reddit.com"+url
    print('Extracting data from %s'%url)
    #Lets wait for 5 seconds before we make requests, so that we don't get blocked while scraping. 
    #if you see errors that say HTTPError: HTTP Error 429: Too Many Requests , increase this value by 1 second, till you get the data. 
    extracted_data.append(parse_comment_page(url))
    #Lets wait for 10 seconds before we make requests, so that we don't get blocked while scraping

Extracting data from https://www.reddit.com/r/pics/comments/87bb1m/an_iranian_teacher_visits_his_cancerstricken/
    Extracting data from https://www.reddit.com/r/gifs/comments/878eka/a_textbook_example_of_how_to_cross_a_road_safely/
    Extracting data from https://www.reddit.com/r/funny/comments/87agu2/special_needs_teachers_put_this_up_today/
    Extracting data from https://www.reddit.com/r/gifs/comments/87a9dz/dog_thrown_from_plane_into_the_abyss_below/
    Extracting data from https://www.reddit.com/r/aww/comments/879b4q/this_is_the_life_we_should_all_aspire_to/
    Extracting data from https://www.reddit.com/r/worldnews/comments/87bbio/facebook_has_lost_100_billion_in_10_days_and_now/
    Extracting data from https://www.reddit.com/r/todayilearned/comments/878ei8/til_in_1994_the_kkk_applied_to_sponsor_a_section/
    Extracting data from https://www.reddit.com/r/aww/comments/87bj4l/dad_get_the_ball_plz/
    Extracting data from https://www.reddit.com/r/gifs/comments/87cw3o/new_silicon_valley_intro_throws_shade_at_facebook/
    Extracting data from https://www.reddit.com/r/mildlyinteresting/comments/879wh7/the_wear_indicators_on_my_tires_show_a_percentage/

The data is quite large, so let’s just look at one record of that we extracted.

print(extracted_data[0])

You can view the output in the gist link here – https://gist.github.com/scrapehero/96840596781006fdb9ba21e4429fc6df

Page Blocked?

If you see errors that say `HTTPError: HTTP Error 429: Too Many Requests` it is because the website has detected you were making many requests or have identified the script as a bot. You can read more on how to prevent getting blacklisted while scraping here.

What’s next

This web scraper is still incomplete. As a result, it can’t save data to a file yet. We will leave that as an exercise to the reader. You can read more about writing files to JSON here https://www.safaribooksonline.com/library/view/python-cookbook-3rd/9781449357337/ in the next part of this web scraping tutorial, we will show you how to use a web scraping framework to build a complete scraper that can scale up locally, save data to files, start, stop and resume execution.

Next Web Scraping Tutorial:
How to scrape Alibaba.com product data using Scrapy Web Scraping Framework