How to scrape Yahoo Finance and extract stock market data using Python & LXML

I tried running your code, but I keep getting hassled by a

SyntaxError: ‘return’ outside function

I think I have your indentation right. Is there any way to post the code somewhere so I can get it straight from the horse’s mouth?

If you don’t feel like writing a scraper I suggest looking at python’s yahoo-finance package. You can get all of the same data with a few lines of code.

Yahoo finance API is not available anymore. I have moved to MarketXLS after this change, much more reliable data.

This code (raw_table_key = table_data.xpath(‘.//td[@class=”C(black)”]//text()’) is not working in python 2.7. So I have changed with this(raw_table_key = table_data.xpath(‘.//td[contains(@class,”C(black)”)]//text()’)). Anyway very nice code and helpful to me. Thanks.

Tried your code and I get :

Fetching data for aapl
Parsing http://finance.yahoo.com/quote/aapl?p=aapl
Writing data to output file

when I go to the output file it is very limited:
{
“”: “172.29”,
“url”: “http://finance.yahoo.com/quote/aapl?p=aapl”,
“ticker”: “aapl”,
“1y Target Est”: 172.29,
“EPS (TTM)”: 8.808,
“Earnings Date”: “2017-10-23 to 2017-10-27”
}

Am I doing something wrong? I use Atom as my editor and running on a Macbook

I am trying to use your code just to get a current price for securities. The slimmed down version looks like this:

from lxml import html
import requests
from exceptions import ValueError
from time import sleep
import json
import argparse
from collections import OrderedDict
from time import sleep

def parse(ticker):
url = “http://finance.yahoo.com/quote/%s?p=%s”%(ticker,ticker)
response = requests.get(url)
parser = html.fromstring(response.text)
summary_table = parser.xpath(‘//div[contains(@data-test,”summary-table”)]//tr’)
summary_data = OrderedDict()
other_details_json_link = “https://query2.finance.yahoo.com/v10/finance/quoteSummary/{0}?formatted=true&lang=en-US&region=US&modules=financialData”.format(ticker)
summary_json_response = requests.get(other_details_json_link)
try:
json_loaded_summary = json.loads(summary_json_response.text)
return json_loaded_summary[“quoteSummary”][“result”][0][“financialData”][“currentPrice”][‘raw’]
except ValueError:
print “Failed to parse json response”
return {“error”:”Failed to parse json response”}

if __name__==”__main__”:
argparser = argparse.ArgumentParser()
argparser.add_argument(‘ticker’,help = ”)
args = argparser.parse_args()
ticker = args.ticker
print parse(ticker)

This works fine except for getting prices of ETFs (e.g. EFA, VWO).

Yahoo provides these prices but the data is slightly different. My question is how to modify your code to work with ETFs. Also, in writing this script how do you view the XML that needs to be parsed since it is dynamically created?

I’m having some trouble getting this working, getting an error on line 12:

C:\Python36\Scripts>python yf.py aapl
File “yf.py”, line 12
print “Parsing %s”%(url)
^
SyntaxError: invalid syntax

Any Ideas to fix this?

Is it possible to enter a list of tickers to have the program generate files for each ticker? Instead of doing it individually?

Hi there,

I tried to run this function parse(ticker) using ticker = ‘APPL’ but when loading for this line:

other_details_json_link = ‘https://query2.finance.yahoo.com/v10/finance/quoteSummary/{0}?formatted=true&lang=en-US&region=US&modules=financialData’.format(‘APPL’)

Also tried this url:

“https://query2.finance.yahoo.com/v10/finance/quoteSummary/{0}?formatted=true&lang=en-US&region=US&modules=summaryProfile%2CfinancialData%2CrecommendationTrend%2CupgradeDowngradeHistory%2Cearnings%2CdefaultKeyStatistics%2CcalendarEvents&corsDomain=finance.yahoo.com”.format(‘AAPL’)

I get: {‘quoteSummary’: {‘result’: None, ‘error’: {‘code’: ‘Not Found’, ‘description’: ‘Quote not found for ticker symbol: APPL’}}}

response is a 404 so I suspect this url doesn’t work anymore?

Cheers,

Ciaran

How does one get the names of all the modules from which data can be extracted?
I am referring to the line of code setting the value of the variable “other_details_json_link”

https://query2.finance.yahoo.com/v10/finance/quoteSummary/INTC?formatted=true&lang=en-US&region=US&modules=Summaryprofile%2cfinancialdata%2crecommendationtrend%2cupgradedowngradehistory%2cearnings%2cdefaultkeystatistics%2ccalendarevents&corsDomain=finance.yahoo.com

Thanks

Is the ‘print’ function the main difference between using Python 2.7 vs 3.6?

I would like to scrap the Statistics and Analysis pages can you please share the code or indicate the changes to add in the code for the summary shared above? Thanks in advance.

Thanks for the great work you do. I’ve been wanting to do something like this for quite some time and you provided me the right motivation. I hope you don’t mind, but I’ve modified your code a bit to add some flexibility. You use the actual webpage people get at Yahoo Finance just for a few pieces of data. For the rest you use an address that returns a nice JSON blob that you use to fill in the rest of the information. It works great but the same custom address doesn’t return much for mutual funds or ETFs. I was able to find a similar address that could be used for mutual funds and ETFs but think a better approach is to just use the publicly known webpage. I was able to manipulate that and produce summary information for stocks (same output as your scipt), mutual funds and ETFs.

The other advantage of doing it this way is that there’s a vast amount of other information available in the JSON blobs that I grab. To find out what is available I suggest using http://beautifytools.com/html-beautifier.php and loading the Yahoo Financial summary page url. Once you click on “Beautify html” you’re presented with a nice tree format of what’s in there. This view will also show where the paths came from for the data I do store.

Here’s the modified code:

from lxml import html
import requests
from time import sleep
import json
import argparse
from collections import OrderedDict
from time import sleep

def matching(string, begTok, endTok):
# Find location of the beginning token
start = string.find(begTok)
stack = []
# Append it to the stack
stack.append(start)
# Loop through rest of the string until we find the matching ending token
for i in range(start+1, len(string)):
if begTok in string[i]:
stack.append(i)
elif endTok in string[i]:
stack.remove(stack[len(stack)-1])
if len(stack) == 0:
# Removed the last begTok so we’re done
end = i+1
break
return end

def parse(ticker):
# Yahoo Finance summary for stock, mutual fund or ETF
url = “http://finance.yahoo.com/quote/%s?p=%s”%(ticker,ticker)
response = requests.get(url, verify=False)
print (“Parsing %s”%(url))
sleep(4)
summary_data = OrderedDict()

# Convert the _context html object into JSON blob to tell if this is an equity, a mutual fund or an ETF
contextStart = response.text.find('"_context"')
contextEnd = contextStart+matching(response.text[contextStart:len(response.text)], '{', '}')

# Store ordered pairs to be written to a file
summary_data.update({'Previous Close':prev_close,'Open':mark_open,'Bid':bid,'Ask':ask,"Day's Range":day_range,'52 Week Range':year_range\
,'Volume':volume,'Avg. Volume':avg_volume,'Net Assets':net_assets,'NAV':nav,'PE Ratio (TTM)':'N/A','Yield':yld,'YTD Return':ytd_return\
,'Beta (3Y Monthly)':beta,'Expense Ratio (net)':exp_rat,'Inception Date':inception_date,'ticker':ticker,'url':url})
return summary_data
except:
print ("Failed to parse json response")
return {"error":"Failed to parse json response"}


if name==”main“:
argparser = argparse.ArgumentParser()
argparser.add_argument(‘ticker’,help = ”)
args = argparser.parse_args()
ticker = args.ticker
print (“Fetching data for %s”%(ticker))
scraped_data = parse(ticker)
print (“Writing data to output file”)
with open(‘%s-summary.json’%(ticker),’w’) as fp:
json.dump(scraped_data,fp,indent = 4)

Hello, I am new to coding I have created a list for all the stocks I would like to scrape, but I do not know how to integrate it into the code that is written.

Hello sir, i just wanted to fetch days range field. what is the best way to do it by scraping. Thanks in advance.

When I run the command to find the stock data I get:

$ python3 yahoofinance_scraper.py APPL

Fetching data for APPL
/usr/lib/python3/dist-packages/urllib3/connectionpool.py:999: InsecureRequestWarning: Unverified HTTPS request is being made to host ‘finance.yahoo.com’. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
warnings.warn(
Parsing http://finance.yahoo.com/quote/APPL?p=APPL
Failed to parse json response
Writing data to output file

What should I do?

I’m having trouble getting a response to my GET request. Anyone else having this difficult with Yahoo Finance?

Wow! You have an amazing site here, and your articles are really informative, consistency is all that’s left now.