> ## Content Index
> Fetch the complete content index at: https://serpapi.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Scrape Yahoo! Finance Home Page with Python
- URL: https://serpapi.com/blog/scrape-yahoo-finance-home-page-with-python/
- Published: 2021-08-21T07:30:13.000Z
- Updated: 2023-04-20T18:29:12.000Z
- Description: Scrape news, header, and right side stock results from Yahoo! Finance Home page using Python.
- Author: Dmitriy Zub
- Tags: Yahoo, Python, Web Scraping

## **Intro**

Currently, we don't have an API that supports extracting data from the Yahoo! Finance page.

This blog post is to show you way how you can do it yourself as [we noticed quite a few questions on Stackoverflow about scraping Yahoo! Finance](https://stackoverflow.com/search?q=yahoo+finance+%5Bbs4%5D) and decided to create a blog about it. The provided DIY solution can be used for personal use.

## What will be scraped

![image](https://res.cloudinary.com/practicaldev/image/fetch/s--zJwS1uZV--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vah9gri016xgdtcal6j6.png)

### Difference between existing Yahoo! Finance parsers

Since [Yahoo! Finance deprecated their API](https://towardsdatascience.com/free-stock-data-for-python-using-yahoo-finance-api-9dafd96cad2e#:~:text=Yahoo%20deprecated%20their%20Finance%20API,data%20in%20the%20DataFrame%20format.) there're a lot of custom solutions out there. The differences between those parsers and the one you'll see below are:

- Firstly, this blog post is more *educational* rather than a complete solution.
- Secondly, all/most of the [existing Yahoo! Finance parsers](https://github.com/search?q=yahoo+finance) (at least those that I looked at) extract only ticket(s) data, without news results that Yahoo! Finance has.

## Process

You can access and navigate through header stock results, news results `json` string data at jsonblob [here](https://jsonblob.com/7a228dbe-f129-11eb-99e5-250f2f4d92cd).

[Dadroit](https://dadroit.com/) was used to preview `json` and copy paths to a specific key, value. Here's what UI looks like:

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2021/08/image.png)

### Get header stock results

Firstly, we need to locate where the data is located since we can't just use `CSS` selectors to extract the data because it is dynamically updating from the server.

The way I found where the data is located is very simple. Just copied the stock name and pasted it in the source code and checked if there's a match under the `<script>` tags.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2021/08/image-1.png)

After several matches were found in the same place under the `<script>` tag I began to look where the `<script>` tag starts and ends in order to extract the correct one using regular expression. Here's the start of the needed `<script>` tag.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2021/08/image-2.png)

And the end of the `<script>` tag.

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2021/08/image-3.png)

Then, regular expression comes into play to extract JSON string from the `<script>` tag. Here's a screenshot to see what is being captured ([link to regular expression](https://regex101.com/r/IJloEU/2)):

![](https://storage.ghost.io/c/a5/00/a5004977-0dd2-4bcd-9292-dd0e05d4c59e/content/images/2021/08/image-4.png)

Note: There's obviously a better `regex` that could be used, I used the easiest, a very basic one.

Next is to extract the data and convert it to iterable JSON string since `re.findall()` returns a `list`, `''.join()` making it a string to use `json.loads()` in the next step.

```python
matched_string = ''.join(re.findall(r'root\.App\.main = (.*);\n+}\(this\)\);\n+</script>', str(all_script_tags)))
matched_string_json = json.loads(matched_string)
```

After that, the for key, value in `dict(...).items()` option was used to make the code shorter and easier.

[.items()](https://www.programiz.com/python-programming/methods/dictionary/items) returns a key, value pair so I don't have to specify which stock symbol to iterate over.

It will iterate over each available stock from the response instead and will substitute the correct stock **key** symbol with appropriate **value** data.

```python
for key, value in dict(matched_string_json['context']['dispatcher']['stores']['StreamDataStore']['quoteData']).items():
   # extracting header stocks data
   # iterating over "value" variable

```

Because `json` response contains unique stock **key** symbols and creating a different function for each stock symbol will lead to a lot of code, and if some symbol will be changed in the future, it will throw an error.  
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5w88tmrnnyyb85rdmdxr.png)

*If you want to iterate over specific stocks, then you can skip this step and write few functions just to iterate over specific stock results.*

Example of Header stock response:

```python
{
  "sourceInterval": 30,
  "exchange": "NYB",
  "regularMarketTime": {
    "raw": 1627325996,
    "fmt": "2:59PM EDT"
  },
  "shortName": "10-Yr Bond",
  "exchangeTimezoneName": "America/New_York",
  "regularMarketChange": {
    "raw": 0,
    "fmt": "0.0000"
  },
  "regularMarketPreviousClose": {
    "raw": 1.276,
    "fmt": "1.2760"
  },
  "exchangeTimezoneShortName": "EDT",
  "exchangeDataDelayedBy": 30,
  "priceHint": 4,
  "regularMarketPrice": {
    "raw": 1.276,
    "fmt": "1.2760"
  },
  "triggerable": false,
  "gmtOffSetMilliseconds": -14400000,
  "firstTradeDateMilliseconds": -252356400000,
  "region": "US",
  "marketState": "REGULAR",
  "quoteType": "INDEX",
  "symbol": "^TNX",
  "language": "en-US",
  "market": "us24_market",
  "regularMarketChangePercent": {
    "raw": 0,
    "fmt": "0.00%"
  },
  "fullExchangeName": "NYBOT",
  "tradeable": false
}

```

### Get Top News Results

```python
for top_news_result_index, top_news in enumerate(matched_string_json_stream):
   # extracting data

```

### Get Top News Video Results

```python
for top_news_video_index, top_news_video in enumerate(matched_string_json_video):
   # extracting data

```

### Get Multiuse News Results

```python
for multiuse_index, multiuse_news in enumerate(matched_string_json_multiuse):
  # extracting data

```

### Get Scrolling News Results

```python
for yahoo_news_index in matched_string_json['context']['dispatcher']['stores']['StreamStore']['streams']['mega.c']['data']['stream_items']:
   # extracting data

```

### Get Right Side Stocks Results

In the dev tools network tab, you can see requests being sent to the server with `GET` method, thus you can call them directly to get a `json` string.

In total, there're 11 URLs (*might be changed in the future*) but you can make a 1 request call with additional symbols added to the URL string without need to make 11 URL calls `async`. Look at `yahoo_finance_urls.py`.

*Note: `User-Agent` needs to be used, otherwise, it will throw a 403 forbidden.* [What is my user-agent](https://www.whatismybrowser.com/detect/what-is-my-user-agent)

The approach was the same as in the previous section of the post, where the received response was converted to a `json` string for further manipulation.

Headers tab  
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kqur9ib1esm3xyq6jbo9.png)

Preview tab  
![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/utevb6jfevkqxv6jyojq.png)

### File Structure

```
Yahoo_Finance_Main.py
├── Yahoo_Finance_Header_Stocks_News.py
├── Yahoo_Finance_Right_Side_Stocks.py
└── Yahoo_Finance_Urls.py

```

### Full Code

yahoo\_finance\_main.py

```python
from yahoo_finance_right_side_stocks import yahoo_get_right_side_stocks

from yahoo_finance_header_stocks_news import (
    yahoo_get_header_stock_data,
    yahoo_get_top_news_data,
    yahoo_get_top_news_video_results,
    yahoo_get_multiuse_news_results,
    yahoo_get_news_results,
)

yahoo_get_header_stock_data()
yahoo_get_top_news_data()
yahoo_get_top_news_video_results()
yahoo_get_multiuse_news_results()
yahoo_get_news_results()

yahoo_get_right_side_stocks()

```

yahoo\_finance\_header\_stocks\_news.py

```python
import requests, lxml, json, re, datetime
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
                  "(KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get('https://finance.yahoo.com/', headers=headers)
soup = BeautifulSoup(html.text, 'lxml')

all_script_tags = soup.select('script')

# https://regex101.com/r/IJloEU/2
matched_string = ''.join(re.findall(r'root\.App\.main = (.*);\n+}\(this\)\);\n+</script>', str(all_script_tags)))
matched_string_json = json.loads(matched_string)

def yahoo_get_header_stock_data():
    for key, value in dict(matched_string_json['context']['dispatcher']['stores']['StreamDataStore']['quoteData']).items():
        symbol = value['symbol']
        exchange = value['exchange']
        full_exchange_name = value['fullExchangeName']

        try:
            short_name = value['shortName']
        except:
            short_name = 'no shorten name'

        exchange_time_zone_name = value['exchangeTimezoneName']
        regular_market_change = value['regularMarketChange']['fmt']
        regular_market_previous_close = value['regularMarketPreviousClose']['fmt']
        regular_market_price = value['regularMarketPrice']['fmt']
        regular_market_change_percent = value['regularMarketChangePercent']['fmt']
        market_state = value['marketState']
        market = value['market']
        quote_type = value['quoteType']

        print(f'Symbol: {symbol}\n'
              f'Short name: {short_name}\n'
              f'Exchange: {exchange}\n'
              f'Full exchange name: {full_exchange_name}\n'
              f'Exchange timezone: {exchange_time_zone_name}\n'
              f'Market state: {market_state}\n'
              f'Market name: {market}\n'
              f'Quote type: {quote_type}\n'
              f'Market price: {regular_market_price}\n'
              f'Market change: {regular_market_change}\n'
              f'Market % change: {regular_market_change_percent}\n'
              f'Market previous close: {regular_market_previous_close}\n')

def yahoo_get_top_news_data():
    matched_string_json_stream = matched_string_json['context']['dispatcher']['stores']['ThreeAmigosStore']['data']['ntk']['stream']

    for top_news_result_index, top_news in enumerate(matched_string_json_stream):
        teaser = top_news['editorialContent']['teaser']
        title = top_news['editorialContent']['title']

        try:
            source = top_news['editorialContent']['content']['provider']['displayName']
        except:
            source = None

        try:
            source_site_link = top_news['editorialContent']['content']['provider']['url']
        except:
            source_site_link = None

        try:
            canonical_url = top_news['editorialContent']['content']['canonicalUrl']['url']
        except:
            canonical_url = None

        try:
            canonical_url_website = top_news['editorialContent']['content']['canonicalUrl']['site']
        except:
            canonical_url_website = None

        try:
            click_through_url = top_news['editorialContent']['content']['clickThroughUrl']['url']
        except:
            click_through_url = None

        try:
            click_through_url_website = top_news['editorialContent']['content']['clickThroughUrl']['site']
        except:
            click_through_url_website = None

        print(f'News result number: {top_news_result_index}\n'
              f'Teaser: {teaser}\n'
              f'Title: {title}\n'
              f'Source: {source}\n'
              f'Source website: {source_site_link}\n'
              f'Canonical URL: {canonical_url}\n'
              f'Canonical URL source: {canonical_url_website}\n'
              f'Click through URL: {click_through_url}\n'
              f'Click through website: {click_through_url_website}\n')

        for resolution in top_news['editorialContent']['thumbnail']['resolutions']:
            thumbnail_size = resolution['tag']
            thumbnail_link = resolution['url']
            print(f'{thumbnail_size} {thumbnail_link}')

def yahoo_get_top_news_video_results():
    matched_string_json_video = matched_string_json['context']['dispatcher']['stores']['ThreeAmigosStore']['data']['videos']['stream']

    for top_news_video_index, top_news_video in enumerate(matched_string_json_video):
        video_title = top_news_video['content']['title']
        video_summary = top_news_video['content']['summary']
        video_duration_not_fixed = top_news_video['content']['duration']

        # seconds converted to minutes
        video_duration_fixed_time = datetime.timedelta(seconds=video_duration_not_fixed)
        video_publication_date = top_news_video['content']['pubDate']
        vide_provider_name = top_news_video['content']['provider']['displayName']
        video_canonical_url = top_news_video['content']['canonicalUrl']['url']
        video_click_through_url = top_news_video['content']['clickThroughUrl']['url']

        print(f'Video number: {top_news_video_index}\n'
              f'Title: {video_title}\nSummary: {video_summary}\n'
              f'Duration: {video_duration_fixed_time}\n'
              f'Publication date: {video_publication_date}\n'
              f'Provider: {vide_provider_name}\n'
              f'Canonical URL: {video_canonical_url}\n'
              f'Click through URL: {video_click_through_url}\n')

        for resolution in top_news_video['content']['thumbnail']['resolutions']:
            thumbnail_size = resolution['tag']
            thumbnail_link = resolution['url']
            print(f'{thumbnail_size} {thumbnail_link}')

def yahoo_get_multiuse_news_results():
    matched_string_json_multiuse = matched_string_json['context']['dispatcher']['stores']['ThreeAmigosStore']['data']['multiuse']['stream']

    for multiuse_index, multiuse_news in enumerate(matched_string_json_multiuse):

        multiuse_title = multiuse_news['content']['title']
        multiuse_content_type = multiuse_news['content']['contentType']
        multiuse_summary = multiuse_news['content']['summary']
        multiuse_provider_name = multiuse_news['content']['provider']['displayName']
        multiuse_provider_url = multiuse_news['content']['provider']['url']
        multiuse_canonical_url = multiuse_news['content']['canonicalUrl']['url']
        multiuse_click_through_url = multiuse_news['content']['clickThroughUrl']['url']
        
        print(f'Multiuse news number: {multiuse_index}\n'
              f'Title: {multiuse_title}\n'
              f'Content type: {multiuse_content_type}\n'
              f'Summary: {multiuse_summary}\n'
              f'Provider: {multiuse_provider_name}\n'
              f'Provider URL: {multiuse_provider_url}\n'
              f'Canonical URL: {multiuse_canonical_url}\n'
              f'Click through URL: {multiuse_click_through_url}\n')

        for resolution in multiuse_news['content']['thumbnail']['resolutions']:
            thumbnail_size = resolution['tag']
            thumbnail_link = resolution['url']
            print(f'{thumbnail_size} {thumbnail_link}')

def yahoo_get_news_results():
    for yahoo_news_index in matched_string_json['context']['dispatcher']['stores']['StreamStore']['streams']['mega.c']['data']['stream_items']:
        title = yahoo_news_index['title']
        summary = yahoo_news_index['summary']
        news_property = yahoo_news_index['property']
        source = yahoo_news_index['publisher']
        original_publication_url = yahoo_news_index['url']
        yahoo_url = f"https://finance.yahoo.com{yahoo_news_index['link']}"

        try:
            thumbnail_medium = yahoo_news_index['images']['img:440x246']['url']
        except:
            thumbnail_medium = None

        try:
            thumbnail_small = yahoo_news_index['images']['img:220x123']['url']
        except:
            thumbnail_small = None

        print(f'Title: {title}\n'
              f'Summary: {summary}\n'
              f'Property: {news_property}\n'
              f'Source: {source}\n'
              f'Original URL: {original_publication_url}\n'
              f'Yahoo URL: {yahoo_url}\n'
              f'Medium thumbnail: {thumbnail_medium}\n'
              f'Small thumbnail: {thumbnail_small}\n')

```

yahoo\_finance\_urls.py

*Note: Stock symbols are changing each day, and that's the reason why they're currently hardcoded to give some sort of "flexibility" to change to whatever stock symbol is needed.*

*These stock symbols can be intercepted since they are loaded over time, but at the moment I haven't figured out how to make it work.*

```python
import requests

def yahoo_finance_right_stock_urls():

    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
                      "(KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
    }
    
    '''
    Symbols below will not be exactly the same as on Yahoo Finance Home Page.
    
    If you want to track specific tickets, leave it as it is and it will work,
    otherwise "symbols" need to be changed to the one on the Yahoo Finance Home page.

    In the further blog posts this might be changed to auto ticket detection.
    '''

    symbols = ','.join([
        "BTC-USD",
        "ETH-USD",
        "USDT-USD",
        "BNB-USD",
        "ADA-USD",

        "ETSY",
        "FSLY",
        "GM",
        "ROKU",
        "UBER",

        "AMD",
        "HOOD",
        "AMC",
        "F",
        "GM",

        "AZZVF",
        "HOOD",
        "EXPI",
        "COUR",
        "NVAX",

        "ZY",
        "CDLX",
        "OCFT",
        "CAR",
        "CGRY",

        "HSSIX",
        "HSSCX",
        "FFBFX",
        "PRGTX",
        "HSSAX",

        "QQQ",

        "BTCUSD",
        "ETHUSD",
        "EURUSD=X",
        "JPY=X",
        "GBPUSD=X"
        
        "HSSIX",
        "HSSCX",
        "FFBFX",
        "PRGTX",
        "HSSAX"
        
        "ES=F",
        "YM=F",
        "NQ=F",
        "RTY=F",
        "ZB=F",

        "GE220121C00010000",
        "GE220121C00012000",
        "GE220121C00015000",
        "GE220121P00007000",
        "GE220121P00010000",

        "LLNW210806C00001000",
        "OBSV210820C00010000",
        "OBSV210820C00012500",
        "OBSV210820C00015000"
    ])

    params = {
        'formatted': "true",
        'crumb': 'FI5oDlMl7HO',
        'lang': 'en-US',
        'region': 'US',
        'symbols': f"{symbols}",
        'fields': 'symbol,shortName,longName,regularMarketPrice,regularMarketChange,regularMarketChangePercent',
        'corsDomain': 'finance.yahoo.com'
    }

    yahoo_urls = requests.get('https://query2.finance.yahoo.com/v7/finance/quote', params = params, headers = headers).text

    return yahoo_urls

```

### Extracting Right Side Stocks

```python
import json
from yahoo_finance_urls import yahoo_finance_right_stock_urls

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
                  "(KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

def yahoo_get_right_side_stocks():

    for right_side_stock_result in json.loads(yahoo_finance_right_stock_urls())['quoteResponse']['result']:
        stock_symbol = right_side_stock_result['symbol']
        short_name = right_side_stock_result['shortName']
        region = right_side_stock_result['region']
        regular_market_time = right_side_stock_result['regularMarketTime']['fmt']
        exchange_time_zone_name = right_side_stock_result['exchangeTimezoneName']
        market_state = right_side_stock_result['marketState']
        quote_type = right_side_stock_result['quoteType']

        try:
            quote_source_name = right_side_stock_result['quoteSourceName']
        except:
            quote_source_name = None

        market = right_side_stock_result['market']
        regular_market_price = right_side_stock_result['regularMarketPrice']['fmt']
        regular_market_change = right_side_stock_result['regularMarketChange']['fmt']
        regular_market_change_percent = right_side_stock_result['regularMarketChangePercent']['fmt']

        print(f'Quote type: {quote_type}\n'
              f'Quote source name: {quote_source_name}\n'
              f'Symbol: {stock_symbol}\n'
              f'Short name: {short_name}\n'
              f'Region: {region}\n'
              f'Regular market time: {regular_market_time}\n'
              f'Exchange time zone: {exchange_time_zone_name}\n'
              f'Market: {market}\n'
              f'Market state: {market_state}\n'
              f'Market price: {regular_market_price}\n'
              f'Price change: {regular_market_change}\n'
              f'Price % change: {regular_market_change_percent}\n')

```

## Links

[Code in the online IDE](https://replit.com/@DimitryZub1/Scrape-Yahoo-Finance-Home-Page-with-Python#main.py) • [Dadroit](https://dadroit.com/) • [treelib](https://treelib.readthedocs.io/en/latest/#basic-usage)