Hello, I am looking to get stock and divided data for a list of stocks in python

Question

I am looking to get stock and dividend data for a list of stocks in python. This is what I have but am not getting any data.

import requests
from bs4 import BeautifulSoup

def get_stock_price(stock):
    url = f'https://finance.yahoo.com/quote/{stock}/'
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')
    price_element = soup.find('div', {'class': 'My(6px) Pos(r) smartphone_Mt(6px)'})
    if price_element:
        price = price_element.find('span').text
      return float(price.replace(',', ''))
   return None

def get_dividend_data(stock):
     url = f'https://finance.yahoo.com/quote/{stock}/dividends?p={stock}'
     response = requests.get(url)
     soup = BeautifulSoup(response.text, 'html.parser')
     dividend_element = soup.find('td', {'class': 'Ta(end) Fw(600) Lh(14px)'})
     if dividend_element:
         dividend = dividend_element.text
        return float(dividend.replace(',', ''))
     return None

stocks = ['MSFT', 'LTC', 'AGNC', 'GLAD', 'HRZN', 'ARCC', 'HASI', 'SBR']
for stock in stocks:
    price = get_stock_price(stock)
    dividend = get_dividend_data(stock)
    print(f"Stock: {stock}, Price: {price}, Dividend: {dividend}")

that gives me

Stock: MSFT, Price: None, Dividend: None Stock: LTC, Price: None, Dividend: None Stock: AGNC, Price: None, Dividend: None Stock: GLAD, Price: None, Dividend: None Stock: HRZN, Price: None, Dividend: None Stock: ARCC, Price: None, Dividend: None Stock: HASI, Price: None, Dividend: None Stock: SBR, Price: None, Dividend: None

I had also used

 import pandas_datareader as pdr
import yfinance as yf

formatted_monthly_dividend = {}
stock_data = {}
ticker_symbols_monthly = ['MSFT', 'LTC','AGNC', 'GLAD', 'HRZN', 'ARCC', 'HASI', 'SBR']

def get_stock_data():
    for ticker_symbol in ticker_symbols_monthly:
        # Get the stock data for the current ticker symbol
        stock = yf.Ticker(ticker_symbol)

        # Get the stock info
        info = stock.info
        if info is None:
            print(f"{ticker_symbol} not found!")
        else:
            # Get the current stock price
            price = info.get('regularMarketPrice', 0)
            # Get the dividend yield
            dividend_yield = info.get('dividendYield', 0)
            # Get the market capitalization
            market_cap = info.get('marketCap', 0)


        # Print the stock data
        print(f'Stock for {ticker_symbol}')
        print(f'Stock price: {price}')
        print(f'Dividend yield: {dividend_yield}')
        print(f'Market capitalization: {market_cap}')
        print("\n")

    for ticker_symbol in ticker_symbols_monthly:
        stock = yf.Ticker(ticker_symbol)
        info = stock.info

        try:
            # Get the current stock price
            price = info['regularMarketPrice']

            # Get the dividend yield
            dividend_yield = info['dividendYield']

            annual_dividend = dividend_yield * price
            if dividend_yield is None:
                annual_dividend = 0
            else:
                annual_dividend = dividend_yield * price

            # Add the stock data to the dictionary
            stock_data[ticker_symbol] = (price, annual_dividend)
        except KeyError:
            print(f"{ticker_symbol} not found!")

and this gives me

Stock for MSFT Stock price: 0 Dividend yield: 0.0085 Market capitalization: 2385119281152

Stock for LTC Stock price: 0 Dividend yield: 0.070199996 Market capitalization: 1335027456

Stock for AGNC Stock price: 0 Dividend yield: 0.1552 Market capitalization: 5474593280

Stock for GLAD Stock price: 0 Dividend yield: 0.1015 Market capitalization: 350961664

Stock for HRZN Stock price: 0 Dividend yield: 0.1031 Market capitalization: 366804480

Stock for ARCC Stock price: 0 Dividend yield: 0.1033 Market capitalization: 10246345728

Stock for HASI Stock price: 0 Dividend yield: 0.060500003 Market capitalization: 2458731776

Stock for SBR Stock price: 0 Dividend yield: 0.1172 Market capitalization: 1067059008

MSFT not found! LTC not found! AGNC not found! GLAD not found! HRZN not found! ARCC not found! HASI not found! SBR not found!

I don't know what I am doing and most of this code came from bard. I have tried a few other ways and it just doesn't seem to work. Any help would be appreciated. Thanks and have a good one.

Answer 1

Of the two partial methods you've got for getting your data, I would suggest sticking with the second one. Method one is scrubbing the data directly from the html, while method two uses the yfinance package to access the yahoo API directly. If the representation of the yahoo page changes significantly, which could happen at any time as front ends often change without warning, scrubbing method one could break. In contrast the API is likely to stay stable and support backwards compatibility even if new versions are released.

I would first scale back your example as simple as possible. You shouldn't jump right into iterating through all the stocks you want. Get one stock working, expand from there. When in doubt stick to KISS (keep it simple stupid).

First off you're not using pandas so don't import it. Start with loading one stock and printing the info so you know what structure the data is in.

import yfinance as yf

ticker_symbol = 'MSFT'
stock = yf.Ticker(ticker_symbol)
info = stock.info
print(info)

Looks like it's a dictionary, to access the current price and the dividend, we just need to know their keys.

import yfinance as yf

ticker_symbol = 'MSFT'
stock = yf.Ticker(ticker_symbol)
info = stock.info
print(info.keys())

Now using the keys, we can pull out the data you are looking for

ticker_symbol = 'MSFT'
stock = yf.Ticker(ticker_symbol)
info = stock.info

current_price = info['currentPrice']
dividend_yeild = info['dividendYield']

print(f'''
    Stock: {ticker_symbol}, 
    Current price: {current_price}, 
    Dividend yeild: {dividend_yeild}
    ''')

Which gives me:

   Stock: MSFT, 
   Current price: 321.19, 
   Dividend yeild: 0.0085

Now once you have the basics working, you can put it back into functions and loops to iterate through and get all the data you need.