web scrap stock data from Reuters

Question

I am a programming beginner and trying to extract key metric data (e.g. Beta) for a stock from Reuters. However, it always come back as blank.

my codes are like this:

from bs4 import BeautifulSoup as bs
import requests
import re

url = 'https://www.reuters.com/markets/companies/TSLA.OQ/key-metrics/price-and-volume'
page = requests.get(url)
bs1 = bs(page.text, 'html.parser')

beta=bs1.find_all('th', class_ ='text__text__1FZLe text__dark-grey__3Ml43 text__regular__2N1Xr text__body__yKS5U body__base__22dCE body__body__VgU9Q',text=re.compile('Beta'))
print(beta)

I know it is not correct but I cannot figure out what to do. please help. Ultimate I want to be extract the Beta info for a stock from Reuters. thank you for your help!!!

Answer 1

You can scrape the site (without inspecting the javascript/json) using Selenium, using bs4 from my previous answer but you can use seleniums functions instead.

from selenium import webdriver
from bs4 import BeautifulSoup as bs


# Initiate webdriver
driver = webdriver.Firefox()

# Fetch the web page
driver.get('https://www.reuters.com/markets/companies/TSLA.OQ/key-metrics/price-and-volume')

# Convert the driver page source to a soup object
soup = bs(driver.page_source, 'html.parser')

# Find the table you want to scrape
table = soup.find('table', attrs={'aria-label':'KeyMetrics'})

# Locate the Keys and Value for each of the rows
keys = [i.text for i in table.select('tbody tr th') if i]
values = [i.text for i in table.select('tbody tr td') if i]

# Convert the two lists into a dictionary for a neater output
data = dict(zip(keys,values))

driver.quit()
print(data)

This will return:

{'Price Closing Or Last Bid': '699.20', 'Pricing Date': 'Jul 05', '52 Week High': '1,243.25', '52 Week High Date': 'Nov 04', '52 Week Low': '620.50', '52 Week Low Date': 'Jul 08', '10 Day Average Trading Volume': '31.36', '3 Month Average Trading Volume': '602.72', 'Market Capitalization': '724,644.30', 'Beta': '2.13', '1 Day Price Change': '2.55', '5 Day Price Return (Daily)': '-4.84', '13 Week Price Return (Daily)': '-35.93', '26 Week Price Return (Daily)': '-39.18', '52 Week Price Return (Daily)': '2.99', 'Month To Date Price Return (Daily)': '3.83', 'Year To Date Price Return (Daily)': '-33.84', 'Price Relative To S&P500 (4 Week)': '5.95', 'Price Relative To S&P500 (13 Week)': '-24.33', 'Price Relative To S&P500 (26 Week)': '-23.90', 'Price Relative To S&P500 (52 Week)': '16.99', 'Price Relative To S&P500 (YTD)': '-17.69'}

Answer 2

Here's one way of collecting the data you need:

from bs4 import BeautifulSoup as bs 
import requests
import re

url = 'https://www.reuters.com/markets/companies/TSLA.OQ/key-metrics/price-and-volume'
page = requests.get(url)
soup = bs(page.text, 'html.parser')

# Locate the Table you wish to scrape
table = soup.select_one('table.table__table__2px_A')

# Locate the Keys and Value for each of the rows
keys = [i.text for i in table.select('tr th') if i]
values = [i.text for i in table.select('tr td') if i]

# Convert the two lists into a dictionary for a neater output
data = dict(zip(keys,values))

This will return:

{'% Change': '671.00',
 'Brent Crude Oil': '-1.40%Negative',
 'CBOT Soybeans': '1,626.00',
 'Copper': '111.91',
 'Future': '1,805.20',
 'Gold': '-0.57%Negative',
 'Last': '+0.35%Positive'}