I am very new to coding, just started yesterday because I am a finance grad student and I need quicker ways of extracting data than by hand.
I am trying to write a script to extract every data point from the CNN Fear and Greed Timeline (historic) Chart found on this website, https://www.cnn.com/markets/fear-and-greed.
The current script I have written is as follows, Bing AI chatbot helped me write it.
import requests
from bs4 import BeautifulSoup
url = 'https://www.cnn.com/markets/fear-and-greed'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
# find the script tag that contains the chart data
script_tag = soup.find('script', string=lambda t: 'highcharts' in t.lower())
# extract the chart data from the script tag
chart_data = script_tag.string.split('series: [')[1].split(']')[0]
# parse the chart data into a list of values
values = [int(x.split(',')[1]) for x in chart_data.split('},{')]
print(values)
I have it saved as "fearandgreedCNNgraph.py" as an "all files" instead of the default ".txt file"
When I enter it into Anaconda, I get this as a result.
import requests
from bs4 import BeautifulSoup
url = 'https://www.cnn.com/markets/fear-and-greed'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
# find the script tag that contains the chart data
script_tag = soup.find('script', string=lambda t: 'highcharts' in t.lower())
# extract the chart data from the script tag
chart_data = script_tag.string.split('series: [')[1].split(']')[0]
# parse the chart data into a list of values
values = [int(x.split(',')[1]) for x in chart_data.split('},{')]
print(values)
I asked Bing AI and it said that I needed to use the website's source code to find the script_tag containing the chart data and it told me to "control+F Highcharts, chart, or data" to find the script_tag that I'm looking for. The only problem is this makes 0 sense to me, the source code of the website is pages long and there's hundreds of results for "chart" and "data" and no results for "highcharts."
Does anyone know how I can modify this code to provide me with the datapoints from the graph in question?
Thank you!
If anyone is wondering what I need this data for, I'm attempting to build an EPS forecasting model that will then ideally be used in another model to help forecast price movements in stocks. I am trying to see if Fear and Greed Index data will be a good addition to my initial model, hence why I want to extract the data. There's a bit more to it than that, but that's the gist of it. My end goal with this is to use it, or an extension of this model, as part of my PhD project.
I attempted to use Bing AI chatbot to help me write the code, it did not work that well as I need to find a script tag containing the chart data, but I have no idea how to do that or what to look for and the chatbot is not helpful with this.