BeautifulSoup can't find my table because of some weird string

Question

Hi I have been trying to get a table from wunderground using BeautifulSoup but it just doesn't work.

I think it could be for the starnge string next to the table header but i can´t fix it.

Here is my code:

from bs4 import BeautifulSoup
import requests

url='https://www.wunderground.com/history/daily/LEMD/date/2020-10-21'

html_content = requests.get(url).text

soup = BeautifulSoup(html_content, "html.parser")

table = soup.find("table", {"class": "mat-table cdk-table mat-sort ng-star-inserted"})

table_data = table.tbody.find_all("tr")

and the error:

Traceback (most recent call last):
  File "weather_poc.py", line 12, in <module>
    table_data = table.tbody.find_all("tr")
AttributeError: 'NoneType' object has no attribute 'tbody'

Answer 1

The data you see is loaded from external URL via JavaScript. You can use requests/json module to load it. For example:

import json
import requests
import pandas as pd


url = 'https://api.weather.com/v1/location/LEMD:9:ES/observations/historical.json?apiKey=6532d6454b8aa370768e63d6ba5a832e&units=e&startDate=20201021&endDate=20201021'
data = requests.get(url).json()

# uncomment this line to print all data:
# print(json.dumps(data, indent=4))

df = pd.json_normalize(data['observations'])
df.to_csv('data.csv', index=False)

Creates data.csv (screenshot from LibreOffice):