I'm new to the web scraping game and currently working on a proejct to scrape the reviews of this product (https://shopee.com.my/Women-Running-Shorts-Elastic-Waist-Running-Yoga-Short-Woman-Sports-Short-Plus-size-Gym-Fitness-Shorts-Sportswear-i.52936573.21378274920?sp_atk=57449ebe-a59d-4f29-a2ce-9c7c302d874d&xptdk=57449ebe-a59d-4f29-a2ce-9c7c302d874d)
I need to get the date, name and coments however I'm facing two issues and google wans't much of a help
- when I call my var it returns none
divs = product_rating.find_all('div', class_= 'shopee-product-rating')
- I need the class name of the comments however the only element I can find style. I'm not sure how else to call it the element I need its class
full code
#imported libraries
# to parse html and xml files' content
from bs4 import BeautifulSoup
# to manage requests with the server
import requests
import time
import datetime
import smtplib
url = 'https://shopee.com.my/Women-Running-Shorts-Elastic-Waist-Running-Yoga-Short-Woman-Sports-Short-Plus-size-Gym-Fitness-Shorts-Sportswear-i.52936573.21378274920?sp_atk=57449ebe-a59d-4f29-a2ce-9c7c302d874d&xptdk=57449ebe-a59d-4f29-a2ce-9c7c302d874d'
headers = {'User-Agent': 'Mozilla/5.0'}
#to check if my request was accepted 200 means yes
response = requests.get(url, headers = headers)
response
pages = response.content
soup = BeautifulSoup(pages, 'html.parser')
soup
#issue 1
product_rating = soup.find('div', class_='shopee-product-comment-list')
print(product_rating)
divs = product_rating.find_all('div', class_= 'shopee-product-rating')
for div in divs:
name = div.find('div', class_= 'shopee-product-rating__author-name')
date = div.find('div', class_= 'class="shopee-product-rating__time"')
#issue 2
comment = div.find('div', class_= 'style' )
for the first problem I have tried using deffirent classes but all return none which causes the error in the 2nd lline. I also tried importing selenium but didn't fix the issue.
for the second one I couldn't find a solution