I've taken a few coding subjects in uni and am trying to analyse tennis statistics by learning selenium which is completely new to me.
The page I'm using is here (https://www.atptour.com/en/scores/results-archive?year=2021) and I'm followinig a guide from this website here (https://www.scrapingbee.com/blog/selenium-python/ , https://www.scrapingbee.com/blog/practical-xpath-for-web-scraping/). The particular problem I'm having is in the second guide website under the subtitle "E-commerce product data extraction".
My Goal is to loop through the tournaments and extract the links located with the 'Results' button, but I'm having trouble as my program is just giving me an emptylist.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
DRIVER_PATH = "C:\Program Files (x86)\chromedriver.exe"
#driver = webdriver.Chrome(executable_path=DRIVER_PATH)
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")
driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
#driver.get("https://www.nintendo.com/")
#print(driver.page_source)
#driver.quit()
# 1 Data Collection
# 1.1 Find Links to All Tournaments
tournaments_2021_url = "https://www.atptour.com/en/scores/results-archive?year=2021"
#tournament_class = "tourney-result"
driver.get(tournaments_2021_url) # print(driver.page_source)
tournaments_2021_url_list = driver.find_elements_by_xpath("//a[@class='button-border']")
print("\n tournament urls \n")
print(tournaments_2021_url_list)
print(len(tournaments_2021_url_list))
driver.quit()
# 1.2 For Each Tournament, Find Links to Each Match
# 1.3 For Each Match, Extract Relevant Statistics
I would expect to have a list of elements or some weird objects and be able to extract the links, but instead I get an empty list with len 0. Thanks for any help.
I took your code and ran it and it's good. It does what it's supposed to. Thus, my advice is to run it thru a debugger and step thru to make sure everything goes as it's supposed to. Remove the headless option as well so you can visually confirm. Check your chrome browser version and make sure it matches with the chromedriver you're using. (although it should give you an error message if the versions don't match.) Finally, if all else fails, try it using another browser, firefox for example, and the appropriate geckodriver.