How do I scrape multiple URLs using Selenium web driver?
Below is my code scraping a title from one amazon link. I want to add more links and scrape multiple titles. Thank you.
import selenium
from selenium import webdriver
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
links = ("https://www.amazon.com/Nintendo-Switch-Neon-Blue-Joy%E2%80%91/dp/B07VGRJDFY/?_encoding=UTF8&pd_rd_w=tUivq&pf_rd_p=cc3d6e86-d477-4b0e-8ce3-a46ea2292f65&pf_rd_r=C62K87403EP0W3NW9T3B&pd_rd_r=5d895c99-d866-4e36-b06c-5e9d0a5d48d4&pd_rd_wg=vuVXa&ref_=pd_gw_crs_zg_bs_468642")
driver.get(links)
driver.implicitly_wait(5)
print(driver.title)
driver.quit()
I did something similar a while ago, I used loops for that instance. It is longer and more complicated than this, but here is the functional, shortened and customised version for you:
The program parses multiple websites and uses them accordingly. But remember that depending on your parsing process, it takes time. You can always use multiprocessing, but this program was not time-critical.