Hello everyone I want to scrape this website but I need to rotate proxies since I always get blocked. [Please note that I am a beginner ]
Also I need to use Selenium webdriver (I tried to use only the requests library with BeautifulSoup but I get the following error 'Please enable JS and disable adblocker' , I also heard that I could use the request-html library [a more recent library , that enables javascript] I tried but with no success... ).
Moreover I use the undetected-chromedriver from this library (I read on forums that this library allowed a better headers optimization [am I right ? ])
So my problematic is the following : how can I rotate proxies using paid proxies (that need a username and a password , I bought datacenter proxies from SmartProxy ) with Selenium undetected chrome driver ?
Here is what I tried :
My proxy list ( paid proxies ) look something like this (they are in a text file called : smartproxies.txt )
http://USERNAME:[email protected]:10001
http://USERNAME:[email protected]:10002
http://USERNAME:[email protected]:10003
http://USERNAME:[email protected]:10004
http://USERNAME:[email protected]:10005
Then I used a for loop to iterate through all my proxies and go to this page : https://api.ipify.org/ a page that allows you to know your ip (just to see if i have correctly rotated proxies ) here is my script :
import undetected_chromedriver as uc
def sele(url, using_proxies=True):
with open('smartproxies.txt', 'r') as f:
proxies_list = f.read().splitlines()
for proxy in proxies_list:
options = uc.ChromeOptions()
options.add_argument(f'--proxy-server={proxy}')
driver = uc.Chrome(options=options)
driver.get(url)
sleep(3)
driver.close()
if __name__ == "__main__":
sele(url="https://api.ipify.org/")
However when the script runs , the page that open at each iteration of the loop shows me the same IP address ! I don't see my error
Also , is it really necessary to use the undetected chromedriver for that or would a regular chrome driver work ?
Thanks !