Can someone explain me how to solve this problems? After I try to run this code with IP rotate it stop work even on static IP. Metod below open cosdna.com and try to search for Ingridient of products. I get IP ban after i try to scrape about 20 products, so i try to rotate my IP to solve this problem. Now it doesent work with and without IP rotate.
@SuppressWarnings("deprecation")
public static String findInCosDNA(String name) throws SessionNotCreatedException {
StringBuilder sklad = new StringBuilder();
sklad.append("Skład/");
System.setProperty("webdriver.gecko.driver",
"C:\\...\\geckodriver.exe");
DesiredCapabilities capabilities = DesiredCapabilities.firefox();
capabilities.setCapability("marionette", true);
capabilities.setCapability(CapabilityType.PROXY, getRandomProxy());
WebDriver driver = new FirefoxDriver(capabilities);
try {
driver.manage().timeouts().pageLoadTimeout(30, TimeUnit.SECONDS);
driver.get(COSDNA);
WebElement textInput = driver.findElement(By.className("input-gen"));
.
.
.
} catch (NoSuchElementException e) {
sklad.setLength(0);
} catch (TimeoutException e) {
sklad.setLength(0);
} catch (WebDriverException e) {
sklad.setLength(0);
}
driver.close();
return sklad.toString();
}
//proxyList is a list of String in "IP:PORT" format
public static Proxy getRandomProxy() {
String IP = proxyList.get(getRandomNumberInRange(0, Proxy.size() - 1));
Proxy prx = new Proxy();
prx.setHttpProxy(IP).setFtpProxy(IP).setSslProxy(IP);
return prx;
}
Error without proxy:
lip 27, 2018 12:18:08 AM org.openqa.selenium.remote.DesiredCapabilities firefox
INFO: Using 'new FirefoxOptions()' is preferred to 'DesiredCapabilities.firefox()'
1532643488955 geckodriver INFO geckodriver 0.21.0
1532643488963 geckodriver INFO Listening on 127.0.0.1:27760
1532643489446 mozrunner::runner INFO Running command: "C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\...\\AppData\\Local\\Temp\\rust_mozprofile.Rci2IkveaCHL"
Unable to read VR Path Registry from C:\...\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\openvrpaths.vrpath
Unable to read VR Path Registry from C:\....\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\openvrpaths.vrpath
1532643491280 Marionette INFO Listening on port 58941
1532643491606 Marionette WARN TLS certificate errors will be ignored for this session
lip 27, 2018 12:18:11 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[Child 21140, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
[Child 21140, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
[Child 11772, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
[Child 11772, Chrome_ChildThread] WARNING: pipe error: *** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
[GPU 9452, Chrome_ChildThread
###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv'
Error with proxy:
lip 27, 2018 12:48:03 AM org.openqa.selenium.remote.DesiredCapabilities firefox
INFO: Using `new FirefoxOptions()` is preferred to `DesiredCapabilities.firefox()`
1532645283430 geckodriver INFO geckodriver 0.21.0
1532645283439 geckodriver INFO Listening on 127.0.0.1:27001
1532645283919 mozrunner::runner INFO Running command: "C:\\Program Files (x86)\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\...\\AppData\\Local\\Temp\\rust_mozprofile.pbDkAIWznMhW"
Unable to read VR Path Registry from C:\...\AppData\Local\openvr\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\AppData\Local\openvr\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\AppData\Local\openvr\openvrpaths.vrpath
Unable to read VR Path Registry from C:\...\AppData\Local\openvr\openvrpaths.vrpath
1532645285883 Marionette INFO Listening on port 59352
Unable to read VR Path Registry from C:\U...\AppData\Local\openvr\openvrpaths.vrpath
1532645286083 Marionette WARN TLS certificate errors will be ignored for this session
1532645286091 Marionette INFO Proxy settings initialised: {"proxyType":"manual","ftpProxy":"77.104.208.138:3128","httpProxy":"77.104.208.138:3128","sslProxy":"77.104.208.138:3128"}
lip 27, 2018 12:48:06 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[Child 14744, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
[Child 14744, Chrome_ChildThread] WARNING: pipe error: 109: file[Child 11348, Chrome_ChildThread] WARNING: pipe error: 109: file z:/build/build/src/ipc/chromium/src/chrome/common/ipc_channel_win.cc, line 346
[Child 11348, Chrome_ChildThread] W*** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
[GPU 21212, Chrome_ChildT
###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv