Python Selenium Stealth scrapy not working

Question

I'm trying to use selenium stealth module to scrap an api url in ubuntu server 18.04.4 LTS. I have 2 servers , one for staging and one for production. I deployed google-chrome-stable and chromemdriver in staging server and installed selenium-stealth in my virtual environment. I'm running stealth using proxy servers and user agent. In staging server, I'm able to run the script and fetch the data correctly. But in live server, this script response is blank data.

Details of packages being used: ChromeDriver: 101.0.4951.41 Google chrome stable: 101.0.4951.54

selenium==3.141.0 selenium-stealth==1.0.6

Below is the page source of the selenium script:

The Selenium configuration is as given below:

import json
from selenium import webdriver
from selenium_stealth import stealth
from selenium.webdriver.common.proxy import *



options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument("start-maximized")
options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
proxy_url = proxies.get('http') # Provided from method call
#options.add_argument("--proxy-server=%s" % proxy_url)
proxy = Proxy({
        'proxyType': ProxyType.MANUAL,
        'httpProxy': proxy_url,
        'sslProxy': proxy_url,
        'noProxy': ''
    })
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
driver = webdriver.Chrome(
            options=options,
            executable_path='/usr/bin/chromedriver',
            desired_capabilities=capabilities
)

stealth(driver,
        languages=["en-US", "en"],
        vendor="Google Inc.",
        platform="Win32",
        webgl_vendor="Intel Inc.",
        renderer="Intel Iris OpenGL Engine",
        fix_hairline=True,
)

driver.get(browse_api_url) # Provided from method call
html = driver.find_element_by_xpath(".//html")

I'm running the same version of chrome, chromedriver, selenium and selenium-stealth in staging server as well. There script is running correctly. Thanks in advance

Answer 1

I ran your code on my machine and it's working fine, using the same version stack as you.

If your staging script is working and live is not, there are these possible solutions that I could think off:

1. your live code is not the same as the staging code
2. antivirus/firewall is blocking your script from running or connecting to webdriver
3. you are not running correct version of chromedriver.exe

Note: including your error message could be really helpful.