ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/LATEST_RELEASE_108.0.5359.71

Question

I am working on a Python script that needs to run in Databricks using Selenium and the webdriver_manager package. The script runs once a week using an Airflow DAG. Recently, I started encountering issues with the ChromeDriver version detection. The code was working fine earlier but now seems to fail due to possibly an update in the Databricks environment or the packages I'm using.

Here's my current code:

from seleniumwire import webdriver 
from seleniumwire.utils import decode as sw_decode
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
import chromedriver_autoinstaller

# Create a new instance of the Chrome driver
#Setting up Chrome webdriver
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')

preferences = {"download.prompt_for_download": False,
               "download.directory_upgrade": True,
               "directory_upgrade": True,
               "safebrowsing.enabled": True }

chrome_options.add_experimental_option("prefs", preferences)

chrome_driver = "/tmp/chromedriver"

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()),
    options=chrome_options
)


# Go to the required web page
driver.get(url)

And here is the error I am facing:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<command-3775777336036441> in <module>
     16 
     17 driver = webdriver.Chrome(
---> 18     service=Service(ChromeDriverManager().install()),
     19     options=chrome_options
     20 )

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/chrome.py in install(self)
     37 
     38     def install(self) -> str:
---> 39         driver_path = self._get_driver_binary_path(self.driver)
     40         if all(test_os not in driver_path for test_os in ["mac_arm64", "mac_x64"]):
     41             os.chmod(driver_path, 0o755)

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/manager.py in _get_driver_binary_path(self, driver)
     31             return binary_path
     32 
---> 33         file = self._download_manager.download_file(driver.get_driver_download_url())
     34         binary_path = self._cache_manager.save_file_to_cache(driver, file)
     35         return binary_path

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/drivers/chrome.py in get_driver_download_url(self)
     43 
     44     def get_driver_download_url(self):
---> 45         driver_version_to_download = self.get_driver_version_to_download()
     46         os_type = self._os_type
     47         # For Mac ARM CPUs after version 106.0.5249.61 the format of OS type changed

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/driver.py in get_driver_version_to_download(self)
     49         if not self._driver_to_download_version:
     50             self._driver_to_download_version = self._driver_version if self._driver_version not in (None, "latest") \
---> 51                 else self.get_latest_release_version()
     52         return self._driver_to_download_version
     53 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/drivers/chrome.py in get_latest_release_version(self)
     77             else f"{self._latest_release_url}_{determined_browser_version}"
     78         )
---> 79         resp = self._http_client.get(url=latest_release_url)
     80         return resp.text.rstrip()
     81 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/http.py in get(self, url, **kwargs)
     34         except exceptions.ConnectionError:
     35             raise ConnectionError(f"Could not reach host. Are you offline?")
---> 36         self.validate_response(resp)
     37         return resp
     38 

/local_disk0/pythonVirtualEnvDirs/virtualEnv-18c7cdb1-838b-427a-a362-0f0baf05cdb9/lib/python3.8/site-packages/webdriver_manager/core/http.py in validate_response(resp)
     13         status_code = resp.status_code
     14         if status_code == 404:
---> 15             raise ValueError(f"There is no such driver by url {resp.url}")
     16         elif status_code == 401:
     17             raise ValueError(f"API Rate limit exceeded. You have to add GH_TOKEN!!!")

ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/LATEST_RELEASE_108.0.5359.71

I tried the following and none of them worked:

1. driver = webdriver.Chrome(
       service=Service(ChromeDriverManager(version="latest").install()), options=chrome_options
   )
   

2. driver = webdriver.Chrome(
       service=Service(ChromeDriverManager().install(version="latest")),
       options=chrome_options
   )
   

For both 1 and 2, I received- TypeError: install() got an unexpected keyword argument 'version'

3.

    driver = webdriver.Chrome(
            service=Service(),
            options=chrome_options
        )

For 3, I run into the following error:

    The version of chrome cannot be detected. Trying with latest driver version
    WebDriverException: Message: Service /root/.cache/selenium/chromedriver/linux64/114.0.5735.90/chromedriver unexpectedly exited. Status code was: 127

How can I modify my code in Databricks without manually specifying the version or modifying the webdriver_manager package source code?

Answer 1

If I go to https://chromedriver.storage.googleapis.com And search for 108.0.5359.71 There is a line "02T18:32:13.086Z"fab81c782b43c0022d6aea598899c384"326108.0.5359.71/chromedriver_linux64.zip166998020650893612022-12-" Plus 4 more lines for other OSes