Webscraping with Selenium without browser

Question

I want to use the Python module Selenium to do web-scraping through a jupyter notebook. The jupyter notebook runs in a docker-container without any web-browser. I want to be able to distribute the notebook so that the web-scraping can be duplicated by other users. The notebook runs on a common jupyter lab container, and it is not possible to update the container to include a browser.

I have tried a number of things:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

And this:

!pip install chromedriver-binary
from selenium import webdriver
import chromedriver_binary  # Adds chromedriver binary to path

driver = webdriver.Chrome('/opt/conda/lib/python3.7/site-packages/chromedriver_binary')

For this last case I have located the binaries using

import chromedriver_binary
print(chromedriver_binary.__file__)

But unfortunately I have not been able to make any of it work.

Answer 1

the chrome driver depends on a local install of chrome - so you'll have to modify the docker image you're using to install chrome first.