Selenium WebDriver not starting properly through flask service using Amazon Lightsail

Question

I'm running into some issues when trying to start a chrome webdriver using selenium.I am not super well versed in Docker or Lightsail. My code works properly when I run it on my local machine. However, I created a Docker container for a flask service that I want to run my code. My html data in the Flask webapp contains a button that starts up a webscraper, and this works fine until I use the flask service. I got the Docker container to work properly, so I'm not really sure if there is a versioning mismatch between my webdriver and chrome, or some other issues that I don't see. Here is the error when I try to start up chrome using selenium through my lightsail flask service:

super().__init__(
File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 61, in __init__
super().__init__(command_executor=executor, options=options)
File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 208, in __init__
self.start_session(capabilities)
File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 292, in start_session
response = self.execute(Command.NEW_SESSION, caps)["value"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome failed to start: exited abnormally.
(timeout: Timed out receiving message from renderer: 60.000)
(The process started from chrome location /opt/google/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Stacktrace:

As you can see, the chrome webdriver never starts up, and I am not sure why. Here is my docker file for the base image of my project (I have a base one and a second one).

# Set base image (host OS)
FROM debian:latest

# Install necessary dependencies
RUN apt-get update -y \
    && apt-get upgrade -y \
    && apt-get install -y \
        wget \
        gnupg \
        unzip \
        xz-utils \
        build-essential \
        zlib1g-dev \
        libncurses5-dev \
        libgdbm-dev \
        libssl-dev \
        libreadline-dev \
        libffi-dev \
        libsqlite3-dev \
        libbz2-dev \
        libnss3-dev \
        openssl \
    && export DEBIAN_FRONTEND=noninteractive

# Install Chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
    && apt-get update && apt-get install -y \
    google-chrome-stable

# Install Chrome WebDriver - seems to get installed with Chrome
#RUN wget -q -O /usr/local/bin/chromedriver https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip \
#    && unzip /usr/local/bin/chromedriver -d /usr/local/bin \
#    && chmod +x /usr/local/bin/chromedriver

# Install Python 3.11.3
RUN wget -O /tmp/python.tar.xz https://www.python.org/ftp/python/3.11.3/Python-3.11.3.tar.xz \
    && mkdir -p /usr/src/python \
    && tar -xJC /usr/src/python --strip-components=1 -f /tmp/python.tar.xz \
    && rm /tmp/python.tar.xz \
    && cd /usr/src/python \
    && ./configure --enable-optimizations --with-ssl \
    && make -j$(nproc) \
    && make install \
    && cd / \
    && rm -rf /usr/src/python \
    && python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"

# Update pip
RUN pip3 install --upgrade pip

# Clean up
RUN rm -rf /var/lib/apt/lists/*

# Copy the dependencies file to the working directory
COPY requirements.txt /tmp/

# Install python dependencies
RUN pip3 install -r /tmp/requirements.txt

And here is my requirements file:

flask===2.2.2
matplotlib===3.8.0
numpy===1.24.3
pandas===1.5.3
pytz===2023.3
selenium===4.18.1
webdriver_manager===4.0.1
werkzeug===2.2.2
waitress===3.0.0
google-auth-oauthlib===1.1.0
google-api-python-client===2.122.0
undetected_chromedriver===3.5.5

I transitioned to using undetected-chromedriver to see if that would fix the problem but I ran into the same issue.

If anyone wants to help, let me know if you need more info. Thanks!

Answer 1

Not sure what your driver script looks like, but suppose something simple like this:

run.py

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()

options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--window-size=1920,1080")

print("Launch Chrome.")
driver = webdriver.Chrome(options=options)

print("Get www.example.com.")
driver.get("http://www.example.com")

print("Close Chrome.")
driver.quit()

Dockerfile (Rather than installing Python onto a Debian base image, just use a Python base image.)

FROM python:3.11.3

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y \
    && apt-get install -y \
        wget \
        gnupg \
        unzip \
        xz-utils \
        build-essential \
        zlib1g-dev \
        libncurses5-dev \
        libgdbm-dev \
        libssl-dev \
        libreadline-dev \
        libffi-dev \
        libsqlite3-dev \
        libbz2-dev \
        libnss3-dev \
        openssl \
    && pip3 install --upgrade pip

RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
    && apt-get update && apt-get install -y \
    google-chrome-stable

RUN rm -rf /var/lib/apt/lists/*

COPY requirements.txt /tmp/

RUN pip3 install -r /tmp/requirements.txt

COPY run.py .

CMD python3 run.py

requirements.txt (Stripped down to only include packages to get a working example.)

selenium===4.18.1