Killing unnecessary chrome processes created by Chromium and Selenium WebDriver

Question

I am trying to run a Selenium WebDriver script that takes quite a long time to complete. The thing is that while executing the said script on my server, after about an hour and a half, actions that before took 10 seconds to complete now take about 2 minutes, which makes the execution extremly slow. Analyzing what the problem could be, I have seen that several crhome processes are created during the execution. For example, when I create the execution at first, I have 4 chrome processes, while, when the execution has spent some hours executing, 24 processes have been created.

I have the following script that tries to rerun the script if an exception has been thrown during the execution and also try to remove unnecessary chrome processes to free resources up every hour, but it seems this is not working, because instead of what I wanted to do, this is killing the main chrome process, meaning it causes the script to stop and having to rerun it from the shell (so only 10 hours of execution would be performed, one per attempt). Is there any way to only get rid of the unnecessary chrome processes in this script?

max_attempts=10
attempt=1
log_file="../logs/execution_log_$(date +"%d%m%Y").txt"
while [ $attempt -le $max_attempts ]; do
  # Kill any running instances of python3, chromedriver, chrome and google-chrome-stable
  pkill -f "python3"
  pkill -f "chromedriver"
  pkill -f "chrome"
  pkill -f "google-chrome-stable"

  # Run script with nohup
  nohup python3 main.py >> "$log_file" 2>&1 &

  # Store the process ID of the script
  script_pid=$!

  start_time=$(date +%s)

  # Wait for the script to finish or check if it is still running
  while ps -p $script_pid > /dev/null; do
    sleep 5
    # Check if the elapsed time exceeds 1 hour (3600 seconds)
    current_time=$(date +%s)
    elapsed_time=$((current_time - start_time))
    if [ $elapsed_time -ge 3600 ]; then
        echo "$(date) Killing zombie Chrome processes..."
        # Kill all chrome processes except for the oldest
        pkill -f "chrome --type=zygote"
        pkill -f "chrome --type=renderer"
        pkill -f "chrome --type=gpu-process"
        pkill -f "chrome --type=utility"
        break
    fi
  done
  echo "$(date) The command has stopped"

  exit_code=$?
  # Check if exit code is different from 0 and if the execution log file's last line is equal to the literal String to break the loop
  if [ $exit_code -ne 0 ] && [ "$(tail -n 1 "$log_file")" = "ENDOF SCRIPT" ]; then
    echo "$(date) Script completed successfully with exit code 0."
    break
  else
    echo "$(date) Script exited with code $exit_code."
    # If it's the last attempt, print a failure message and exit
    if [ $attempt -eq $max_attempts ]; then
      echo "$(date) Maximum attempts reached. Script failed."
      # Kill any running instances of python3, chromedriver, chrome and google-chrome-stable
      pkill -f "python3"
      pkill -f "chromedriver"
      pkill -f "chrome"
      pkill -f "google-chrome-stable"
      exit 1
    fi

    echo "$(date) Restarting command..."
    attempt=$((attempt + 1))
  fi
done

Answer 1

I am not sure if you are already doing this, but if not, you could combine your Selenium script with a unit test runner. When your script is combined with a test runner you can execute useful setUp and tearDown actions before and after the test has run. As an example even after an error the Selenium resources should be correctly closed / disposed of. (Also I am not sure in which language your Selenium script is written in so I just added a Python example...)

import unittest

from selenium import webdriver

class Tests(unittest.TestCase):
    def setUp(self):
        # create a new Chrome session
        self.driver = webdriver.Chrome()
        self.driver.implicitly_wait(30)
        self.driver.maximize_window()
        # navigate to the application home page
        self.driver.get("https://www.google.com/")

    def test(self):
        # your test code

    def tearDown(self):
        # close the browser window
        self.driver.quit()