I have a Python script that uses Selenium to do some web-page clicking and scraping. Script is running on Ubuntu, running on an EC2 instance. The basic code:
from selenium import webdriver
from bs4 import BeautifulSoup as bs
import datetime
from datetime import datetime as dt
import re
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import ElementNotVisibleException
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
#Set driver options
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('--window-size=1420,1080')
options.add_argument('--headless')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
options.add_argument("--disable-notifications")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'/Applications/chromedriver_91')
# Do a bunch of stuff................
driver.quit() #invoke after web-scraping
Does driver.quit()
, in this instance, essentially do the same thing as the Linux command pkill chrome
? Sometimes, this script will crash because there's not enough memory. Using pkill chrome
in combination with pkill -f "(chrome)?(--headless)"
in the terminal itself usually kills all the processes and frees up memory, and the script will work after that.
Is driver.quit()
sufficient to close all Chrome processes in headless or otherwise? Is adding something in my Python script like:
import os
os.system("pkill chrome")
Doing anything that driver.quit()
is not already doing? I just want to minimize the chances of a crash by making sure Chrome is completely closed after Python script is run.
driver.quit()
quits (closes) only this specificdriver
object.This will definitely not close any other running
driver
processes.Also process involved by chromedriver appears as
chromedriver
orchromedriver (32 bit)
, notchrome
.chrome
process is your Chrome browser, not the Selenium webdriver.To ensure closing the chromedriver you can use
try-except-finally
involvingdriver.quit()
inside thefinally
block.I'm not sure this is the optimal approach since AFAIK this may affect reporting mechanism.