Error while running local python script on a virtual environment (Google Colab)

Question

I'm trying to run a Python script for web scraping that was originally created to run locally. I made the necessary modifications to adapt it to the Colab environment, but the script can't retrieve results from the website, although it works normally when run locally.

The modified script:

!pip install selenium
!apt-get update
!apt-get install -y libgconf-2-4
!apt-get install -y libxss1 libappindicator1 libindicator7
!apt-get install xvfb

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.common.by import By
import os
import time


# Set up Chrome options
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

# Set up the Chrome driver
driver = webdriver.Chrome(options=options)

driver.get("https://www.virustotal.com/gui/home/search")

wait = WebDriverWait(driver, 15)
search_input = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('home-view').shadowRoot.querySelector('vt-ui-search-bar').shadowRoot.querySelector('input#searchInput')"))))
search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")
time.sleep(10)

result = wait.until(EC.element_to_be_clickable((driver.execute_script("return document.querySelector('file-view').shadowRoot.querySelector('vt-ui-main-generic-report').shadowRoot.querySelector('vt-ui-detections-widget').shadowRoot.querySelector('div.positives')"))))

result_text = str(result.text)
print(result_text)


driver.quit()

The received error :

JavascriptException: Message: javascript error: Cannot read properties of null (reading 'shadowRoot')
  (Session info: chrome-headless-shell=120.0.6099.71)
Stacktrace:
#0 0x55599bde0d33 <unknown>
#1 0x55599ba9df87 <unknown>
#2 0x55599baa4533 <unknown>
#3 0x55599baa6e44 <unknown>
#4 0x55599bb36e43 <unknown>
#5 0x55599bb17342 <unknown>
#6 0x55599bb36297 <unknown>
#7 0x55599bb170e3 <unknown>
#8 0x55599badf044 <unknown>
#9 0x55599bae044e <unknown>
#10 0x55599bda5861 <unknown>
#11 0x55599bda9785 <unknown>
#12 0x55599bd93285 <unknown>
#13 0x55599bdaa41f <unknown>
#14 0x55599bd7720f <unknown>
#15 0x55599bdce028 <unknown>
#16 0x55599bdce1f7 <unknown>
#17 0x55599bddfed4 <unknown>
#18 0x791291309ac3 <unknown>

Answer 1

You were very close. You just forgot to hit enter after inputting 759b1fcbdf4ad54bb8adc2c060df8ab5 into the search box.

Just add the below code to hit enter key:

search_input.send_keys("759b1fcbdf4ad54bb8adc2c060df8ab5")
search_input.send_keys(Keys.ENTER)

I ran your code by just adding the above line and below is the output:

39

Process finished with exit code 0

Note: You will need below import statement:

from selenium.webdriver import Keys