How to solve raise TimeoutException(message, screen, stacktrace) in selenium python

Question

So, I have been working on my first selenium project. Here is the website I want to scrap https://sekolah.data.kemdikbud.go.id/index.php/Chome/profil/8F9F7295-595A-4C42-A4AA-0009B36BCBE1 when you scroll down, there is a clickable element "Siswa". and when it clicked, it shows some tables. My problem is, I always get error when I try to interact with that element. I have make sure that I have a right XPATH before. I also tried to change the Xpath and the locator, but it still error. here's my code:

import openpyxl
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
import pyautogui as py
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = 'https://sekolah.data.kemdikbud.go.id/'
chrome_options = Options()
chrome_options.add_experimental_option("detach",True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),
                          options=chrome_options)
driver.get(url)

# choose kabupaten/kota
x_element = driver.find_element(By.ID, 'kode_kabupaten')
x = Select(x_element)
x.select_by_visible_text("Prov. Jawa Timur - Kota Surabaya")

# choose jenjang
y_element = driver.find_element(By.ID, 'bentuk_pendidikan')
y = Select(y_element)
y.select_by_visible_text("SMA")

submit = driver.find_element(By.XPATH, '//button[text()="Cari Sekolah"]')
submit.click()

sch_name = driver.find_element(By.XPATH, '//*[@class="text-info"]/b')
sch_name.click()

siswa_link = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, '//a[contains(@href, "#pd")]')))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//a[contains(@href, "#pd")]'))).click()

and here's the error message that shown:

Traceback (most recent call last):
  File "C:\Users\user\PycharmProjects\dikbud\main.py", line 37, in <module>
    EC.presence_of_element_located((By.XPATH, '//a[contains(@href, "#pd")]')))
  File "C:\Users\user\PycharmProjects\dikbud\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:
    GetHandleVerifier [0x006A72A3+45731]
    (No symbol) [0x00632D51]
    (No symbol) [0x0052880D]
    (No symbol) [0x0055B940]
    (No symbol) [0x0055BE0B]
    (No symbol) [0x0058D1F2]
    (No symbol) [0x00578024]
    (No symbol) [0x0058B7A2]
    (No symbol) [0x00577DD6]
    (No symbol) [0x005531F6]
    (No symbol) [0x0055439D]
    GetHandleVerifier [0x009B0716+3229462]
    GetHandleVerifier [0x009F84C8+3523784]
    GetHandleVerifier [0x009F214C+3498316]
    GetHandleVerifier [0x00731680+611968]
    (No symbol) [0x0063CCCC]
    (No symbol) [0x00638DF8]
    (No symbol) [0x00638F1D]
    (No symbol) [0x0062B2C7]
    BaseThreadInitThunk [0x757A7BA9+25]
    RtlInitializeExceptionChain [0x778BBD2B+107]
    RtlClearBits [0x778BBCAF+191]

Help me to solve this problem, so I can interact with clickable element named "Siswa" and scrapping the table's data.

Answer 1

Root cause of the issue: If you notice, when you click on submit button, new page loads in a new tab. In such situation, selenium will still try to locate the desired element in the current tab(and not the new tab).

Solution: After clicking submit, you need to switch the context to new tab, then perform the required actions on new tab.

Code: Refer the code below to switch the context to new tab in python selenium.

submit = driver.find_element(By.XPATH, '//button[text()="Cari Sekolah"]')
submit.click()

# Below line will store the current window in a variable
window_before = driver.window_handles[0]

sch_name = driver.find_element(By.XPATH, '//*[@class="text-info"]/b')
sch_name.click()

# Below line will store the new tab in a variable
window_after = driver.window_handles[1]
# Below line will switch to new tab
driver.switch_to.window(window_after)

siswa_link = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//a[contains(@href, "#pd")]')))
siswa_link.click()

PS: Use these two variables window_before and window_after, to switch as per your intended actions on the respective tabs.