How to find screen coordinates of found text?

Question

About 5 years ago this question was asked:

"On a web page in my web browser (preferably, but not necessarily Firefox), I can search (byctrl+f) for a given text "abc" within the body text of the page. From there then I must move the mouse cursor to another (relative) position (height plus x pixels), and there I must do a mouse click.

I cannot do this otherwise since the needed info is not contained in the source code but is fetched by mouse click from the web server. The problem for me is to identify the position of the found text "abc", in order to move the mouse cursor there; from there it's easy.

I currently try to solve my problem by searching for the background color which changes for the text "abc" when found, but the same color is found in lots of other positions on the screen, so this is unreliable, and finding the text "abc" as a graphic is unreliable, too. So I'm looking for an alternative, programmatic way to identify the position of found text, if there is any."

==> I'm currently facing the same problem and so far haven't really found a solution! I'm using Python, but libraries such as pyautogui do not include any way (as far as I can tell) of obtaining the position of text located via Ctrl-F. I'm hoping for some solution that works under Windows and Linux, if possible. Any solutions/workarounds/suggestions would be greatly appreciated! Wayne

Answer 1

You can use location of python to get coordinates of an element.

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import time

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://www.google.com/search?q=google&rlz=1C1CHBF_enIN889IN890&oq=google&aqs=chrome..69i57j0i271l3j69i60l3.2749j0j1&sourceid=chrome&ie=UTF-8")
time.sleep(2)
# linkel is first search item's website address
linkel = driver.find_element(By.XPATH, "(//*[contains(text(), 'https://www.google.co.in')])[1]")
print(linkel.text)
loca = linkel.location
print(loca)

Output:

https://www.google.co.in
{'x': 28, 'y': 184}

Process finished with exit code 0

I would also like to say that the coordinates would vary with different screen dimensions.

Answer 2

My final approach to this problem only requires pyautogui. The first step is to use the "find" option (Ctrl-f) in Chrome to locate the text in question. The found text is then highlighted, generally in Orange (RGB=255,150,50) but sometimes in Yellow (RGB=255,255,0). I then take a screenshot using im = pyautogui.screenshot(). Finally, I search that image ('im') to look for pixels of the appropriate background color in order to identify the upper-left and lower-right corners of the highlighted rectangle. Using those two locations it is easy to compute the center of the highlighted text.

I'm sure this is not the most efficient way of searching for the location of text on the screen, but this method certainly seems fast enough for many applications.

This method should be platform-independent, assuming that pyautogui is available. Also, no other special libraries such as Selenium are needed.

NOTE: This approach will locate the FIRST occurrence of the text being sought, so care should be taken when defining that text.

Hoping that other folks may find this approach useful, Wayne