Is there a way to handle dynamically loaded selectors with clicks on pages in scrapy-playwright?

409 Views Asked by At

I have a use case like this. Suppose if I crawl a website abc.com using scrapy playwright the page it loads are of 3 different types of pages like

page1->#selector1 
page2->#selector2
page3->#selector3

and it changes dynamically there is no guarantee which loads first. I want to click on the selector based on which is loaded using scrapy-playwright.

Is there a way to do this dynamically based selector loaded?

I tried using

PageMethod("click","#selector1"),
PageMethod("click","#selector2"),
PageMethod("click","#selector3") 

and this is getting timed out because at that point in time #selector2 is loaded first on the page causing the operation to fail as the first method is PageMethod("click","#selector1")

1

There are 1 best solutions below

2
Jaky Ruby On

Maybe you can check the selector is visible and then you can click it?

if PageMethod("isVisible","#selector1"):
    PageMethod("click","#selector1")    
if PageMethod("isVisible","#selector2"):
    PageMethod("click","#selector2")    
if PageMethod("isVisible","#selector3"):
    PageMethod("click","#selector3")

I think you can usue the method isVisible, because scrapy_playwright support the same you have in playwright: https://github.com/scrapy-plugins/scrapy-playwright#supported-methods

And of course, isVisible() is a supported method in playwrgiht: https://playwright.dev/python/docs/api/class-page#page-is-visible