I need to automate interactions with a website that uses DWR (Direct Web Remoting) tech. For many functions/interactions, I can POST a prescribed payload via python requests and drive the required behaviour without emulating a user navigating through the interface, but not all. I'm trying to use playwright to fill-in the more complicated gaps like file uploads. I've gotten pretty far, but I'm stuck on trying to select/locate elements.
Unfortunately I can only describe the problem as the actual site is restricted.
The basis of the problem seems to be some kind of disconnect between what's displayed in the browser, and what appears to be the source code/DOM that describes the page:
- When following activity with Developer Tools, upon clicking a particular element, there's a pop-up of sorts that I can see, and interact with. But if I right-click on an element in this pop-up and try to "inspect", it doesn't appear to exist in the page source. If I do this a second time, something triggers, and I do see it nested deep within an iframe.
- Using playwright to do the equivalent above, I can drive clicking on the initial element to trigger the pop-up, and I can take a playwright-screenshot to reflect exactly the manual observations. But if I output the page source, it's "old" source. Logically I also I can't find the element when parsing via BeautifulSoup. I've tried catching the pop-up event with
with page.expect_popup() as popup_infoper "Handling Pop-ups" at https://playwright.dev/python/docs/pages, but it times-out.
I'm not a web developer, and as I understand it DWR is somewhat dated. It seems like javascript can render new web elements locally to browser without necessarily updating the DOM. Is there anything that suggests I'm not understanding this correctly? More importantly, are there events I can use to catch the visual updates and/or triggers I might use to force updating of the source/DOM?