Trying to act on a page redirect with JavaFX and WebView

Question

I am trying to script logging in to a 3rd party site to obtain a value. For several years I was able to use javaFX to do this, but it recently stopped working. I have tried using selenium with some success, however the website seems to dissuade this and it frequently fails. I have spent several hours trying to get this working with JavaFx but I have hit a brick wall. I am able to log in through the first page. The site then refreshes the page without causing a reload and I believe this is the behavior that jewelsea talks about in this answer.

I listen for SUCCEEDED on the webEngine.getLoadWorker().stateProperty() which allows me to process the initial log in page. However, a terms page is then loaded and I can listen to webEngine.locationProperty() for the change, however it is fired before the page is loaded, so I can't process the page that is being loaded.

I have tried webengine.load() in the event handler for the locationProperty change, if I do it in the same thread, the app crashes. If I put it to run later with Platform.runLater(), it ends up in a constant loop between the first and second pages.

Hopefully there is an easy answer that I am missing that someone else will provide.

Further to Andrei-Paul's comments, I did try a delay in the on the locationProperty but it does not load until the event handler fires. I had considered firing an event in a seperate thread, after a delay but have not tried that. I did try force loading the new page in the location property event handler, however that ended up in a loop.

After David's comment, I was able to create a program that should illustrate the issue. This is somewhat contrived as my issue happens after entering credentials, however this seems to show the same behavior.

When running this code, when the page loads for Choose account type(s), click continue. The page loads and has Choose brokerage account type, however the LoadWorker never fires an event, and I am not able to process the newly loaded web page. The output I get from the program is shown after the program. My goal is to interact with the page, change the fields and submit the form. I can programmatically do this by executing javascript in the LoadWorker event when the first page is successfully loaded but not any subsequent pages. (Incidentally, I am requesting a longer lifetime for the token that I am trying to obtain that would make this problem less of an issue for me)

import com.sun.webkit.dom.HTMLDocumentImpl;
import javafx.application.Application;
import javafx.concurrent.Worker;
import javafx.scene.Scene;
import javafx.scene.layout.VBox;
import javafx.scene.web.WebEngine;
import javafx.scene.web.WebView;
import javafx.stage.Stage;
import org.w3c.dom.NodeList;

public class WebProblem extends Application {

    public static void main(String[] args) {
        launch();
    }

    @Override
    public void start(Stage stage) throws Exception {
        final Object lock = new Object();
        WebView webView = new WebView();
        WebEngine webEngine = webView.getEngine();
        webEngine.locationProperty().addListener((observable, oldValue, newValue) -> {
            System.out.println("LocnProp was " + oldValue + "\n->"+newValue + "\nWorker state is: "+
                    webEngine.getLoadWorker().stateProperty().get());

            HTMLDocumentImpl doc = (HTMLDocumentImpl) webEngine.getDocument();
            NodeList inputs = doc != null ? doc.getElementsByTagName("H1"): null;
            String title = doc != null ?  doc.getTitle() : "<null>";
            System.out.println("Doc title is " + title  +"\tHeading 1 of doc is "
                    + (inputs != null ? inputs.item(0).getTextContent() : " <null>"));
        });
        webEngine.onStatusChangedProperty().addListener((observable, oldValue, newValue) -> {
            System.out.println("statusChanged: "+ newValue);
        });
        webEngine.getLoadWorker().stateProperty().addListener((observable, oldValue, newValue) -> {
            HTMLDocumentImpl doc = (HTMLDocumentImpl) webEngine.getDocument();
            String title = doc != null ?  doc.getTitle() : "<null>";
            System.out.println("LoadWorker State: "+ newValue +"\tDoc title is " + title);
            if (newValue == Worker.State.SUCCEEDED) {
                NodeList inputs = doc.getElementsByTagName("H1");
                System.out.println("Heading 1 is " + inputs.item(0).getTextContent());
            }
        });


        webEngine.documentProperty().addListener((ov, oldDoc, doc) -> {
           if (doc != null ) {
               System.out.println("Document Property Listener: " + doc);
           }
        });

        webEngine.load("https://express.etrade.com/etx/rtao/ma/account-category");

        VBox vBox = new VBox(webView);
        Scene scene = new Scene(vBox, 960, 600);

        stage.setScene(scene);
        stage.show();
    }
}

The results of running this and clicking on the initial continue button:

LocnProp was null ->https://express.etrade.com/etx/rtao/ma/account-category Worker state is: READY
Doc title is <null> Heading 1 of doc is  <null>
LoadWorker State: SCHEDULED Doc title is <null>
LoadWorker State: RUNNING   Doc title is <null>
Document Property Listener: [object HTMLDocument]
LoadWorker State: SUCCEEDED Doc title is Get started with E*TRADE!
Heading 1 is Choose account type(s)
LocnProp was https://express.etrade.com/etx/rtao/ma/account-category ->https://express.etrade.com/etx/rtao/ma/account-types
Worker state is: SUCCEEDED
Doc title is Get started with E*TRADE!  Heading 1 of doc is Choose account type(s)