I am trying to scrape a government site that serves results paginated with no next button in Octoparse.
Is there a non-messy way to format a relative XPath so that it clicks the next pages in order?
Page One = .//*[@id='datagrid_results']/tbody/tr[42]/td/a[1]
Page Two = .//*[@id='datagrid_results']/tbody/tr[42]/td/a[2]
I've tried the Octoparse tutorials, but they only mention how to loop in situations where there is a next button. Is this a situation where I am better off using Selenium?
I got it to work with concurrent scripts running, which is fine since Octoparse limits things to 20,000 results for any one scrape. What I did was copy a script, and then insert a click to the second page of results, pages 41-80. Then I copied that script and added a second click to the third page of results, 81-120...and so on. It worked without doing anything else as the relative XPath was exactly the same.