Scrolling to bottom of infinite scroll page then getting html document

Question

I'm kinda lost as to how to carry out this task as I'm pretty new to golang. I'm trying to scroll to the bottom of a page that uses infinite scroll in order to load all of the elements and then save the response as an HTML file (or even better just getting the tags) as obviously it will only return part of the elements if there is no scroll as it's infinite scroll. There's also no discernible footer or something I could jump to using Rod or chromedp, any help ? :)

Answer 1

It depends on how the infinite scroll page is implemented.

With chromedp, we usually scroll a page like this:

package main

import (
    "context"
    "time"

    "github.com/chromedp/chromedp"
    "github.com/chromedp/chromedp/kb"
)

func main() {
    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        // Disable the headless mode to see what happen.
        chromedp.Flag("headless", false),
    )

    ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()

    ctx, cancel = chromedp.NewContext(ctx)
    defer cancel()

    if err := chromedp.Run(ctx,
        chromedp.Navigate("https://intoli.com/blog/scrape-infinite-scroll/demo.html"),
    ); err != nil {
        panic(err)
    }

    for i := 0; i < 10; i++ {
        if err := chromedp.Run(ctx,
            // Option 1 to scroll the page: window.scrollTo.
            chromedp.Evaluate(`window.scrollTo(0, document.documentElement.scrollHeight)`, nil),
            // Slow down the action so we can see what happen.
            chromedp.Sleep(2*time.Second),
            // Option 2 to scroll the page: send "End" key to the page.
            chromedp.KeyEvent(kb.End),
            chromedp.Sleep(2*time.Second),
        ); err != nil {
            panic(err)
        }
    }
}