Go Colly how to find requested element?

906 Views Asked by At

I'm trying to get specific table to loop through its content using colly but table its not being recognized, here's what I have so far.

package main

import (
    "fmt"
    
    "github.com/gocolly/colly"
)

func main() {
    c := colly.NewCollector(
        colly.AllowedDomains("wikipedia.org", "en.wikipedia.org"),
    )
    
    links := make([]string, 0)

    c.OnHTML("div.mw-parser-output", func(e *colly.HTMLElement) {
        
        e.ForEach("table.wikitable.sortable.jquery-tablesorter > tbody > tr", func(_ int, elem *colly.HTMLElement) {
            fmt.Println(elem.ChildAttr("a[href]", "href"))
            links = append(links, elem.ChildAttr("a[href]", "href"))
        })
    })
    
    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Visiting", r.URL.String())
    })

    c.Visit("https://en.wikipedia.org/wiki/List_of_countries_and_dependencies_by_population")
    fmt.Println("Found urls for", len(links), "countries.")
}

I need to loop thought all of the tr elements in the the table.

1

There are 1 best solutions below

0
On BEST ANSWER

Turns out name of the class is actually wikitable.sortable even though in chrome console is shown as wikitable sortable jquery-tablesorter. I dont know why the names are different like this but it solved the problem for me.