How do I maintain logged in session with golang for scraping?

Question

I'm trying to scrape data from a website that requires user/password login using go. With python this is simple using requests lib:

import requests

session = requests.Session()
session.post("https://site.com/login", data={ 'username': 'user', 'password': '123456' })

# access URL that requires authentication
resp = session.get('https://site.com/restricted/url')

What is a simple way to accomplish the same thing with golang? thanks.

Answer 1

Create a custom HTTP Client instance and attach a cookie jar to it.

Answer 2

I wrote a scraping framework called Colly which handles HTTP sessions out of the box. You can achieve the mentioned functionality similarly:

c := colly.NewCollector()
c.Post("https://example.com/login", map[string]string{"user": "x", "pass": "y"})

The code can be found on GitHub. A complete example of handling authentications is also available.