Python requests works fine, when trying same request in golang is not working as expected

Question

Python code, keep in mind that the __cf_bm cookie has to be valid in order to get a json response. The __cf_bm is only valid for around 30min, so you have to get a cookie from https://kick.com on your own to replicate this example.

import requests

headers = {
'Accept':'application/json',
'Cookie':'__cf_bm=yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy',
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}


resp = requests.request('GET', 'https://kick.com/api/v1/channels/xqc', headers=headers)
print (resp.text)

The user-agent is tied to the cookie, make sure to use your user agent with which you got your cookie, not the one in the example. The accept: application/json is not that important, can be omitted.

The problem is when i make the SAME request in golang with the same headers and cookies and user-agent, that i get a cloudflare response, (which is the same response, when the __cf_bm cookie has expired) but i know that the cookie has not expired yet.

Golang Code, go version go1.20.5 linux/amd64

package main

import (
    "compress/gzip"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
)

func MakeRequest(url string) (*string, error) {
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
    req.Header.Set("Accept", "application/json")
    req.Header.Set("Connection", "keep-alive")
    req.Header.Set("Accept-Encoding", "gzip, deflate")
    req.AddCookie(&http.Cookie{Name: "__cf_bm", Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy"})

    client := &http.Client{}
    resp, err := client.Do(req)
    var body string

    if err != nil {
        return &body, err
    }

    defer resp.Body.Close()

    var reader io.ReadCloser
    reader, _ = gzip.NewReader(resp.Body)
    bytes, _ := ioutil.ReadAll(reader)
    body = string(bytes)
    return &body, nil
}

func main() {
    data, _ := MakeRequest("https://kick.com/api/v1/channels/xqc")
    fmt.Println(*data)
}

I added Connection and Accept-Encoding header which python requests adds by default but golang doesn't.

The headers are the same in python and golang but the response is not, i'm doing something wrong or is the way golang sends the request. I can't see what im doing wrong...

Answer 1

Is the gzip mod needed?

bytes, _ := ioutil.ReadAll(resp.Body)
body = string(bytes)
return &body, nil

Answer 2

Firstly, the commentator above was right.Gzip mod not needed here. Next the mistake was : if you create a variable of type bytes.Buffer (without initialization) and assign it to a field of type io.Reader, then after checking io.Reader for nil there will be an error: invalid memory address or nil pointer dereference. More about that How to correctly check io.Reader for nil?

The right code:

func main() {
request, err := http.Get("https://kick.com/api/v1/channels/xqc")
if err != nil {
    log.Fatal(err)
}
myCookie := &http.Cookie{
    Name:  "__cf_bm",
    Value: "yBP__EtLom8aR55x2yIow3GYMpKSMiAAATdk4xHO7CA-1687137561-0-AbAdr3HpoqE97ImTJi4fWsbDp8iOzl6PBzrSwDnFBoTENSfjb7ZOTVx5YX4fuuUMnrIxzqgyWE+kQlENR3ouM3vkVKSWPljxD/JRJJVGGWFy",
}
request.Request.AddCookie(myCookie)
request.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
request.Header.Set("Accept", "application/json")
request.Header.Set("Connection", "keep-alive")

body, err := io.ReadAll(request.Body)
if err != nil {
    fmt.Println(err)
}
defer request.Body.Close()
st := string(body)
fmt.Println(st)
}