Force plain text encoding with node https get request

Question

Problem

Related to Get UTF-8 html content with Node's http.get - but that answer isn't working for me.

I'm trying to call the Stack Overflow API questions endpoint:

https://api.stackexchange.com/2.3/questions?site=stackoverflow&filter=total

Which should return the following JSON response:

{"total":21951385}

Example Code

I'm using the https node module to submit a get request like this:

const getRequest = (url: string) => new Promise((resolve, reject) => {
    const options: RequestOptions = {
        headers: {
            'Accept': 'text/*',
            'Accept-Encoding':'identity',
            'Accept-Charset' : 'utf8',
        }
    }

    const req = get(url, options, (res) => {
        res.setEncoding('utf8');
        let responseBody = '';
        res.on('data', (chunk) => responseBody += chunk);
        res.on('end', () => resolve(responseBody));
    });

    req.on('error', (err) => reject(err));
    req.end();
})

And then invoking it like this:

const questionsUrl = 'https://api.stackexchange.com/2.3/questions?&site=stackoverflow&filter=total'
const resp = await getRequest(questionsUrl)
console.log(resp)

However, I get the response:

▼�
�V*�/I�Q�22�454�0�♣��♥‼���↕

What I've Tried

I've tried doing several variations of the following:

• I'm calling setEncoding to utf8 on the stream

• I've set the Accept header to text/* - which

  Provides a text MIME type, but without a subtype

• I've set the Accept-Encoding header to identity - which

  Indicates the identity function (that is, without modification or compression)

This code also works just fine with pretty much any other API server, for example using the following url:

https://jsonplaceholder.typicode.com/todos/1

But the StackOverlow API works anywhere else I've tried it, so there must be a way to instruct node how to execute it.

Answer 1

My suggestion is to use an http library that supports both promises and gzip built in. My current favorite is got(). http.get() is like the least featured http request library anywhere. You really don't have to write all this yourself. Here's what your entire code would look like with the got() library:

const got = require('got');

function getRequest(url) {
    return got(url).json();
}

This library handles all these things you need for you automatically:

1. Promises
2. JSON conversion
3. Gzip decoding
4. 2xx status detection (other status codes like 404 are turned into a promise rejection which your code does not do).

And, it has many, many other useful features for other general use. The days of coding manually with http.get() should be long over. No need to rewrite code that has already been written and well-tested for you.

FYI, there's a list of very capable http libraries here: https://github.com/request/request/issues/3143. You can pick the one that has the API you like the best.

Answer 2

Response Header - Content-Encoding - Gzip

As jfriend00 pointed out - looks like the server isn't respecting the Accept-Encoding value being passed and returning a gzipped response none-the-less.

Unzipping Response

According to the answer on How do I ungzip (decompress) a NodeJS request's module gzip response body?, you can unzip like this:

import { get } from 'https'
import { createGunzip } from 'zlib'

const getRequest = (url: string) => new Promise((resolve, reject) => {
    const req = get(url, (res) => {
        const buffer: string[] = [];

        if (!res.headers['content-encoding']?.includes('gzip')) {
            console.log('utf8')
            res.on('data', (chunk) => buffer.push(chunk));
            res.on('end', () => resolve(buffer.join("")))
        } else {
            console.log('gzip')
            const gunzip = createGunzip();
            res.pipe(gunzip);
            gunzip.on('data', (data) => buffer.push(data.toString()))
            gunzip.on("end", () => resolve(buffer.join("")))
            gunzip.on("error", (e) => reject(e))
        }
    });

    req.on('error', (err) => reject(err));
    req.end();
})