node-fetch and axios GET call hangs forever

Question

I have this small node-typescript project that gets a bunch of links from shops like ikea, homedepot and amazon from google sheets, makes fetch calls to get some information from a product, and then saves it back into google sheets.

I am running into this problem where most of the fetch calls are working just fine. But for some reason, the a few of them (specifically the home depot ones) just hangs forever. Doesn't timeout, doesn't throw an error, just basically hangs indefinitely.

What I've tried

• I tried with both node-fetch and axios, and both are having the same issue.
• I saw this GH Issue where the person said it doesn't happen on node-fetch v2. I was using ^3.4.4, so I tried switching so a few v2 versions, but still didn't work.
• Tried adding a few different User-Agents and Accept into the request header, and still nothing

My Code

try {
        const productNumber = extractHomeDepotProductNumber(url);
        if (!productNumber) continue;

        const headers = {
          'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
          'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
          'Accept-Encoding': 'gzip, deflate, br',
      }

        const res = await axios.get(
          `https://www.homedepot.ca/api/productsvc/v1/products/${productNumber}/store/7142?fields=BASIC_SPA&lang=en`, {headers}
        );
        console.log('I got res') // this never gets printed
        const resWithImage = await axios.get(
          `https://www.homedepot.ca/api/fbtsvc/v1/fbt/products/${productNumber}/store/7142?checkStockAndPrice=true&lang=en`, {headers}
        );
        console.log('I got resWithImage')
} catch (error) {
        console.log(error) // this never gets printed
}

https://github.com/augustoapg/f2-google-sheet/blob/main-node/app/index.ts#L111-L125

Example of URL with Issues

• https://www.homedepot.ca/product/kraus-mateosingle-lever-pull-down-kitchen-faucet-chrome/1000860141, which is used to get the product number of 1000860141, which is then used to make the fetch call to https://www.homedepot.ca/api/productsvc/v1/products/1000860141/store/7142?fields=BASIC_SPA&lang=en, which is the one that gets the fetch stuck

Fun fact (not sure if relevant)

I was actually doing this project with Bun, but decided to change to Node for issues with deployment. On Bun, this was working just fine when running locally. When building the project, however, I was having the same error.

Answer 1

I got it to work by setting the headers of the request to this:

const headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
}

If I remove any one of these, I get no response. Note, I'm also using a newer version of Chrome for the user agent (just because that's what my local Chrome agent is), but your version works for me too when combined with these other headers.

And, here's my whole stand-alone test code:

import axios from 'axios';


const productNumber = '1000860141';

const headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36',
    'Accept': 'application/json',
    'Accept-Encoding': 'gzip, deflate, br',
}

try {
    const res = await axios.get(
        `https://www.homedepot.ca/api/productsvc/v1/products/${productNumber}/store/7142?fields=BASIC_SPA&lang=en`, 
        { headers, timeout: 2000 }
    );

    console.log(res.data);
} catch(err) {
    console.log(err.message);
}