I am using npm module node-fetch to call online shop URLs; some are timing out.
This happens when the call is made with http1.1 which is used in most API npm modules such as node-fetch. Can someone help me to understand how to make a call with http1.1 and get a response back for following examples:
curl -m 10 -v --http1.1 "https://www.dhgate.com" \
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.98" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "Connection: keep-alive" \
-H "Content-Type: application/json"
or
curl -m 10 -v --http1.1 "https://www.abercrombie.com/shop/us" \
-H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.98" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "Connection: keep-alive" \
-H "Content-Type: application/json"
In node-fetch I tuned on debugging which is printing logs from http npm module by NODE_DEBUG=http,net,stream and here are logs, perhaps it's helpful for troubleshooting:
HTTP 67536: createConnection www.dhgate.com:443::::::::::::::::::::: [Object: null prototype] {
protocol: 'https:',
slashes: true,
auth: null,
host: 'www.dhgate.com',
port: 443,
hostname: 'www.dhgate.com',
hash: null,
search: '?wgu=8477_127107_16977422063155_b7229050b9&wgexpiry=1729278206&f=bm%7Caff%7Cwg%7C55163%7C8477_127107_16977422063155_b7229050b9%7C0%7C%7C%7Cus%7C%7C%7C&utm_source=webgains_us&utm_medium=Affiliate&utm_content=0&utm_campaign=affid',
query: 'wgu=8477_127107_16977422063155_b7229050b9&wgexpiry=1729278206&f=bm%7Caff%7Cwg%7C55163%7C8477_127107_16977422063155_b7229050b9%7C0%7C%7C%7Cus%7C%7C%7C&utm_source=webgains_us&utm_medium=Affiliate&utm_content=0&utm_campaign=affid',
pathname: '/product/fashion-women-sandals-summer-flats-sexy-ankle/479784035.html',
path: null,
href: 'https://www.dhgate.com/product/fashion-women-sandals-summer-flats-sexy-ankle/479784035.html?wgu=8477_127107_16977422063155_b7229050b9&wgexpiry=1729278206&f=bm%7Caff%7Cwg%7C55163%7C8477_127107_16977422063155_b7229050b9%7C0%7C%7C%7Cus%7C%7C%7C&utm_source=webgains_us&utm_medium=Affiliate&utm_content=0&utm_campaign=affid',
method: 'GET',
headers: [Object: null prototype] {
Accept: [
'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.98'
],
'Accept-Encoding': [ 'gzip, deflate, br' ],
Connection: [ 'keep-alive' ],
'User-Agent': [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36'
],
'Content-Type': [ 'application/json' ]
},
agent: undefined,
_defaultAgent: Agent {
_events: [Object: null prototype] {
free: [Function (anonymous)],
newListener: [Function: maybeEnableKeylog]
},
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object: null prototype] {
keepAlive: true,
scheduling: 'lifo',
timeout: 5000,
noDelay: true,
path: null
},
requests: [Object: null prototype] {},
sockets: [Object: null prototype] {
'www.dhgate.com:443:::::::::::::::::::::': []
},
freeSockets: [Object: null prototype] {
'track.webgains.com:443:::::::::::::::::::::': [Array]
},
keepAliveMsecs: 1000,
keepAlive: true,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 1,
maxCachedSessions: 100,
_sessionCache: { map: [Object], list: [Array] },
[Symbol(kCapture)]: false
},
keepAlive: true,
scheduling: 'lifo',
timeout: 5000,
noDelay: true,
servername: 'www.dhgate.com',
_agentKey: 'www.dhgate.com:443:::::::::::::::::::::'
}
NET 67536: pipe false null
NET 67536: connect: find host www.dhgate.com
NET 67536: connect: dns options { family: undefined, hints: 1024 }
NET 67536: connect: autodetecting
HTTP 67536: sockets www.dhgate.com:443::::::::::::::::::::: 1 2
HTTP 67536: outgoing message end.
STREAM 67536: read 0
STREAM 67536: need readable false
STREAM 67536: length less than watermark true
STREAM 67536: do read
NET 67536: _read - n 16384 isConnecting? true hasHandle? true
NET 67536: _read wait for connection
STREAM 67536: resume
STREAM 67536: resume true
STREAM 67536: flow true
STREAM 67536: read undefined
STREAM 67536: need readable true
STREAM 67536: length less than watermark true
STREAM 67536: reading, ended or constructing false
NET 67536: connect/multiple: only one address found, switching back to single connection
NET 67536: connect: attempting to connect to 23.44.84.179:443 (addressType: 4)
NET 67536: afterConnect
NET 67536: _read - n 16384 isConnecting? false hasHandle? true
NET 67536: Socket._handle.readStart
STREAM 67536: read 0
STREAM 67536: need readable true
STREAM 67536: length less than watermark true
STREAM 67536: reading, ended or constructing false
STREAM 67536: onWriteComplete 0 undefined
NET 67536: _onTimeout
HTTP 67536: CLIENT socket onTimeout
NET 67536: destroy
NET 67536: close
NET 67536: close handle
NET 67536: emit close
HTTP 67536: CLIENT socket onClose
HTTP 67536: removeSocket track.webgains.com:443::::::::::::::::::::: writable: false
NET 67536: _onTimeout
HTTP 67536: CLIENT socket onTimeout
STREAM 67536: readableAddChunk null
STREAM 67536: onEofChunk
STREAM 67536: emitReadable_ false 0 true
STREAM 67536: flow true
STREAM 67536: read undefined
STREAM 67536: endReadable false
STREAM 67536: read 0
STREAM 67536: endReadable false
STREAM 67536: endReadableNT false 0
STREAM 67536: endReadableNT true 0
NET 67536: _final: not ended, call shutdown()
NET 67536: afterShutdown destroyed=false
NET 67536: destroy
NET 67536: close
NET 67536: close handle
NET 67536: emit close
HTTP 67536: CLIENT socket onClose