Using dns resolve based custom look up function takes longer for parallel requests

570 Views Asked by At

I am running hundreds of parallel HTTP requests for all different domains using node http module. If I understood it correctly, node http module does dns lookup using worker threads and hence number of lookups at a time are limited by libuv threads. In order to speed up execution and avoid this supposed congestion, i supplied my custom lookup function which used dns resolve.

However, I am surprised that total time to process (receive http full response i.e res end event) 300 http requests increased from 1.4 seconds (when using default lookup function) to 6 seconds (when using dns resolve)

My custom look up function

const staticLookup = () => async (hostname: string, _: null, cb: Function) => {
                console.log("DNS look up started for: " + hostname);
                // const ips = await dnsPromises.resolve(hostname);
                dns.resolve(hostname, (err, records) => {
                    const ips = records;
                    // console.log("ips: " + JSON.stringify(ips));
                    if (ips.length === 0) {
                        throw new Error(`Unable to resolve ${hostname}`);
                    }
                    console.log("DNS look up ended for: " + hostname);
                    cb(null, ips[0], 4);
                });
            };


As far as I know there is no DNS cache on linux based OS. What is causing such a big jump when replaced with dns resolve? Does it not use same dns servers as dns lookup ? Still 6 seconds is a long time as compared to 1.4 seconds for processing http requests

0

There are 0 best solutions below