I am using the request module to crawl a list of URLs and would like to limit the number of open sockets to 2:
var req = request.defaults({
forever: true,
pool: {maxSockets: 1}
});
req(options, function(error, response, body) {
... code ...
done();
however, when looping over an array of URLs and issuing a new request to each - that does not seem to work.
is there a way to get the current number of open sockets to test it?
I believe that
maxSocketsmaps tohttp.Agent.maxSockets, which limits the number of concurrent requests to the same origin (host:port).This comment, from the developer of
request, suggests the same:In other words, you can't use it to limit the number of concurrent requests in general. For that, you need to use an external solution, for instance using
limiterorasync.queue.