Speed issues looping through node-fetch API calls - is there a better way?

569 Views Asked by At

I am fetching over 7 million records from an API using fetch-node in a node js script.

The records each have a sequential id starting from 1, so I using a for loop to access each record. The main issue I am having is speed!

Initially I did one a time, but I have tried using promise all to do them in batches (see code below) - which has greatly improved the speed, but I think there must be a more efficient way to do this.

(async () => {

    const start = 1;
    const records_to_fetch = 7000000;
    const finish = start + records_to_fetch + 1;

    for(i = start; i < finish; i += 5) {
        
        await Promise.all([
            getHistory(i),
            getHistory(i+1),
            getHistory(i+2),
            getHistory(i+3),
            getHistory(i+4),
        ]);
        
    };
}

})();

Without the promise.all, the API can't handle the volume of requests when doing it in batches, however as each function call is independent from each other I don't need all 5 to finish before starting the next one. Ideally I would have 5 "slots" (or more in reality) so I can limit the number of concurrent calls, but when one finishes the next id is called.

Part of the problem is that I don't even know how to phrase my question when searching for an answer! I am hoping someone can understand what I'm trying to achieve and give me some help!

0

There are 0 best solutions below