How to send large number of requests from axios?

Question

I want to send about 2000 api calls to retrieve data from some api. How do i archive this without having any bottlenecks using axios?

Answer 1

2000 api calls in parallel or one after another? it would be really resource intensive if it is in parallel, depending on your device it might me possible or maybe not.

but if it is one after another it can be easily achieved with a simple loop.

here is what I would do in case of parallel implementation: this would be a little trial and error in the beginning to find the best parallelism factor.

use lodash chunks to make the complete 2000 requests to chunks of size x api request objects (reference: https://www.geeksforgeeks.org/lodash-_-chunk-method/) now for each chunk, we will make api calls in parallel. that is each chunk will be in sequential order but within each chuck we will make parallel api calls.

sample code:

// Requiring the lodash module
// in the script
const _ = require("lodash");

async function _2000API_CALLS() {

 const reqs = [... your 2000 request params]
 
 // Making chunks of size 100
 const chunks = _.chunk(reqs, 100) // x = 100

 for(const c of chunks) {
   const responses = await Promise.all(c.map(() => {
    return axois... // api call logic
   }))
   // accumulate responses in some array if u need to store it and use it later
 }
}

Answer 2

If you need to execute all the requests in parallel, here is an example:

const endpoints = [
  'url_1',
  'url_2',
  // ...
];
await Promise.all(endpoints.map(async url => {
  const data = await axiosClient.get(url);

  // do sth with the response
});