I have a scenario where my node js client collects data from rest api.
Scenario : my api endpoint is like this http://url/{project} where project is parameter. the project comes from a Database table.
here is my procedure:
- I am getting all the projects names from Database to a list
- using a loop calling rest endpoint for every project in the list
My Query: If I have less number of projects in the Database this procedure working fine but, If I have around 1000 projects to collect, the requests are taking long time and some times failing due to timeout errors.
How can I scale this process so that it finish collecting data in a good amount of time?