how to manage a series of API calls in node.js

Question

I have a series of API endpoints that I need to poll in series in order to build up a canonical list:

/cities
/homes
/parcelvalue

so that polling /cities gives me a list of cities, polling /homes gives me a list of all addresses in a given city, and polling /parcelvalue gives me the value of a given home.

Ultimately, I am going to build up a file tree that saves the incremental output of polling each endpoint into a directory for auditing purposes:

|
\-cities/cities.json
|
\-homes/homes.json
|
\-parcelvalue/parcelvalue.json

so that cities.json is the raw output of the /cities enpoint, homes.json contains the results of polling the /homes endpoint for each city in cities.json, and parcelvalue.json contains the output of polling the /parcelvalue endpoint for each home in homes.json. I want each step to rely on disk I/O, so that the function that builds homes.json starts with the cities.json file.

The easiest way that I can see to do this would be by having each step 1. synchronously load the file that might inform it, 2. synchronously poll the appropriate end point, and then 3. synchronously saving the file that each function might generate:

a getCities() function might check to see if cities.json existed, and produce that file from the endpoint if it did not, a getHomes() function might import cities.json and then poll the /homes endpoint for each city in its list and then generate homes.json, and a getParcelValues() function might import homes.json and then poll the /parcelvalue endpoint for each home in its list and then generate parcelvalue.json

(this is a simplification, I'd like to add by-function filtering and error-handling, etc., but I'm trying to keep this example focused.)

Unfortunately, a thorough examination of synchronous http request libraries this evening hasn't resulted in any that will work in my circumstance.

Is there a good way to use something like async.series or another similar async workflow library to make use of, i.e. the async request library? I would like each of these functions to be called in series, getCities() -> getHomes() -> getParcelValues(), and to be encapsulated as much as might be possible.

Answer 1

try something like this :

var async = require('async');

    async.auto({
        cities : function(cb) {
    //do your stuff ...
            if (err)
                cb(null, false);
            else 
                cb(null, result);
        },
        homes : ['cities', function(cb, result) {
            if(!result.cities)
                cb(null, false)
            else {
                // do your stuff
                if (err)
                    cb(null, false);
                else 
                    cb(null, homeResult);
            }
        }], 
        parcels : ['homes', function(cb, result) {
            if(!result.homes)
                cb(null, false)
            else {
                // do your stuff
                if (err)
                    cb(null, false);
                else 
                    cb(null, parcelResult);
            }
        }]
    },function(err, allResults){
        //do final works
    });