Returning csv data from multiple files in a directory in JavaScript

Question

I am trying to get data from multiple CSV files using 'csvtojson'.

1. Acquired the array of file names in a directory.
2. Ran forEach to get the data from different CSV files in JSON and pushed into a single array.

The problem is I cant send data to the client because promises take longer to fetch data and I am unable to figure out a way.

Heres my code :

router.get("/covidData/", (req, res) => {
const jsonData = [];
const files = fs.readdirSync(__dirname.replace("routes", "public/covidData"));


const fetchData = () => {
  files.forEach(async (file) => {
    let response = await csvtojson({
      includeColumns: /Date Announced|Detected State|Current Status/,
    }).fromFile(__dirname.replace("routes", "public/covidData/" + file));

    jsonData.push(...response);
  });
};
fetchData();
res.send(jsonData);
});

Answer 1

Your issue is that you need to await all of the promises before you try to send data back to the client. You can do that by mapping over your promises and then using await Promise.all before calling res.send. The following should get you most of the way there.

router.get("/covidData/", async (req, res) => {
  const files = fs.readdirSync(__dirname.replace("routes", "public/covidData"));

  const data = files.map((file) => {
    return csvtojson({
      includeColumns: /Date Announced|Detected State|Current Status/,
    }).fromFile(__dirname.replace("routes", "public/covidData/" + file));
  });

  const finalData = await Promise.all(data);
  res.send(finalData.flat());
});