Axios stream does not 'await' when I process line by line

2.1k Views Asked by At

I need to read a csv file from an external location. Then I need to parse each line and insert them one by one in the DB. Once all is done, I need to do some other DB operations so I need to wait for the parsing/processing is done. But in my code example below, in the .on('data', ...), the 'await' does not wait and the process finishes before all the records are processed. Can you help me to understand what I'm doing wrong? Thanks!

import { promisify } from 'util'; 
import * as stream from 'stream'; 
import csvParser from 'csv-parser'; 
import axios from 'axios';

async function newFunc(url: string) {
  console.log('START newFunc')
  const userService = new UserService()

  const axiosResponse = (async () => {
    const finishedDownload = promisify(stream.finished);
    const csv = csvParser({separator: '|'}) //fs.createWriteStream('/path/to/saved/file');
  
    const response = await axios({
      method: 'GET',
      url: url,
      responseType: 'stream',
    });

    await response.data
        .pipe(csv)
        .on('data', async (userBeneRequest: IUserBeneficiaryBulkStandardRequest) => {
            const a = await userService.getUserById(userBeneRequest.email!) // this is just an example of a DB operation
            console.log('got the user', a)
        })

    await finishedDownload(csv);
  })();

  console.log('FINISHED newFunc')
  return axiosResponse

}

Then in the logs I get:

START newFunc 
FINISHED newFunc 
got the user 1 
got the user 2 
got the user 3 
got the user 4 
got the user 5

As you can see, the 'got the user 1', etc is showing after the 'FINISHED newFunc' and not in the middle of it.

0

There are 0 best solutions below