How to push new request into the queue and send success message that it is done after it pushed to the queue

Question

I have a microservices that process media and at a time I need to provide my microservices to process only one request at atime. So, I need to make a queue using Kue library and one by one pushed new request if it comes. Once 1st request is finished then go to next request but thing is that I have never used queue based processing. please suggest me how to do?

exports['v1'] = (request, response) => {
  /** Image processing and send to s3 */

  const urls = request.body;

  urls.forEach(async url => {
    await bufferConvertUtil
      .buffer(url)
      .then(buffer => {
        return processHelperApp.image(buffer, url);
      })
      .then(data => {
        console.log(data);
      })
      .catch(error => {
        console.log('errr ' + error);
        fs.appendFile('log.txt', `"${url}",\n`, (err, data) => {
          if (err) console.log(err);

        });
      });
  });

  responseUtil.success(response, 'done');
};

I need to add new request into queue using kue or rabbitmq library. Once one queue is finished then processing then process next request.

Answer 1

The best way to do this is using a native for loop and awaiting for all your processes. Like so:

function addErrorToLog(url) {
    return new Promise((resolve, reject) => {
        fs.appendFile('log.txt', `"${url}",\n`, (err, data) => {
            if (err) console.log(err);

            resolve(err || data);
        });
    });
}

exports['v1'] = async (request, response) => {
    /** Image processing and send to s3 */

    const urls = request.body;
    for (let i = 0; i < urls.length; i++) {

        try {
            const url = urls[i];
            const buffer = await bufferConvertUtil
                .buffer(url)
            const data = await processHelperApp.image(buffer, url);
            console.log(data);
        } catch (error) {
            console.log('errr ' + error);
            await addErrorToLog(url);
        }
    }

    responseUtil.success(response, 'done');
};

This way you can wait until one operation finishes to process the next, and I put the error log to a separate function so it is easier for you to change the code without changing the structure.

Hope to have helped!

Answer 2

Finally I have done this by the kue library. Its very simple, anyway see my code

`


 const kue = require('kue')
    const queue = kue.createQueue();

    exports['v1'] = (request, response) => {
    // This is to push the data
    queue.create('key_name', {
      your_key: your_val
    })

    // do the stuff the queue data
    queue.process('key_name', (job, done())=>{
      your_task_funtion(job.data, cb)  // job.data will fetch your data witch you have stored into the queue
    })
    }

`