How to make a Redis (Bull) queue in a Node.js environment that's deployed to Heroku?

Question

I've looked through all of the documentation, and I'm having some trouble finding examples of the correct implementation. First of all, I have the Heroku Redis add-on as you can see below:

https://i.stack.imgur.com/9kcXW.png

I installed Redis and Bull to my Node.js environment and set up Redis as follows:

const redis = require('redis');
const Queue = require('bull');
const redisClient = redis.createClient(process.env.REDIS_URL, {
    tls: {
        rejectUnauthorized: false
    }
});
const bullQueue = new Queue('queue', process.env.REDIS_URL)

I'm trying to run the below function as a background task (this is what Heroku recommends for functions that take longer than 0.5 seconds to complete):

app.post('/', async function(request, response) {
  const client = await pool.connect() //pool is just a node-postgres pool
  try {
    await client.query('BEGIN')
    let data = await function1(argumentList);
    await function2(data);
    await client.query('COMMIT')
  } catch(err) {
    await client.query('ROLLBACK')
    console.log(err)
  }
  try {
    await client.query('BEGIN')
    const setOfItems = await function3()
    var array = []
    for (item of setOfItems) {
      if (conditional) {
        array.push(item)
      }
    }
    await client.query('COMMIT')
    let job = await bullQueue.add()
    response.send(JSON.stringify(array))
  } catch(err) {
    await client.query('ROLLBACK')
    console.log(err)
  }
});

This function does some web scraping, database calls, and other things, so like I said it takes a few seconds to complete. How should I change my function to add the request to the background queue and then return the JSON.stringified 'array' back to the user. Sorry if this is a noob Redis question, but I've looked at all the docs and I really am not sure how to proceed.

Answer 1

For starters, const redis = require('redis'); isn't necessary. Bull connects to Redis under the hood, so you need only provide process.env.REDIS_URL for Heroku as you're doing. (heroku config | grep REDIS returns the endpoint URLs if needed)

Currently, all of your database code isn't delegated to the task queue at all. Try registering a processing function associated with the queue using

bullQueue.process(job => {/* your database work goes here */})

The parameter job needs to be populated with whatever serializable data your worker needs to do its job. You populate this parameter using

bullQueue.add(/* job data goes here */)

A high-level approach might be something like:

const Queue = require('bull');
const bullQueue = new Queue('queue', process.env.REDIS_URL)

bullQueue.process(async job => {
  console.log(job.data.argumentList);

  /* do SQL stuff */
  
  return [1, 2, 3, 4];
});

app.post('/', async (request, response) => {
  const job = await bullQueue.add({argumentList: ['some', 'data']});
  const array = await job.finished();
  response.send(JSON.stringify(array))
});

Many other strategies exist for getting the results. See returning job completions for some ideas.

See also:

• Queue#process
• Queue#add
• How to listen to the completed event in bull queue - just for the current job.