how to serialize jobs in BullMQ

Question

How to serialize jobs in job queue of BullMQ? I need that the next job is scheduled for execution only when the previous job is finished. I am using BullMQ 3.

Answer 1

You can use the FlowProducer to create this type of behaviour.

An example would be:

const flowProducer = new FlowProducer({connection: {
        host: "host_here",
        port: 6379
    }});

    const jobChain = await flowProducer.add({
        name:'example',
        data: { field: 'value'},
        "queueName",
        children: [{
            name: 'example2', //can be same/different name
            data: {field: 'otherValue'},
            "queueName", // can be a different queuename
            children: ...
        }],
    });

Here the the "example" job would only be executed when "example2" is successfully processed. This has to do with both the FlowProducer behaviour & general understanding of the job lifecycle.

Lifecycle info: https://docs.bullmq.io/guide/architecture

Flows info: https://docs.bullmq.io/guide/flows

Answer 2

Berkan gave you already the reference to the BullMQ documentation about Flows (see https://docs.bullmq.io/guide/flows). If you look it up, it has an example where more than one level of hierarchy is used, you chain it as deep as you like. Here’s the code example:

import { FlowProducer } from 'bullmq';
const flowProducer = new FlowProducer();

const queueName = 'assembly-line';
const chain = await flowProducer.add({
  name: 'car',
  data: { step: 'engine' },
  queueName,
  children: [
    {
      name: 'car',
      data: { step: 'wheels' },
      queueName,
      children: [{ name: 'car', data: { step: 'chassis' }, queueName }],
    },
  ],
});

The order of processing would be: chassis, wheels and finally engine, so from the lowest level (chassis) up till the parent (engine).