My Bullmq working not running tasks for all jobs in the queue

1.2k Views Asked by At

I am running a background job to generate pdfs using puppeteer and i am emailing out these pdfs using recipients name and email. below is my worker logic

const sendCertificatesWorker = new Worker(
queueName,
async job => {
const {
data: { id, name, email, link },
collectionId,
nameOfOrganization,
thumbnail,
raw
} = job.data;

    // Generate Certificate
    const filePath = await GenerateCertificatePdfs(job.data.data, raw);
    await job.updateProgress("certificate generated");
    
    //Send email
    const result = await sendCertificatesEmail({
      name,
      email,
      link,
      filePath,
      nameOfOrganization,
      thumbnail
    });
    
    if (result.error) {
      await job.updateProgress(
        `email not sent and the reason is: ${result.message}`
      );
    
      await Certificate.updateOne(
        { _id: id },
        { $set: { status: "not sent" } }
      );
    
      // Update DB
      await job.updateProgress(
        `cert with id ${id} updated in database as not sent`
      );
    } else {
      await job.updateProgress(`email sent`);
    
      await Certificate.updateOne({ _id: id }, { $set: { status: "sent" } });
    
      // Update DB
      await job.updateProgress(
        `cert with id ${id} updated in database as sent`
      );
    }
    
    // Update collection
    const collection = await Collection.findById(collectionId);
    
    if (collection) {
      await job.updateProgress("updating the collection..");
      await collection.save();
    }
    
    return { sent: result.error };

},
{ connection: workerConnection }
);

The problem now is that all jobs are added to the queue but not all gets executed. some exits the worker before compeleting any of the worker tasks. take a look at the logs from my server


A job with ID 22 is waiting
A job with ID 23 is waiting
A job with ID 24 is waiting
Job 22 is now active; previous status was waiting
Job 23 is now active; previous status was waiting
job with 23 is currently in progress: certificate generated
job with 23 is currently in progress: email sent
job with 23 is currently in progress: cert with id 64afda0da236e94d7290bd47 updated in database as sent
job with 23 is currently in progress: updating the collection..
23 has completed and returned false
Job 24 is now active; previous status was waiting
23 has completed!
22 has completed and returned false
job with 24 is currently in progress: certificate generated
job with 24 is currently in progress: email sent
job with 24 is currently in progress: cert with id 64afda0da236e94d7290bd3f updated in database as sent
job with 24 is currently in progress: updating the collection..
24 has completed and returned false
24 has completed!

Job Id 22 exited without generating the pdf nor email it.

I had reason to believe that it was a memory issue so i set up my redis instance like this

// Configure redis connection
const queueConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: false,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});

const workerConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: true,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});

But the issue still persist. Any one got tips on what might be going on?

0

There are 0 best solutions below