I am running a background job to generate pdfs using puppeteer and i am emailing out these pdfs using recipients name and email. below is my worker logic
const sendCertificatesWorker = new Worker(
queueName,
async job => {
const {
data: { id, name, email, link },
collectionId,
nameOfOrganization,
thumbnail,
raw
} = job.data;
// Generate Certificate
const filePath = await GenerateCertificatePdfs(job.data.data, raw);
await job.updateProgress("certificate generated");
//Send email
const result = await sendCertificatesEmail({
name,
email,
link,
filePath,
nameOfOrganization,
thumbnail
});
if (result.error) {
await job.updateProgress(
`email not sent and the reason is: ${result.message}`
);
await Certificate.updateOne(
{ _id: id },
{ $set: { status: "not sent" } }
);
// Update DB
await job.updateProgress(
`cert with id ${id} updated in database as not sent`
);
} else {
await job.updateProgress(`email sent`);
await Certificate.updateOne({ _id: id }, { $set: { status: "sent" } });
// Update DB
await job.updateProgress(
`cert with id ${id} updated in database as sent`
);
}
// Update collection
const collection = await Collection.findById(collectionId);
if (collection) {
await job.updateProgress("updating the collection..");
await collection.save();
}
return { sent: result.error };
},
{ connection: workerConnection }
);
The problem now is that all jobs are added to the queue but not all gets executed. some exits the worker before compeleting any of the worker tasks. take a look at the logs from my server
A job with ID 22 is waiting
A job with ID 23 is waiting
A job with ID 24 is waiting
Job 22 is now active; previous status was waiting
Job 23 is now active; previous status was waiting
job with 23 is currently in progress: certificate generated
job with 23 is currently in progress: email sent
job with 23 is currently in progress: cert with id 64afda0da236e94d7290bd47 updated in database as sent
job with 23 is currently in progress: updating the collection..
23 has completed and returned false
Job 24 is now active; previous status was waiting
23 has completed!
22 has completed and returned false
job with 24 is currently in progress: certificate generated
job with 24 is currently in progress: email sent
job with 24 is currently in progress: cert with id 64afda0da236e94d7290bd3f updated in database as sent
job with 24 is currently in progress: updating the collection..
24 has completed and returned false
24 has completed!
Job Id 22 exited without generating the pdf nor email it.
I had reason to believe that it was a memory issue so i set up my redis instance like this
// Configure redis connection
const queueConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: false,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});
const workerConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: true,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});
But the issue still persist. Any one got tips on what might be going on?