I am building an application with a node/express backend. The endpoint has a really long response time due to heavy computation (take 1 - 2 mins). Heroku times me out at 30 seconds.
I'm using Kue and redis in order to fulfill the computation in redis.
A little bit of pseudo 1. When a user hits the endpoint 2. Initiate the job 3. Send response after invoking job to not time out of heroku 4. Once Job is complete, respond/send completed job/data => which will go into redux.
Here is my code so far:
const redis = require('redis');
const client = redis
.createClient();
const kue = require('kue');
const queue = kue.createQueue();
router.post('/endpoint', (req, res) => {
const param = req.body.param;
var job = queue
.create('job', param)
.priority('high')
.save(err => {
if (err) {
console.log('failed');
process.exit(0);
return;
}
job.on('complete', result => {
res.send(result);
console.log('completed');
});
job.on('failed', errorMessage => {
console.log(errorMessage);
process.exit(0);
});
});
queue.process('job', async (job, done) => {
// heavy heavy computation
// heavy heavy heavy heavy
// heavy heavy heavy
//output is results => passed into promise
done(null, await Promise.all(results));
});
res.json({ message: 'job in progress' });
});
The problem with this code is that I can't send a header response once it has been set. I am very new to redis and very new kue and background jobs. I've read webhooks can help, but I don't have much experience with webhooks either.
Thank you in advance!