I am building a job queue (Nodejs+Mongodb)which has this kind of structure :
{
"_id" : ObjectId("53e09abd23bbaeea776598e1"),
"createDate" : ISODate("2014-08-05T08:50:05.643Z"),
"status" : "new",
......
}
I have defined 4 statuses in my system as : new, waiting, working, failed, done
Requirement: I need to pick the most recent job in the job queue with status as "new".
Currently, I am doing this by using
jobQueue.findAndModify(
{
status: 'new',
},
{date: 1},
{$set: { ... } },
{remove: false},
function (err, job) {......
which works fine when the job documents are less. But there have been situations when document nos. rises to 50k. Then the locking period (using mongostat) goes beyond 100%(at times reaches 200% ). Hence my server is not able to deliver the new jobs in such scenario.
Can anybody suggest me a more efficient way to this query??
Thanks in advance :)