We are considering using MongoDB capped collections as our fifo queues. Our requirements are the following:
- Processing of the messages should be done in insertion order
- No messages should be lost/skipped
- We should be able to start consuming from a specific offset
However we are facing the following issue: Capped collections are guaranteeing read by insertion order. However the _ids are not guaranteed to be monotonic. This means that if there are multiple producers the following situation can occur:
[
...
{
_id: 5b72f12599757c9e26c0946b,
...
},
{
_id: 5b72f12599757c9e26c0946d,
...
},
{
_id: 5b72f12599757c9e26c0946c,
...
},
{
_id: 5b72f12599757c9e26c0946e,
...
},
...
]
This means that if we start consuming with the following code:
const cursor = collection
.find({ _id: { $gt: "5b72f12599757c9e26c0946d" } })
.tailable()
.cursor();
Then the message with 5b72f12599757c9e26c0946c will be skipped.
So my questions are the following:
- Is it possible to guarantee monotonic ids on a capped collection?
- Is it possible to start consuming from a specific offset without skiping messages with out of order _id?
- Are we missing something?
Thanks in advance.