mongodb capped collections consume from offset

68 Views Asked by At

We are considering using MongoDB capped collections as our fifo queues. Our requirements are the following:

  1. Processing of the messages should be done in insertion order
  2. No messages should be lost/skipped
  3. We should be able to start consuming from a specific offset

However we are facing the following issue: Capped collections are guaranteeing read by insertion order. However the _ids are not guaranteed to be monotonic. This means that if there are multiple producers the following situation can occur:

[
  ...
  { 
    _id: 5b72f12599757c9e26c0946b,
    ...
  },
  { 
    _id: 5b72f12599757c9e26c0946d,
    ...
  },
  { 
    _id: 5b72f12599757c9e26c0946c,
    ...
  },
  { 
    _id: 5b72f12599757c9e26c0946e,
    ...
  },
  ...
]

This means that if we start consuming with the following code:

const cursor = collection
  .find({ _id: { $gt: "5b72f12599757c9e26c0946d" } })
  .tailable()
  .cursor();

Then the message with 5b72f12599757c9e26c0946c will be skipped.

So my questions are the following:

  1. Is it possible to guarantee monotonic ids on a capped collection?
  2. Is it possible to start consuming from a specific offset without skiping messages with out of order _id?
  3. Are we missing something?

Thanks in advance.

0

There are 0 best solutions below