We are currently investigating how to implement Marten as an event store for our monolith application. For simplicity the scenario is scoped to this:
One monolith application that:
- Stores events in an outbox table
- Uses an outbox service to publish the events (using MassTransit and RabbitMQ)
One microservice that have:
- Marten as an event store
- A consumer that will subscribe to specific events (from the outbox service) and append those to Marten (using MassTransit and RabbitMQ)
How can we ensure the ordering of the events that the microservice will consume from the outbox service? As we know, ordering is not guaranteed using RabbitMQ and if the events are appended in the wrong order to Marten, the whole idea of the event sourcing fails.
Also, there might be errors while trying to consume or append the events to Marten. How to deal with that and make sure that we do not append the events in the wrong order to the event store in Marten? Also, how to deal with errors when we try to consume and append events to Marten? It seems very likely that we will end up with an event stream that is out of order. How are people dealing with this?
If we were to couple the outbox service with the Marten Event store then we could append the event to the stream instead of using RabbitMQ for publishing the event, this would ensure the ordering as we would never move forward to the next event unless the previous was successfully appended. Is this the solution we should use to guarentee the ordering? It doesn't feel right and would create a highly coupled architecture.
Any guidance and suggestions are very welcome.
On the sending side if you can have a gapless sequence you can send in any order. On the receiving side you can insert them into a temp table. And read from the table into marten. When you experience a gap you stop till the next number in the sequence has landed.
In the sending side you can have a gapless sequence by populating an incrementing sequence into a column