Event sourcing microservices: How to manage timestamp

Question

We have microservices, each generating events that are being stored by a event-sourcing repository. We use Cassandra to store the event data.

As you may know, the order of the events is important.

When we generate these events from different services running in different machines, how to manage the time (timestamp) going out of sync across these thereby resulting in an event order mismatch.

Answer 1

This is typically done via vector clocks:

A vector clock is an algorithm for generating a partial ordering of events in a distributed system and detecting causality violations.

Answer 2

As you may know, the order of the events is important.

In some cases - but you'll want to be careful not to confuse time, order, and correlation.

When we generate these events from different services running in different machines, how to manage the time (timestamp) going out of sync across these thereby resulting in an event order mismatch.

Give up the idea that there is an "order" to events that are happening in different places. There is no now.

Udi Dahan on race conditions in the business world:

A microsecond difference in timing shouldn’t make a difference to core business behaviors.

If your micro service boundaries are correct, then events happening in two difference services at about the same time are coincident -- there isn't one correct ordering of them, because (to stretch an analogy) they are in different light cones. The only ordering that is inherently real is that within a single aggregate event history.

What can make real sense is tracking causation; these changes in this book of record are a reaction to those changes in that book of record.

One simple form of this is to track happens-before, which is where ideas like vector clocks begin to appear.

In most discussions that I have seen, this information would be passed along as meta data of the recorded events.

Answer 3

If I understand your problem correctly, you're trying to guard writes, i.e. to make sure that a microservice instance is up to date with all the relevant events before making another write.

In that case, have a look at lightweight transactions, which can be used to implement optimistic locking in Cassandra.

This talk by Christopher Batey is a very good start.