How to handle multiple update event when there is more then one replica of a pod

Question

I have two services name Product and Order. Order table inside OrderDb has price and productId columns for storing product price and product id that ordered. Order services has 3 replica.

Now, suppose a product is ordered and it's id 80, and a series of sequential update event fired from product service to order services for that particular product:

event1:{productId: 80, price: 500}
event2:{productId: 80, price: 600}
event3:{productId: 80, price: 400}
event4:{productId: 80, price: 900}
event5:{productId: 80, price: 100}

so the end price should be 100 for that product, but sometimes these events are processed in random order such as

event1:{productId: 80, price: 500}
event2:{productId: 80, price: 600}
event5:{productId: 80, price: 100}
event4:{productId: 80, price: 900}
event3:{productId: 80, price: 400}

since event 3 processed last, price become 400.

Answer 1

This is generally up to your database. I see you put NATS in the tags so I'm assuming you mean you have some kind of worker queue model, but you probably have a database of record behind that with its own consistency model. With event streaming systems where you want to defend against out of order or multi-delivery, you would include more info in the queue message such as an object version, or just the previous price. In the latter, simpler case it would be like

event1:{productId: 80, price: 500, oldPrice: 0}
event2:{productId: 80, price: 600, oldPrice: 500}
event3:{productId: 80, price: 400, oldPrice: 600}
event4:{productId: 80, price: 900, oldPrice: 400}
event5:{productId: 80, price: 100, oldPrice: 900}

That would let your code refuse to apply the operation if the base state no longer matches. But that's pretty limiting, you don't want everything to fail after a re-order, you just want convergent behavior. This the point where I just yell "VECTOR CLOCKS" and leap out of the window. Designing distributed, convergent systems is indeed quite hard, look up the term CRDT as a starting point.

Answer 2

I think your issue results from not exactly knowing about the delivery guarantees of your message broker (probably NATS).

First, you should work out which guarantees you need, and then choose a messaging technology. It makes a huge difference if you need strict ordering on all events, on all events regarding one entity, or no ordering at all. Same goes for delivery semantics: at-most-once, at-least-once, exactly-once.

If those things are clear, then choose messaging technology fulfilling these requirements, otherwise you will end up adding complexity by workarounds (like the suggestion by coderanger). There are many messaging protocols out there, like AMQP, MQTT, NATS, Kafka, etc...

Please be aware that this is the cost of having a distributed architecture! There is no way around thinking about the issues of distributed systems when doing microservices.