I'm currently developing a Dot Net 7 microservices architecture using RabbitMQ, and I'm encountering challenges with message consumption. My setup involves a topic exchange and queue binding to ensure that each service consumes messages from a dedicated queue (e.g., the user service consumes messages from the user queue).
The complication arises when there are multiple instances of the same service, leading to multiple consumers listening on the same queue. I'm seeking guidance on achieving the following objectives:
1- Ordering: I need to guarantee that messages related to the same user, such as "user-created" and "user-updated" are processed in the correct order because if "user-created" message is produced and then a "user-updated" message is produced the "user-created" might take time to be consumed by consumer instance 1 while the "user-updated" message has been picked up and consumed by consumer instance 2.
2- Affinity: It's crucial that messages for a specific user consistently get processed by the same consumer instance, ensuring data consistency.
I've explored solutions like the topic exchange, but with the presence of multiple instances, I'm unsure about the most effective approach. Could you provide insights, best practices, or examples to help me address these challenges and maintain message order and consumer affinity in a microservices environment with multiple active instances of the same consumer?
In Kafka, when you set a message key, Kafka uses consistent hashing to determine the partition for that key, ensuring messages with the same key always go to the same partition and consequently, the same consumer instance. I couldn't achieve this using RabbitMQ.