I have an app with a RabbitMQ consumer consuming messages from a queue bound to a Consistent hash exchange. The idea here is to process messages with same routing key in sequential order (i.e routing key & app instance should have 1:1 mapping). When the app instances are scaled up, new queues get created along with the consumer which are again bound to the same consistent hash exchange. Now the exchange starts routing messages to the new queue as well.
But the problem here is that some of the routing keys which were earlier getting routed to a different queue get routed to the new queue.
eg: Lets say I have 2 app instances A1 & A2 consuming messages from Q1 & Q2 queues respectively. Both these queues are bound to a single consistent hash exchange. Here messages with routing keys R1 & R2 are routed to Q1. R3 & R4 are routed to Q2.If a new queue Q3 gets added by adding 1 more app instance A3, messages R2 & R4 may start getting routed to Q3. Challenge: we have messages M1-M1000 with routing key R2 coming in. M2000-M3000 with routing key R4 coming in. A1 & A2 were able to consume 100 messages (1900 more in ready state each queue). App scaling happened at this juncture. A3 came up and Q3 got created. If new messages come with routing keys R2 & R4 they get into Q3 and are consumed immediately. But the messages that were already in ready state get consumed later in A1 & A2. Order of processing messages with same routing key is messed up.
For now the app scaling is a manual activity. Its not auto scaling. Its a dotnet app hosted in cloudfoundry.
How do I change this design such that order of messages processed does not change while scaling the app instances up or down. I have no issues if the routing-key assignment to queue changes after all the ready messages are consumed.
I had a thought of unbinding all the queues from exchanges until all messages with ready state get consumed and then scaling the app. (already have some code that writes messages into DB when RMQ connection goes down so that they are later published to exchange when connection resumes. this can be reused so as to not lose the messages until the app scaling completes.) Wondering if there is any better way to achieve this.