We have a number of dotNet services running in ECS which consume data from RabbitMQ queues, I want to scale up the number of instances based on the number of messages in the queues.
Let's say I want to have one instance running if the queue is below 100, two instances for 100-199 and so on up to 500+ (5 instances). [The values here are for example purposes only]
I have created a CloudWatch Alarm which monitors the queue in question and goes into alarm when it hits 100.
I have enabled the service auto-scaling, set the min (1) and max (5) and created a scaling policy which monitors that alarm and sets the scaling actions as follows...
Add 1 task from 100 to 200
Add 1 task from 200 to 300
Add 1 task from 300 to 400
Add 1 task from 400 to +infinity
This appears to work though I do seem to see more instances than I expect sometimes.
What I cannot work out is how to scale down correctly. If I add another policy for scale down, as follows...
Remove 1 task from 100 to 200
Remove 1 task from 200 to 300
Remove 1 task from 300 to 400
Remove 1 task from 400 to +infinity
...I see the number of tasks bouncing up and down (say) between two and three when the messages are perhaps at 250.
Can someone please point me to a good example of how exactly to set this up. I've found a number of articles which skirt round the edges but can't find that one decent example of exactly how to configure this scenario.
Your help would be much appreciated.