Canary release when Queues are involved

Question

Fowler says a small percentage of traffic is routed to the Canary version while the old version is still running.

This is assuming that the routing can be controlled at the Load balancer/router level.

We have a use case where a micro-service consumes off a queue and does some processing. We were wondering how the routing can be controlled to direct a subset of traffic to the canary consumer.

One of the options we considered is to have a separate "Canary queue" but the problem is that the producers now have to be aware of this queue which sounds like a smell.

This seems like a common problem where queues are involved. Any ideas on how Canary was adopted for such applications?

Answer 1

There are 2 approaches for canary deployment of queue workers:

1. A dedicated canary queue
2. A common queue

Both these approaches have their pros and cons which are covered in detail here: http://www.varlog.co.in/blog/canary-deployment-workers/

Answer 2

As you wrote, the goal of the canary release is to drive a small fraction of live traffic through a new deployment to minimize the potential impact of flaws in the new deployment. When you do not control the routing to the service under deployment, you can adjust the percentage of traffic handled by the new deployment by adjusting the percentage of new version services to current version services.

For example, your queue is being processed by a pool of 100 service instances at v1. To canary test the next version, deploy 1 to 10 of v2 and turn off 1 to 10 of v1. This will approximate routing 1 to 10% of the traffic to the new service.

If expected throughput of the new version of the service is significantly different, consider adjusting the ratio of new services to old.

If you current deployment of services is very small, consider temporarily increasing the total number of deployed current services before deploying an instance of the new service. For example, assume your active deployment is 3 services. Deploying 6 more of your current service before deploying 1 of your new version service will allow you to keep the traffic to the canary closer to 10%.