RabbitMQ Bunny Parallel Consumers

Question

I have built an application which consists of one publisher, several queues and several consumers for each queue. Consumers on a queue (including the queue) share the channel. Other queues use a different channel. I am observing that for different queues, tasks are being worked on parallel but for a specific queue this is not happening. If I publish several messages at once to a specific queue, only one consumer works while the other ones wait until the work is ended. What should I do in order for consumers to work on parallel?

workers.each do |worker|
  worker.on_delivery() do |delivery_info, metadata, payload|
    perform_work(delivery_info, metadata, payload)
  end
  queue.subscribe_with(worker)
end

This is how I register all the consumers for a specific queue. The operation perform_work(_,_,_) is rather expensive and takes several seconds to complete.

Answer 1

RabbitMQ works off the back of the concept of channels, and channels are generally intended to not be shared between threads. Moreover, channels by default have a work thread pool size of one. A channel is an analog to a session.

In your case, you have multiple consumers sharing a queue and channel, and performing a long-duration job within the event handler for the channel.

There are two ways to work around this:

1. Allocate a channel per consumer, or
2. Set the work pool size of the channel on creation See this documentation

I would advocate 1 channel per consumer since it has a lower chance of causing unintended side-effects.