Getting too many channels error for a period of time

Question

I have 16 queues & mulitple consumer servers for those queues. I have created one dedicated channel for each queue to consume messages. Consumer & dispatch channels on each server share same connection.

---

When I dispatch messages to each queue, I do the following:

1. create a new channel
2. bind channel to the queue with proper routing
3. dispatch the message
4. close the channel

---

I have lots of incoming webhooks from Shopify & these webooks contents are dispatched to specific queues.

While processing each message, I need make an API call to Shopify. Shopify API has rate limit. If I hit rate limit once, I redispatch all messages from the consumer back to rabbitmq with a delay header of 1 minute(time required to clear the API rate limit).

Now, when I have several consumers running with lots of messages in the queue & I re-dispatch those messages, I get too many channels error for a period of time. How can I avoid this error?

---

I tried to keep 2 dedicated channels per queue:

1. for conusmer purpose only
2. for dispatch purpose only

For, 16 queues, & around 11 consumer servers. This way, I always have to keep 352 channel open. This caues CPU utilization on rabbitmq host server to reach >90% which is also an issue. As the server can crash any time.

Answer 1

I found the solution to the problem after digging through the RabbitMQ documentation.

Instead of creating a new channel for each dispatch, I created a single channel & kept it alive for the entire connection session. When creating the channel, I asserted all the exchanges that would be used by my queues.

Then I just publish the messages to the desired exchange with the routing key. As my queues are already bonded with the exchanges & listen for messages with a given routing key, the messages end up in the correct queue!

This way I can maintain just 01 connection & only 01 channel per server!