I have 16 queues
& mulitple consumer servers
for those queues. I have created
one dedicated
channel for each queue
to consume
messages. Consumer
& dispatch
channels
on each server share same connection
.
When I dispatch messages
to each queue, I do the following:
- create a new channel
- bind channel to the queue with proper routing
- dispatch the message
- close the channel
I have lots of incoming webhooks from Shopify & these webooks contents are dispatched to specific queues.
While processing each message, I need make an API call to Shopify. Shopify API has rate limit. If I hit rate limit once, I redispatch all messages from the consumer back to rabbitmq with a delay header of 1 minute(time required to clear the API rate limit).
Now, when I have several consumers running with lots of messages in the queue & I re-dispatch those messages, I get too many channels
error for a period of time. How can I avoid this error?
I tried to keep 2 dedicated channels per queue:
- for conusmer purpose only
- for dispatch purpose only
For, 16 queues, & around 11 consumer servers. This way, I always have to keep 352
channel open. This caues CPU utilization on rabbitmq host server to reach >90%
which is also an issue. As the server can crash any time.
I found the solution to the problem after digging through the
RabbitMQ documentation
.Instead of creating a new channel for each dispatch, I
created a single channel
&kept it alive
for theentire
connectionsession
. Whencreating
the channel, Iasserted
all theexchanges
that would be used by my queues.Then I just
publish
the messages to thedesired exchange
with therouting key
. As my queues are already bonded with the exchanges & listen for messages with a given routing key, the messages end up in the correct queue!This way I can maintain just
01
connection & only01
channel per server!