Shovel Configuration between 2 AWS MQ private RabbitMQ clusters

Question

So I have 2 AWS MQ RabbitMQ clusters in the same vpc without public accessibility. Both are multi az Clusters with 3 nodes.

Lets say

• rabbitmq-1
• rabbitmq-2

Now I want to create a dynamic shovel from 1 to 2 in the Rabbitmq management console.

I'm trying to create with

• source_protocol = amqp091
• source_uri = amqps://user1:pass1@b-1xxxx:5671
• destination_protocol = amqp091
• destination_uri = amqps://user2:pass2@b-2xxxx:5671

But the shovel status is stuck in starting:

Also in cloudwatch sometimes I see the following error :

Shovel with the name 'test-bill ' was not found on the target node '[email protected]' and / or virtual host '/'

I don't get what exactly happens with the shovel creation and the 3 nodes. Sometimes I can't even delete shovel from the console. I think it is randomly distributed between nodes. Any ideas on what is the issue and shovel doesn't connect successfully?

Answer 1

So With AWS Support help , it seems that 2 private AWS MQ RabbitMQ clusters cannot connect via shovel.

Answer 2

Yes, right Shovel does not transfer messages directly between private brokers. But you can make this happen with the help of one public broker between them. you need to create one public broker to act as a middleman between two brokers.

so there is basically two process: first, you have to transfer messages between your source private broker to the public broker, and second is transfer message between the public to destination private broker. in both cases, you need to create a shovel in the private broker.

first process: in your private source broker create one shovel. Give the queue name for the source and destination broker. but When specifying src-uri (amqp://), do not change this field because the rabbitmq automatically takes care of its own URI if you give the src-uri in this amqps://{broker-name}:{password}@{host}:{port} formate it won't work and stuck on the starting state. Now change dest-URI according to the public broker URI; for example amqps://{broker-name}:{password}@{host}:{port}. create the shovel and it starts in running state. so the first step is done all the messages are transferred from a private source to a public broker.

second process: in your destination broker create a shovel and now your public broker is your source broker and the destination broker is your private broker. so the src-uri is amqps://{broker-name}:{password}@{host}:{port} and dest-uri is amqp:// again do not change this fild. now create a shovel with the appropriate source and destination queue name. check that the shovel is in running state and all the messages are transferred between public to private.

so in this way, you transfer messages between your two private brokers without much effort.