Separate dead letter exchange necessary in RabbitMQ?

Question

I've set up a dead letter routing with my queue to requeue rejected messages with a delay of several seconds, preventing temporary consumer errors to clog up the queue. I've set this up so both the work queue and the dead letter queue are bound to the same exchange:

Externally produced, incoming messages are routed to the exchange, which places them in the work queue. During processing the message, a consumer might fail due to some temporary errors (think a crawler receiving an error 500 from a website).
Instead of rejecting the message and having it placed at the head of the queue again (leading to an infinite loop), we route rejected messages (with requeue=0) to the exchange, adding the dead letter queue as the routing key. Here, every message receives a TTL of X seconds, after which it will be rejected, and therefore routed back to the exchange with the routing key se to the original work queue.

However, looking at literature and examples online, everyone seems to recommend routing to a separate dead letter exchange:

Externally produced, incoming messages are routed to the work exchange, which places them in the work queue. If a consumer fails, messages are rejected (with requeue=0) and will be routed to the dead letter exchange. The dead letter exchange routes the messages to the dead letter queue, where the message TTL will expire, and the again-rejected messages will be routed back to the work exchange.

---

Are there some crucial advantage of the second design compared to the first? I cannot identify any, but then again I'm not too confident with RabbitMQ.

Answer 1

It depends on the type of exchange you're using, and what other routing you need to do. If you use the same exchange for both the original messages and the re-queued ones, you will need to distinguish between:

• New messages, which might be routed to multiple queues, or none at all
• Failed messages, which should be routed only to the delay queue
• Delayed messages, which should be routed only to the single queue where they failed

In my implementation of this pattern I use two extra exchanges (both declared dynamically on-demand) so that it is as independent as possible from the original routing configuration:

• The original message is acknowledged by the consumer, and manually re-published, to a "begin" exchange. This allows some extra flexibility, such as setting custom headers on the message, and having multiple delay queues with different TTLs attached to the same work queue.
• The "begin" exchange is a fanout exchange, with a binding to a specific "waiting" queue, as in your second diagram.
• When the message TTL expires in the "waiting" queue, it is routed to a separate "done" exchange, set as the Dead Letter Exchange.
• That exchange is also a fanout exchange, with only a binding to the original work queue. This ensures no extra copies of the message are created to other queues which successfully processed it the first time.
• The message therefore arrives back in the original queue, with its original routing key.