Pulsar message bus: Can we configure message retention per topic?

Question

We have a usecase where our messages processor, processes messages with very complex calculations on messages from few topics. These calculations are not finishing within the rention period configured. Instead of increaseing message retention globally for all topics can we set this for the few topics where we need messages to be retained for more time? Message retention per topic is possible?

Answer 1

This is exactly the use case that namespaces were designed for. In Pulsar, a namespace is an administrative unit within a tenant that contains a subset of topics. The configuration policies set on a namespace apply to all the topics created in that namespace. The best way to address this issue is use the either the REST API or the pulsar-admin CLI tool to create a new namespace, e.g.

pulsar-admin namespaces create <current-tenant>/<a-new-namespace>

Once you have the new namespace, you can then configure the retention policy for the new namespace using the CLI tool, e.g.

$ pulsar-admin namespaces set-retention <current-tenant>/<a-new-namespace> \
  --size -1 \
  --time -1

Finally, you will want to re-create the topics in the new namespace, and change your code to use these new topics.

Answer 2

If you use a subscription on the topic, then you don't have to worry about message retention. Messages in a subscription backlog are not removed until they are acknowledged, regardless of the retention setting. Just use the Consumer interface in the client and acknowledge each message after the calculations are done.

For more details about retention and subscription backlogs, see this Understanding Pulsar Message TTL, Backlog, and Retention

Answer 3

Most documentation refers to these policies set at namespace. I see no mention of ability to set these at topic level.

But if you dig into Admin API, I see API capabilities to set most useful policies at topic level - which should be overriding any at namespace Example: https://pulsar.apache.org/admin-rest-api/?version=3.0.0/#operation/PersistentTopics_setInactiveTopicPolicies https://pulsar.apache.org/admin-rest-api/?version=3.0.0/#operation/PersistentTopics_setMessageTTL

Its not clear why documentation explicitly mention this AND Admin UI also does not expose this.

In general it does make sense to manage this at namespace. But its important for developers to know if there is a way to override this at topic level. My understanding is we can do so with Admin API