Kafka: Adjust retention automatically based on lag

Question

Given that I write an application that collect data from IoT devices and let my customer subscribe to this data by providing me with their http endpoint credentials.

I have to deal with their endpoint not responding or being slow, therfore I will bufferize the messages until sent (consumed), which will require storage.

To limit this storage I am wondering if I can watch the lag of my consumers and, when it reaches a threshold automatically increase the topic rentention (that will decrease automatically back later)

This would help me to set a short retention by default and yet be able to handle unavailable external endpoints without loosing messages. (Of course, if lag keeps growing, I'll have to take other actions).

My question is then, is this possible with Kafka ? and are there things I should be carafeful whith when doing this way ?

Many thanks

Answer 1

You can adjust retention time of topic vis kafka command line tools:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name test --alter --add-config  retention.ms=55000

Or if you want to do it within your code take a look at TopicCommand class.