Kafka persistent storage is growing, though cleanup and retention policies are set

Question

We are seeing a steady growth in the persistent storage for Kafka, around 10% per day - effectively implying the configs are not tuned. There are 3 brokers and 5 topics. Retention was configured for 2 hours.

Following are the config parameters set -

Topic level :

   retention.ms=7200000
   delete.retention.ms=7200000
   segment.ms=7200000
   cleanup.policy=delete
   file.delete.delay.ms=1
   segment.bytes=150000

Broker Level:

"log.cleaner.delete.retention.ms": "7200000"
"log.retention.ms": "7200000"

Any help would be appreciated. Thanks in advance.

Answer 1

Seems to have fixed this issue.. Steps performed -

1. logged into kafka broker
2. fetch the list of files in the descending order of their sizes from the /opt/kafka/data-0/logs/ path.
3. mapped the file to a topic already present in kafka - should be easily identifiable because of the naming convention
4. applied the above settings on each of the topics - whose log files were greater than 10 MB
5. retention time set to 30 minutes, instead of 3 hours as posted in the question.