Apache Kafka: reduce kafka disk usage

Question

I have a question about Kafka's disk. Kafka will fail when its disk become full. So I want to reduce the disk usage to less than x% by discarding the old data stored on the Kafka disk (or discarding a copy of the data) when the Kafka disk usage reaches x%. Do I need to modify the Kafka source code to do this?

Answer 1

You can configure retention.bytes for your topics.

This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes.

See https://kafka.apache.org/documentation/#topicconfigs