how Compaction works in Apache Kafka

Question

My input is 1:45$ and we are processing the message and next I am updating the 1:null.

I do see still the 1:45$ in the topic along with the 1:null (I can see both the messages)

I want output to be 1:null in the same topic.

I have used this code:

kafka-topics --create --zookeeper zookeeper:2181 --topic latest-       product-price --replication-factor 1 --partitions 1 --config "cleanup.policy=compact" --config "delete.retention.ms=100"  --config "segment.ms=100" --config "min.cleanable.dirty.ratio=0.01"



kafka-console-producer --broker-list localhost:9092 --topic latest-    product-price --property parse.key=true --property key.separator=::

1::45$
1::null


kafka-console-consumer --bootstrap-server localhost:9092 --topic latest-product-price --property  print.key=true --property key.separator=:: --from-beginning

But I do not find any compaction in my case and need some inputs to make the value as 1::null

Answer 1

Compaction in Kafka is not immediate. If you send two messages with the same key to a compacted topic, and you have a live consumer on that topic, that consumer will see both messages come through.

Periodically, there's a background cleaner thread that goes looking for duplicate keys in compacted topics, and removes the overwritten records, so that a consumer that pulls down the data after that log cleaner has run, will only see the last change/update for a particular key. So, topic compaction seems to be better suited for consumers that run periodically, not ones that are active 100% of the time.

One thing that you can tune how often this background log cleaner thread runs, to maybe run those consumers more often. Look for the log.cleaner configuration parameters in the Kafka documentation: https://kafka.apache.org/documentation/#brokerconfigs

There's a good explanation on how Kafka log compaction works at this link: https://medium.com/swlh/introduction-to-topic-log-compaction-in-apache-kafka-3e4d4afd2262