kcat get latest message from a topic with a specified key

Question

I am doing integration tests for Kafka and I produce a message with a key integration-test-sub-key and I want to read it later and compare result to what is expected.

I tried doing

$ kcat -b localhost:9092 -t topic-name -C -o -1 -c 1 -k integration-test-sub-key

to filter the message by integration-test-sub-key key, but it doesn't work.

How to do this? Do I need to use grep or something?

Answer 1

Yes, you need to use grep.

Kafka topics are not indexed by key. You need to scan the whole topic using -o earliest, not -o -1, as that means one offset from the end, then remove -c 1.

Also, -k <value> is only a producer option, not a valid consumer flag.

---

If this is a use-case you want to use often, then use ksqlDB, or Kafka Streams to create a KTable, where you have all keys with their latest values.