Kafka message aggregate

Question

I have messages in Kafka topic which is compacted as below

1::3$
2::4$

Later sometime, some more values has been added to the same keys

1::64$
2::145$

My desire to achieve from above messages is from a topic as below

1::3$,64$
2::4$,145$

Any suggestion how to combine the value to the same key in a topic

Answer 1

You can use kafka streams to do that:

Topology:

1. Source processor: Read input topic as KStream

2. Processor 1: GroupbyKey() -> produces KGroupedTable, pre req. to aggregate

3. Processor 2: Aggregate() -> produces a KTable with combined values like you need. You can materialize the KTable as compacted topic. Then you can read it with a kafka consumer or with interactive queries.