Tombstone records not processed in Kstream-Ktable join

270 Views Asked by At

I have this kstream topology:

KStream<String, Data> eventStream = streamsBuilder.stream("event",
        Consumed.with(Serdes.String(), eventSerde);

KTable<String, Result> resultKTable = streamsBuilder.table("result",
        Consumed.with(Serdes.String(), resultSerde));

eventStream.leftJoin(resultKTable, new Joiner())
        .to("result", Produced.with(Serdes.String(), resultSerde));

When I sent tombstone record in "event" topic, it gives me this error

Skipping record due to null join key or value. key=[3428642] value=[null] topic=[event] partition=[0] offset=[21]

EDIT
I did the following workaround to handle tombstone records:

eventStream.filter((k,v) -> v != null)
           .leftJoin(resultKTable, new Joiner())
           .to("result", Produced.with(Serdes.String(), resultSerde));
eventStream.filter((k,v) -> v == null).to("result");
1

There are 1 best solutions below

2
hradecek On

It is not an error, but warning.

As written in Kafka Streams documentation for KStream-KTable left join:

Input records for the stream with a null key or a null value are ignored and do not trigger the join

So, this is expected scenario.


Side note: There is written blogpost at luppeng.wordpress.com about this and possible workaround, which might come handy.