How to send kafka matching data to other topic

Question

How to push matching data between topic 1 and topic 2 in another topic 3 ? when sending messages from producer to consumer?

Answer 1

I have not worked with Spark but I can give you some direction form Apache Storm perspective Apache Storm

1. Build a topology with 2 kafka spouts each consuming from topic1 and topic2

2. Consume this data in a bolt and compare the data. You may use single bolt or series of successive bolts. You may need to use some persistence viz. mongodb or something such as redis or memcache , depending on you comparison logic

3. Push the common data to new kafka topic Send data to kafka from Storm using kafka bolt

This is very Apache Storm specific solution, may not be the most ideal or suitable or efficient one, but aimed to give general idea

Here is a link to basic concepts in storm Storm Concepts

Answer 2

I've been working with Spark for over six months now, and yes it is absolutely possible. To be honest, fairly simple. But putting spark on is a bit exaggerated for this problem. What about Kafka Streams? I have never worked with them, but should this not solve exactly this problem?

If u want to use spark: Use the Spark Kafka integration (I used spark-streaming-kafka-0-10) to consume and to produce the Data, shoud be very simply. Than look for the Spark streaming Api in the documentation.

A simple join about the 2 DStreams should solve the problem. If u want to store Data who doesn`t match u can window it or use the UpdateStateByKey function. I hope it helps someone. Good Luck :)