kafka consumers reading and writing to same partition

Question

can 2 consumers from different groups read from the same topic and partition x and than both write into another topic and partition y?

I would implement a consumer strategy in which one consumer discards data that the other consumer processes. The order in which the data is persisted by the producers into the shared partition is not important.

i just want to know if this is possible

Answer 1

You could have two different approaches here :

• The consumers are in different consumer groups. It means that both will receive all messages and it's up to your business logic to ignore some message for a consumer while the other is processing them
• The consumers are in the same consumer group so that they'll receive different partitions and you have messages delivered to a consumer a not to the other. You are leveraging a Kafka feature but it depends on how you will "partition" your data if using the default round robin way or using a key per message or a custom partitioner.

The only concern I see in the second solution is the rebalance. If a consumer goes offline and then comes back, a rebalance happens and the partitions could be distributed in a different way. I.e. you start having consumer C1 reading from partitions P1 and P2 and C2 reading from partitions P3 and P4. Imagine that C2 goes offline. First of all if it's offline for more time, the C1 will get all partitions (I don't know if it's good for your scenario) but when C2 comes back online the rebalance happens and assigned partitions could be different, i.e. P1 and P2 assigned to C2 (not C1 anymore) and P3 and P4 to C1 (not C2 anymore); it depends if this sort of consumers swap on partitions is a problem for your application logic.