How to implement consumption peer to peer with sarama library?

Question

Now, I try to use Shopify/sarama library(kafka go client) to implement multiple consumers in the same consumer group, and I found several consumer groups with same client did not work.

I have the following testing: Suppose the topic have 3 partitions, and the group Id is "groupId":
1. TEST1
(1)

newClient, err := sarama.NewClient(brokerList, &cfg)

(2)

group1, err := sarama.NewConsumerGroupFromClient(groupId, newClient)  
group2, err := sarama.NewConsumerGroupFromClient(groupId, newClient)
group3, err := sarama.NewConsumerGroupFromClient(groupId, newClient)

The testing is try to use a kakfa client(newClient) to create multiple groups, and the program will block one group, and the other two groups will not response, it seems to block the following code: group.Consume(ctx, strings.Split(topics, ","), consumer) and then a moment later, the program rebalance (setup/clean up) repeatly, the result is that the consumer consumed all partitions, not each consumer consume per partition.

2. TEST2
(1)

newClient1, err := sarama.NewClient(brokerList, &cfg)
group1, err := sarama.NewConsumerGroupFromClient(groupId, newClient1)

(2)

newClient2, err := sarama.NewClient(brokerList, &cfg)
group2, err := sarama.NewConsumerGroupFromClient(groupId, newClient2)

(3)

newClient3, err := sarama.NewClient(brokerList, &cfg)
group3, err := sarama.NewConsumerGroupFromClient(groupId, newClient3)

In this testing, we can see each consumer group with a new client, the result indicates that each consumer will consume a partition, which accord with expection.

So, How I can implement multiple consumer groups bind the same kafka client, rather than creating a new client per consumer group?

Answer 1

This isn't the recommended way to run multiple consumers. Compile one Go app with only one consumer. Run three processes instead. They will rebalance by using the same consumer group, but they will no longer not block one another within the same process.