We have 3 spring kafka consumers, using ackmode.batch (from my understanding, it will only commit the offsets after it processed the whole batch). As well as 3 partitions with all a committed offset.
During rolling update re-deployment, we are noticing missing messages. With 1 consumer, we aren't able to reproduce the missing messages, so it seems the missing messages only occur if we have 3 consumers. The rebalancing, which will take a bit of time, will eventually rebalance the 3 consumers to the 3 partitions.
I'm wondering what the issue of the message loss could be, what exactly happens during a rebalance whilst some consumers may be re-started and are in the midst of processing messages? If the messages are not fully processed, the offsets shouldn't be committed, hence the messages will be re-processed, but in my case, the messages are not being re-processed, hence missing messages.