When running a project I encountered some RebalanceInProgressException which occurrs once or twice a week and make the consumer fail. Here is the message:
org.apache.kafka.common.errors.RebalanceInProgressException:
Offset commit cannot be completed since the consumer member's generation is
already stale, meaning it has already participated another rebalance and got
a new generation. You can try completing the rebalance
by calling poll() and then retry commit again
I'd like to understand the fundamentals of Kafka and what actually heppened under the hood causing this exception. Does it mean that committing happened when rebalance was in progress? Why is it prohibited and caused exception?