We have below setup MSK Topic(10 partitions) -> Trigger -> Lambda(Single Group - Single consumer)
Lambda Details: Batch Size : 1 Reserved Concurrency : 1 Lambda Retry : 0 Lambda Timeout : 2mins
We have a MSK Topic with 10 partitions. MSK trigger listens to the topic and triggers the lambda function. Lambda function will take 20 to 40 seconds to complete.
Producer sends message in bulk. Eg: they push 10 messages together. In our consumer lambda it is processed single event at a time.
We have observed, Lamda function reprocess the message that is already processed. While investigating, we have noticed, consumer is getting rebalaced often.
Topic with 1 Partition: We have changed the topic to 1 partition. MSK Topic(1 partition) -> Trigger -> Lambda(Single Group - Single consumer) After changing the topic to 1 partition it works fine.
Can you please help us to debug this issue. Why it is reprocessing message in 10 partitions?
If a Kafka consumer doesn't receive a heartbeat so often (configurable, but maybe not in Lambda), then it will rebalance a group.
Kafka doesn't know if/when record-content is processed; it only knows about offsets. If you rebalance, then an offset is not committed, and so this will be re-polled, and re-processed.
You need to store some lock, such as in DynamoDB along with a UUID per record to prevent global re-processing of the actual record content.
If you have 10 partitions, you should set concurrency up to 10 as well.