Kafka go-lang working of enable.auto.offset.store=false with auto.commit = false

Question

Using go-lang 1.18 with confluent-kafka-go v1.8.2

used enable.auto.commit = false config. we are manually committing the offset once the successful processing of the message. However, even after setting this config once we got an error while processing, we are not seeing the message with the same key. (that means somehow offset is getting committed even in error scenarios ). Note: for the error scenario, it took 8-9 sec to process and completely call it an error.

Also got this from link It is recommended to set `enable.auto.offset.store=false` for long-time processing applications and then explicitly store offsets (using offsets_store()) after message processing, to make sure offsets are not auto-committed prior to processing has finished.

Que:

1. By default how much time does kafka wait till it auto-commits the offset.
2. Do we have a mechanism to stop this auto-commit at all.

Answer 1

Offsets are committed at the intervals configured in auto.commit.interval.ms, which by default is 5 seconds.

Setting enable.auto.commit to false should be enough to disable auto-committing completely.

Answer 2

Committing an offset is basically moving the stack pointer and committing all records up to that offset. So, as soon as you successfully process and commit an offset (which should be offset+1 btw), you are committing all the error records too. Manual commit also has to deal with the rebalancing act which can also be a challenge. I find it easier and more accurate to set auto commit and then repost things to the topic that would qualify for a retry. But that may be because I'm just being lazy lol.