Description
I'm using this kafka client - https://github.com/confluentinc/confluent-kafka-dotnet Consumers hangs when committing offset after few seconds internet connection lost.
How to reproduce
I have running working consumer - everything is OK. But If I turn Off Wifi and turn On again on my LapTop (consumer on my local machine - but production the same) - it consumes first messege - and hangs on Commit - as a result - Zombie consumer thread. I'v tried to run Commit in separated thread with timeout - in case of timeout - to Close current consumer - and recreate it - but it hangs on Close() method ( the same - with Unassign and Unsubscribe and Dispose methods)
The only one way works - I run Commit() in separated thread with timeout- in case of timeout - just recreate another consumer loop (thread) -without calling Close(), Unassign ,Unsubscribe ,Dispose methods. But the number of threads increases every time I do this trick - maybe because of deadlocked threads.
Maybe similar - https://github.com/dpkp/kafka-python/issues/1728 and this - https://github.com/confluentinc/confluent-kafka-dotnet/issues/1552
[ 1.6.3 (but 1.5.3 - the same)] Confluent.Kafka nuget version. [ Windows 10 Home] Operating system.
Consumer Configuration:
var config = new ConsumerConfig
{
BootstrapServers = _settings.BootstrapServers,
GroupId = consumerGroupName,
EnableAutoCommit = false,
AutoOffsetReset = AutoOffsetReset.Earliest,
SaslUsername = _settings.SaslUsername,
SaslPassword = _settings.SaslPassword,
SecurityProtocol = KafkaProducer.SecurityProtocol,
SaslMechanism = KafkaProducer.SaslMechanism,
AllowAutoCreateTopics = true,
Debug = "consumer,cgrp,topic,fetch"
};
Logs: Here everything is OK:
..............
[thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Fetch topic vad-UpdateFrameworkFromImportCommand [1] at offset 25 (v4) [thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Fetch topic vad-ImportFrameworksConsumerRetry [0] at offset 0 (v4) [thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Fetch 2/2/2 toppar(s)
Wifi is turned off:
[thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Disconnected (after 18793ms in state UP) [thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Fetch backoff for 500ms: Local: Broker transport failure [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Disconnected (after 18781ms in state UP) [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Fetch backoff for 500ms: Local: Broker transport failure [thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Disconnected (after 24894ms in state UP) [thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Fetch backoff for 500ms: Local: Broker transport failure [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:GroupCoordinator]: GroupCoordinator: 94.131.241.251:9093: Disconnected (after 24050ms in state UP) [thrd:main]: Group "ImportFrameworksConsumer" changed state up -> query-coord (join-state steady) [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord sasl_plaintext://94.131.241.251:9094/2: Disconnected (after 18793ms in state UP) Confluent.Kafka.Consumer2[System.String,System.String] sasl_plaintext://94.131.241.251:9093/1: Disconnected (after 18781ms in state UP) Confluent.Kafka.Consumer2[System.String,System.String] sasl_plaintext://94.131.241.251:9095/3: Disconnected (after 24894ms in state UP) Confluent.Kafka.Consumer2[System.String,System.String] GroupCoordinator: 94.131.241.251:9093: Disconnected (after 24050ms in state UP) Confluent.Kafka.Consumer2[System.String,System.String] 4/4 brokers are down Confluent.Kafka.Consumer2[System.String,System.String] [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query Exception thrown: 'System.Net.Sockets.SocketException' in System.dll Exception thrown: 'System.ObjectDisposedException' in System.dll Exception thrown: 'System.Net.WebException' in System.dll Exception thrown: 'System.Net.WebException' in System.dll Exception thrown: 'System.ObjectDisposedException' in System.dll Exception thrown: 'System.Net.WebException' in System.dll Exception thrown: 'Elasticsearch.Net.ElasticsearchClientException' in Elasticsearch.Net.dll [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Group "ImportFrameworksConsumer": no broker available for coordinator query: intervaled in state query-coord [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [0]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [1]: broker is down: re-query [thrd:main]: Topic vad-ImportFrameworksConsumerRetry [2]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [0]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [1]: broker is down: re-query [thrd:main]: Topic vad-UpdateFrameworkFromImportCommand [2]: broker is down: re-query [thrd:main]: Consumer group session timed out (in join-state steady) after 10184 ms without a successful response from the group coordinator (broker 1, last error was Success): revoking assignment and rejoining group [thrd:sasl_plaintext://94.131.241.251:9094/bootstrap]: sasl_plaintext://94.131.241.251:9094/2: Connect to ipv4#94.131.241.251:9094 failed: Unknown error (after 21036ms in state CONNECT) sasl_plaintext://94.131.241.251:9094/2: Connect to ipv4#94.131.241.251:9094 failed: Unknown error (after 21036ms in state CONNECT) Confluent.Kafka.Consumer2[System.String,System.String] [thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Connect to ipv4#94.131.241.251:9095 failed: Unknown error (after 21040ms in state CONNECT) [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Connect to ipv4#94.131.241.251:9093 failed: Unknown error (after 21041ms in state CONNECT) sasl_plaintext://94.131.241.251:9095/3: Connect to ipv4#94.131.241.251:9095 failed: Unknown error (after 21040ms in state CONNECT) Confluent.Kafka.Consumer2[System.String,System.String] sasl_plaintext://94.131.241.251:9093/1: Connect to ipv4#94.131.241.251:9093 failed: Unknown error (after 21041ms in state CONNECT) Confluent.Kafka.Consumer2[System.String,System.String] [thrd:GroupCoordinator]: GroupCoordinator: 94.131.241.251:9093: Connect to ipv4#94.131.241.251:9093 failed: Unknown error (after 21054ms in state CONNECT) GroupCoordinator: 94.131.241.251:9093: Connect to ipv4#94.131.241.251:9093 failed: Unknown error (after 21054ms in state CONNECT) Confluent.Kafka.Consumer`2[System.String,System.String]
Wifi is turned on:
[thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Fetch topic vad-ImportFrameworksConsumerRetry [1] at offset 0 (v4) [thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Fetch topic vad-UpdateFrameworkFromImportCommand [2] at offset 18 (v4) [thrd:sasl_plaintext://94.131.241.251:9095/bootstrap]: sasl_plaintext://94.131.241.251:9095/3: Fetch 2/2/2 toppar(s) [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Fetch topic vad-UpdateFrameworkFromImportCommand [0] at offset 9 (v4) [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Fetch topic vad-ImportFrameworksConsumerRetry [2] at offset 0 (v4) [thrd:sasl_plaintext://94.131.241.251:9093/bootstrap]: sasl_plaintext://94.131.241.251:9093/1: Fetch 2/2/2 toppar(s)