I have a Kafka cluster with 3 brokers configured as follows:
Broker-0 : broker.id=0 ; listeners=PLAINTEXT://localhost:9092 ; log.dirs=/opt/data/kafka/logs-0
Broker-1 : broker.id=1 ; listeners=PLAINTEXT://localhost:9093 ; log.dirs=/opt/data/kafka/logs-1
Broker-2 : broker.id=2 ; listeners=PLAINTEXT://localhost:9094 ; log.dirs=/opt/data/kafka/logs-2
The producer targets Broker-2 to publish messages
sh-3.2# ./kafka-console-producer.sh --broker-list localhost:9094 --topic clusterTopic
I started testing with all brokers up and running
sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 2,0,1
sh-3.2#
A this point, all messages published by the producer are received by the consumer
Then I killed Broker-0 (the current leader)
sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0 Leader: 1 Replicas: 0,1,2 Isr: 2,1
sh-3.2#
At this point, no messages published by the producer are received by the consumer anymore
Then, I enabled back Broker-0
sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0 Leader: 1 Replicas: 0,1,2 Isr: 2,1,0
sh-3.2#
At this point, all messages published by the producer are received properly by the consumer
Then I killed Broker-1 (the current leader)
sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 2,0
sh-3.2#
At this point, all messages published by the producer are still received properly by the consumer.
Why is the behavior so different from broker to broker?
Update #1
I noticed something that might be helpful:
When I kill Broker-1, I see the following warning on the consumer console (a bunch of times)
[2018-06-07 13:59:00,866] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
But when I kill Broker-0, I see these warning messages (a bunch of times)
[2018-06-07 14:04:08,433] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 2147483647 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-06-07 14:04:08,434] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
I wonder what is the broker with ID=2147483647?