Kafka cluster does not work if broker with ID=0 is down

2.1k Views Asked by At

I have a Kafka cluster with 3 brokers configured as follows:

Broker-0 : broker.id=0 ; listeners=PLAINTEXT://localhost:9092 ; log.dirs=/opt/data/kafka/logs-0
Broker-1 : broker.id=1 ; listeners=PLAINTEXT://localhost:9093 ; log.dirs=/opt/data/kafka/logs-1
Broker-2 : broker.id=2 ; listeners=PLAINTEXT://localhost:9094 ; log.dirs=/opt/data/kafka/logs-2

The producer targets Broker-2 to publish messages

sh-3.2# ./kafka-console-producer.sh --broker-list localhost:9094 --topic clusterTopic

I started testing with all brokers up and running

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 0   Replicas: 0,1,2 Isr: 2,0,1
sh-3.2#

A this point, all messages published by the producer are received by the consumer

Then I killed Broker-0 (the current leader)

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 1   Replicas: 0,1,2 Isr: 2,1
sh-3.2#

At this point, no messages published by the producer are received by the consumer anymore

Then, I enabled back Broker-0

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 1   Replicas: 0,1,2 Isr: 2,1,0
sh-3.2#

At this point, all messages published by the producer are received properly by the consumer

Then I killed Broker-1 (the current leader)

sh-3.2# ./bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic clusterTopic
Topic:clusterTopic  PartitionCount:1    ReplicationFactor:3 Configs:
Topic: clusterTopic Partition: 0    Leader: 0   Replicas: 0,1,2 Isr: 2,0
sh-3.2#

At this point, all messages published by the producer are still received properly by the consumer.

Why is the behavior so different from broker to broker?

Update #1

I noticed something that might be helpful:

When I kill Broker-1, I see the following warning on the consumer console (a bunch of times)

[2018-06-07 13:59:00,866] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

But when I kill Broker-0, I see these warning messages (a bunch of times)

[2018-06-07 14:04:08,433] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 2147483647 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-06-07 14:04:08,434] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20882] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

I wonder what is the broker with ID=2147483647?

0

There are 0 best solutions below