Not able to read messages from kafka consumer in kafka cluster setup

Question

I have created two kafka brokers in a kafka cluster. When one broker is down I am not able to get any data to kafka consumer. I am using this command to read messages from consumer:

bin/kafka-console-consumer.sh --topic test_kafka_cluster \
    --bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning

Answer 1

There are two parameters that affect topics availability for a consumer:

• min.insync.replicas (minISR): Minimum number of in-sync partition's replicas.
• replication.factor (RF): Total number of partition's replicas.

If you want your consumer to survive a broker outage, then you must have RF > minISR. With RF=2 and minISR=2 you can't tolerate any broker down, with RF=3 and minISR=2 you can tolerate 1 broker down, with RF=5 and minISR=2 you can tolerate 3 brokers down, and so on.

Note that the internal __consumer_offsets topic is used to store consumer offsets and has a default RF value of 3, which is not achievable in your cluster of 2 nodes. So you also need to set offsets.topic.replication.factor=1 at the cluster level.

Answer 2

Here as per your console consumer configuration, IP address used here are 127.0.0.1 and 127.0.0.2 and two bootstrap servers are configured as 9092.

Verify both the ip's are reachable

bin/kafka-console-consumer.sh --topic test_kafka_cluster \
--bootstrap-server 127.0.0.1:9092,127.0.0.2:9092 --from-beginning

Ideally when we run tow kafka broker instance it will be running in two different ports.

Presuming Kafka is running in local Eg: localhost:9092 localhost:9093

Kafka instances running on two different host: Eg: 127.0.0.3:9092, 127.0.0.2:9092

If Kafka is running on docker/docker toolbox:

Console consumer on Docker toolbox:

docker exec <container-name> kafka-console-consumer --bootstrap-server 192.168.99.100:9093 --topic <topic-name> --from-beginning

Console consumer on Docker:

docker exec <container-name> kafka-console-consumer --bootstrap-server localhost:9093 localhost 9092 --topic <topic-name> --from-beginning