org.apache.kafka.clients.NetworkClient Bootstrap broker bootstrap-servers-ip:9092 disconnected

Question

I am running apache kafka on my local system and it is running absolutely fine. But during smoke testing my application is not able to connect to the kafka cluster. It keeps throwing the following error endlessly:

[2016-11-22T23:04:35,017][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker <host1>:9092 disconnected
[2016-11-22T23:04:35,474][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker <host2>:9092 disconnected
[2016-11-22T23:04:35,951][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker <host1>:9092 disconnected
[2016-11-22T23:04:36,430][WARN ][org.apache.kafka.clients.NetworkClient] Bootstrap broker <host2>:9092 disconnected

I am using the below consumer config to connect:

propsMap.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "<host1>:9092,<host2>:9092);
propsMap.put("zookeeper.connect", "<host1>:2181,<host2>:2181");
propsMap.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);  
propsMap.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
propsMap.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "15000");
propsMap.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
propsMap.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
propsMap.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
propsMap.put(ConsumerConfig.GROUP_ID_CONFIG, "test");

Could it be network issue on smoke servers due to which my deployment server is not able to connect to the kafka servers? Because it is working fine on my local and 2 other testing environments.

Could it have something to do with the kafka version?

Or do I need to add some other config such as SSL in this case to connect?

I am new to Kafka, it would really help if someone could point me in the right direction!

Answer 1

If you are using the Kafka 0.9.x.x client or later (which you are if you are using spring-kafka), you don't need the zookeeper.connect property (but that shouldn't cause your problem).

If the broker is down, you should get something like

WARN  o.apache.kafka.clients.NetworkClient - Connection to node -1 could not be established. Broker may not be available.

I suggest you look at the server logs to see if there's anything useful there. You need to talk to your admins to figure out if you need SSL/SASL/Kerberos, etc to connect.

Answer 2

This may be due to server moved to different address or not available at moment. If you still want to go ahead with this assuming the server will come up later, but do not want logs to keep printing "server disconnected" in an infinite loop, use this property.

reconnect.backoff.ms

The base amount of time to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop. This backoff applies to all connection attempts by the client to a broker.

Type: long Default: 50 Valid Values: [0,...]

By default, it retries every 50 milliseconds to reconnect a failed host, this can be increased to, lets say, 5 minutes (300,000ms). By doing so, your logs wouldn't print the infinite disconnection message.

[OPTIONAL] Also, if you are using Apache Camel for routing purpose, use the similar sounding property in camel-kafka component bean definition.

reconnectBackoffMs (producer)