Topic not present in metadata after 60000ms

Question

Use case scenario : I am trying to make sure that in case the broker (one of bootstrap ones) which producer connects to in order to fetch the meta data is not available, then after blocking it for max.block.ms it should retry (atleast 3 retries or until let's say metadata.fetch.timeout.ms is reached - this property no longer exists) with the other brokers available in the nodes list. Properties like retries and retry.backoff.ms is applicable for the sender thread. Can somebody help me configure the Kafka properties for my use case scenario? I believe the properties that will help in this situation are metadata.max.age.ms, request.timeout.ms

Answer 1

You can try below settings to ensure that the producer client retries connecting to different brokers in the list if the initial connection to the bootstrap broker fails

       bootstrap.servers=broker1:9092,broker2:9092,broker3:9092
metadata.max.age.ms=5000
retries=3
retry.backoff.ms=1000
request.timeout.ms=30000
connections.max.idle.ms=10000

I do not believe Kafka has a particular property to configure number of retries for metadata fetching. The metadata.max.age.ms and connections.max.idle.ms properties indirectly influence this behavior.