How to allow Kafka producer to start even if available node in the brower is lower than replication factor?

Question

I am currently working with Kafka and i have a configuration with 2 nodes. (I know that i must have at least 3 nodes for HA, but for many reasons i can't have more than 2 nodes)

So, i have 2 nodes configured with kafka with a replica factor = 2 and that is working great. If one node crash, my events are still sent to my consumers.

The problem is that if i start a new producer, i have an error ": error while fetching metadata … invalid REPLICATION_FACTOR". Because my producer try to create a topic (with replica factor = 2), but a this moment i only have one node available.

Is there a way / configuration to allow this behavior ? I would like to be able to start my producer even if have only have 1 kafka node available, and then when the second node will be up again then the replication will happen.

Thank you for your help guys.

Answer 1

my producer try to create a topic

Firstly, you should disable auto topic creation on the broker.

In any case, the producer is not the problem... If the topics already exist, then you would simply set acks=1 in the producer, and min.in.sync.replicas=1 as a topic config.

For non existent topics, you really have no option here other than ensuring your cluster is fully operational. You'll get the same error using AdminClient directly (or other tooling to create topics).

You'll also need to ensure the offsets and transaction topics have more than one replica, otherwise consumers will also stop working when the one replica becomes unavailable.