Quarkus/SmallRye Reactive Messaging/Kafka connect with multiple clusters

Question

Currently I have a Quarkus app which consumes from a Kafka topic and produces on another Kafka topic. It uses SmallRye Reactive Messaging. It works well. Due to external changes the topic to produce on and the topic to consume from will be on Kafka servers which are on a different cluster (and should not/cannot be combined in one cluster).

In the application configuration (yaml) we set the Kafka server (broker):

kafka:
  bootstrap:
    servers: localhost:9092

Adding a server here does not help, it then tries to spread data over the brokers which is not my intention.

Is it possible to connect to multiple clusters (maybe set a server per topic)? Can't find anything on that on internet not in the Quarkus docs nor in the SmallRye docs.

Answer 1

It's not crystal clear but it's explained on the documentation for exemple on the kafka inbound documentation it is said that you can configure your broker gloablly (using kafka.<props>) or per channel.

You can pass to each channel all supported Kafka properties :

mp.messaging.incoming.chanel1.bootstrap.servers=kafka1:9092
mp.messaging.incoming.chanel2.bootstrap.servers=kafka2:9092

Answer 2

Remove the below property from property file

kafka.bootstrap.servers

Next add the below property . What we are doing here is instead of setting bootstrap servers globally for all your channel in your application, we are setting for individual channel .

mp.messaging.incoming.channel1_name.bootstrap.servers=kafka1:9092 mp.messaging.incoming.channel2_name.bootstrap.servers=kafka1:9092

Once you set the above property in your application.properties and run it will show docker container failed to start ... something like that and a warn message like No config value of type [java.lang.String] exists for: kafka.bootstrap.servers.

This is due to dev services that is active. For this to go away we need to add one more property.

quarkus.devservices.enabled = false

now run the application and it should resolve the issue .