I am currently trying to create a setup using Kafka MirrorMaker 2 that resembles the MirrorMaker 1 setup we are currently using. Our apps are not configured to be able to consume from multiple topic queues, so replicating to a topic with a tagged name is not currently an option. Our configuration uses an Active-Passive setup where only a singular Kafka cluster is ever being consumed from.
We have 2 sites, for the purpose of explanation I'l call them siteA and siteB. siteA is normally active (primary) while siteB is normally passive (DR). Each site has a kafka cluster as well as a kafka mirrormaker 2 node that is configured to only produce to the kafka cluster in it's own site. To prevent cyclical replication, only one direction of replication is ever set to true. I have managed to get the topics to replicate between the two clusters without renaming using the following config replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy
However, whenever I flip the replication (stopping replication from siteA to siteB and turning on replication from siteB to siteA), the Mirrormaker 2 instance replicates the messages back to siteA that siteA had previously replicated to siteB. Is there a way to make this work without duplicating the messages or is replicating to a tagged topic the only way forward?
I've tried increasing the frequency of emit.checkpoints.interval.seconds, manually turning on group.offsets.enabled for the relative clusters, and increasing the frequency of sync.group.offsets.interval.seconds. Regardless of these changes, the nessages seem to still be duplicated.
I have already been in your shoes and probably I will disappoint you but this is not possible using the policy in question. The mention policy is used only for active/passive DR scenarios.
If you still want to achieve this behavior then you have two options:
a) Using default replication policy and set prefix for each existing topic like - siteA.topic1/siteB.topic1
b) Recreate the cluster from siteA when you would like to go back to region A and want to replicate everything back from siteB -> siteA. It is because of the nature of MirrorMaker no matter the version of it.
I hope this helps !