I am using Mirror Maker 2.0 to replicate the data from one cluster to another cluster. I have done the setup and it is working fine with few topics.
I am facing issues when we are trying to increase the load on the Mirror Maker. Before pushing the actual traffic, I am trying to perform load-test using kafka-producer-perf.sh script.
After increasing the load, Mirror Maker started trottling at certain transaction limit
Example:
At source cluster I am producing 50k/sec records each of size 256bytes on the topic which as 5 paritions. At destination we are seeing 23k/sec. Even if we increase the producer transation per-sec (TPS) at source side, the desintation TPS remained same i.e. 23 k/sec
At source I tried to reduce records size to 64 bytes with 50k/sec records, it went till 37k/sec at destination side. After that it is constant.
Basically the throughput at destination side is getting decrease with increase in the payload size.
To optimise this I did some tweak on multiple config parameters such as
clusters = cluster1, cluster2
max.tasks=8
cluster1.max.poll.records = 20000
cluster1.receive.buffer.bytes = 33554432
cluster1.send.buffer.bytes = 33554432
cluster1.max.partition.fetch.bytes = 33554432
cluster1.message.max.bytes = 37755000
cluster1.compression.type = gzip
cluster1.max.request.size = 26214400
cluster1.buffer.memory = 524288000
cluster1.batch.size = 524288
Even after making all these changes, not observed any significant change in the throughput at which it is being produced at destination topic.
Wanted to know if there is some limitation on this throughput or if I am missing any property to configure? or Is there anyone who has done some performance testing on the Kafka Mirror Maker and has stats available which can guide like what parameter affected the overall performance?