In order to smooth traffic between all Kafka partitions, I tried to make a custom partitioner (extending kafka.producer.Partitioner) on my producers to replace default partitioner that only change partitions every 10 minutes.
My partitioner uses a simple round robin algorithm to switch from a partition to another every 200 messages. As producers are configured with batch.num.messages=200 and producer.type=async, traffic should jump from a partition to another every 200 messages, that is to say for every producer's request. As far as I understand, this should not modify brokers' input traffic, considering the number of requests.
But, after some tests, it seems that the amount of request was multiplied by 2 when using this custom partitioner.
Why do we have this increase? Is there a way to fix this problem, or at least a way to limit traffic increase?
Thanks for your help.
NB : I am using Kafka 0.8.2-beta, producers coded in scala 2.10