How to assign specific ID to Kafka Topic Partition

Question

I am new to Apache Kafka. I want to assign a our user id as id to the topic partition. Is there a way to assign our own user-id to partition. I did research for couple hours on this, but didn't find any article related to assigning an ID to partition.

While publishing a message to Topic I want to use the user-id as key. So that all messages goes into the same partition. And I want to make sure that one partition should contain only one user related messages.

Can I use this user-id in consumers while consuming messages from partition?

Is there a way to achieve this functionality?

Answer 1

Id generation logic is up to your own application/services.

The default behavior of Kafka DefaultPartitioner is to place all same IDs into the same partition, ordered by arrival time (not necessarily producer time without extra producer configuration); there is nothing you need to do for this

Can I use this user-id in consumers

It's just the record key, so yes...

If you mean you want to assign consumer certain partitions based on the ID, then you'd need to reverse the DefaultPartitioner hash function

Answer 2

To implement the interface, you can customize the partition logic:

public interface Partitioner

detail: https://kafka.apache.org/26/javadoc/org/apache/kafka/clients/producer/Partitioner.html#partition-java.lang.String-java.lang.Object-byte:A-java.lang.Object-byte:A-org.apache.kafka.common.Cluster-