Topic partition on basis of partition Key

Question

I am trying to split a stream into multiple partition on the basis of a partition key but apparently its not working. The implementation is such that I have a Class lets say Metrices

public class Metrice {
    public string MetriceType { get; set; }
    public double MetriceValue { get; set; }
}

The metrices will be consumed asynchronously one at a time and might be having different MetriceType. What I am trying to achieve is create partition on the basis of MetriceType. What I have tried so far.

1. Setting Message key (partitionKey) as MetricesType;

 public Message<string, string> FormatMessage(string partitionKey, string message)
 {           
   return  new Message<string, string> { Key = partitionKey, Value = message };           
 }

The messages are always published at Partition.Value= 0

2. Partitoner class in Confluent.Kafka library, was hoping for something similar to Custom Partitoner, this link but couldn't find any .net implementation.

So my question is, is there any way to split my income messages on the basis of a property in this case MetriceType and publish them on their dedicated partitions (ordering is essential) or the only option I have is to create a Topic using AdminClient and hardcoding the partition count, or is there an alternative approach I can look into. Thanks in advance.

Answer 1

Compared to the Java SDK, you'll need to externalize your own partitioner method that returns an int.

The ProduceAsync method, for example, of the producer accepts an integer parameter for the partition you want to send the message to

Otherwise, if the topic only has one partition, then, yes, the keys will always get hashed into that