Scaling Kafka Consumer to 200 Threads without Increasing Partitions (100k Events/15 min)

Question

Context: I'm building a high-throughput event processing application that needs to handle 100k events within 15 minutes. Currently, this is achieved using 200 parallel threads in IBM MQ within an IBM ACE application on OpenShift Red Hat CloudPak (app takes 9ms to process single event).

Challenge: We're migrating to Kafka and want to maintain the same throughput with 200 threads. However, when using Kafka node in IBM ACE message flow as far as i know , if I'm using same group id then I can connect only one thread to one single partition from same consumer group. Increasing partitions to 200 is undesirable due to (administrative overhead, cost).

How can we achieve 100k events/15 min with 200 parallel threads in Kafka without increasing the number of partitions?

Additional Information:

IBM ACE version (12.0.10.0)

IBM MQ 9.3.x

Duplicate message consumption should never occur

Answer 1

You cannot have more consumers than partitions in a group, but that doesn't mean those consumers are limited to single threads. See https://github.com/confluentinc/parallel-consumer

Answer 2

You can make 20 Topics with each 10 partitions or 10 Topics with each 20 partition.Which will be less overhead to a single topic. Each thread will assigned to a single partition and will perform much better