hi I have a Kafka cluster using 3 Brokers (EC2 instances) and there is a topic with 100 partitions.
I wanted to observe how much a messages per second a certain number of consumers can handle.
So I've sent messages by [20, 100, 500, 1000, 5000, 10000, 20000, 40000] per second. And for each message send rate I used [1, 10, 20, 30, ..., 100] consumers with each consumer to work on a single ec2 instance.
I used kafka-python for making Producers and Consumers. Burrow, telegraf, influxDB, Grafana was used to observe consumer lag by partition and total consumer lag.
I got the total consumer lag from the this http endpoint. burrow endpoint: /v3/kafka/live/consumer/{consumer-group}/status
The result came out like the images.
