We are using Kafka to process events. We have 2 use cases which resulting more events.
- Sometimes we are getting lot of bot traffic.
- Some of our customers are generating more traffic frequently.
This is causing delay in processing other customers events.
So I want to add some rate limiting based on the customer. As per my research there is no way to add rate limiting while publishing message to Topic.
Our application is developed on Micronaut framework.
Is there a way to add limits at consumer level?
Is there any java related frameworks on top of Kafka to implement rate limiting ?
You are asking about rate limiting, but what you are trying to achieve is prioritizing processing of data, don't you? This is not an easy thing, because it really depends on your flow, but I'd suggest to consider couple things:
parallelism
per source (or route), which provides QoS for main data flow and in case that lower priority has spikes in incoming data it would cause back pressure for its processing, but without affect on higher priority flow.