Apache Kafka - Observability - Monitor Data Consumer

Question

I have one ApacheKafka Consumer for each topic and each consumer consumes data inside a thread.

I need to implement SpringBoot HealthIndicators that will monitor if the KafkaConsumer is consuming data. Currently I have done it programmatically by catching exceptions and incrementing a count. However, what is the standard way of doing this? Are there any metrics or any solutions that is already provided out of the box that will help me understand the health of the app based on data consumption.

Answer 1

You could use kafka client interceptors for custom monitoring and logging requirements.

Incase of consumers, Consumer Interceptors can be used. It will allow you to run custom code before processing your data or when offset is committed

There is a similar interceptor for producers