I am using Spring Cloud Streams with a Kafka binder, and deploying the application to Kubernetes. This includes the default K8S readiness and liveness probes.
For HTTP traffic, the K8S readiness probe will cause the service to be removed from the service load balancer, so that incoming HTTP traffic is not routed to a pod which cannot process requests reliably. In extremis, the liveness probe will cause K8S to restart a container.
What I've noticed with SCS and Kafka is that whilst the application is unhealthy (e.g. database connection is failing) the polling and processing of messages continues.
I can see that if its just a readiness check failing, then removing the consumer from the group is undesirable, but a temporary pause in message polling might not be a bad thing - otherwise lots of messages end up failing during processing.
What I'd like to ask is probably two-fold:
- does anyone have any examples of how to stop this happening, i.e. so that the Kafka consumer pauses whilst the application readiness state is unhealthy?
- Would the community consider that this is something which would be useful to have out of the box with Spring Cloud Streams, i.e. that it is (somehow) integrated with the application health checks so that messages aren't consumed by an unhealthy application?