Kafka consumer health check and recreate

Question

We use Spring Kafka Client on a project. Recently we noticed that if a Kafka Consumer dies due to OutOfMemoryError service continues operating normally and no new consumers get created. The only way to fix this is to monitor OOM in logs and restart the service manually. We are looking for a way to make consumer recreation automatic, e.g.:

1. Force Spring (somehow) to detect died consumers and in runtime create new ones.
2. In case of OOM in consumer thread kill the entire service, so that AWS auto-scaling group can create a new instance of the service.

Any suggestions or ideas are appreciated. Thank you!

Answer 1

I have found a solution related to option 2 mentioned above. Starting java version "1.8.0_92" there are a few JVM options allowing to kill entire JVM in case of OOME:

 1. -XX:+ExitOnOutOfMemoryError  
 2. -XX:+CrashOnOutOfMemoryError

were added, see the release notes.

ExitOnOutOfMemoryError When you enable this option, the JVM exits on the first occurrence of an out-of-memory error. It can be used if you prefer restarting an instance of the JVM rather than handling out of memory errors.

CrashOnOutOfMemoryError If this option is enabled, when an out-of-memory error occurs, the JVM crashes and produces text and binary crash files.