I have a 3 partition kafka topic to handle events and push the messages to exexternal service.
This is done by spring cloud kafka binding with functional approach in a kubernetes cluster.
In happy case working as expected.
When external service down ,we added a 3 times retry mechanisms introducing a retry topic .
Problem aarise if till the retry completed and still external service available.
What should be the approach need to follow.
Solution 1.Push it to database from dlq and procees through from a cron job manner.
Solution 2 .If retry exusted and still external service down ,then pause the listener without acknowledge ing the message and resume once external service up (cercit breaker pattern),or application restart.
Sol 2 is work fine in case of 1 instance set up as it will resume the listener.
Qs : How to resume other listener running in other instance .(any listner rebalncing )
Is there any option read that particular offset and delete it on demand basis in spring cloud stream.
Or any better solution available ??