I'm currently trying to refactor the processing of JMS messages to work in a distributed/cloud environment. To allow a better retry and error handling the messages are first stored to the database with a JPA entity and then read by spring integration jpa inbound adapter. This works fine as long as just a single instance of my service is running. However when multiple instances are running, the instances try to process the same message even after introducing a processing state on the persisted messages.
I have already tried to save the JMS messages in a JDBC message store, however then I would have to define a group identifier according to which an instance could select a message which is not really possible since the number of instances is dynamic and I can not assign a group id for each instance. Another possibility could be some kind of distributed lock with a LockRegistry but I couldn't make that work.
Do you have any hint/advice how I could implement the following requirements the best with spring integration:
- JMS message should be persisted
- Any instance can pick up the message and process it
- If the processing fails there will be a retry for x times (could also be retried by another instance)
- If an instance crashes or gets killed during the processing the message must not be lost
Is there maybe some spring-cloud component which could be helpful? I'm happy about every hint in which direction I should go.