As part of a performance scenario, my route is consuming 1,000,000 files off an AwsLinux EBS volume. MaxMessagesPerPoll is set at 100, route threads set at 10. A done file is generated for each document. The route can successfully consume all documents.
The issue is, at some point the done file is locked and can’t be deleted by thread A causing thread B, at the next poll, to attempt to process the correlated file again. This fails since thread A eventually deletes the file after successfully processing it.
How can I configure the route to avoid this? Should I move away from done files to a read lock strategy?