Wait for all Files after a poll to be processed before continuing processing

Question

I would like to proccess all the files located in a specific folder using <int-file:inbound-channel-adapter> and only after having tranformed/moved all files that have beed retrieved in a single poll make a call to the database to update a table.

Can this be done using and Aggregator. Is it possible to know filenames count in advance. In order to configure a release strategy.

Is there another option ?

Answer 1

Yes. This really looks like a task for an aggregator if you need to know all those files information for updating database.

It is indeed not possible to know the number of file to make the standard RealeaseStrategy happy for some specific number.

I would suggest to have some static correlation key and expireGroupsUponCompletion = true to be able to form a new group on subsequent directory polls when new files are appeared. You need to be sure that you don't do thread shifting with your process and have poller configured for a fixed-delay. This way a new poll request won't happen until the current is done.

To make everything work in the end I suggest to look into ReceiveMessageAdvice and implement its afterReceive() the way that you check for a Message<?> result if it is null which means no more files to emit as messages from the <int-file:inbound-channel-adapter>. This null could be a signal for your <aggregator> to release the group. So, you just send some marker message to an aggregator input channel from this afterReceive() and some custom RealeaseStrategy on that aggregator would check if this marker message is present in the group to return true for its contract.

You may then somehow filter out that marker message downstream from an aggregator result before updating the database.

See more info about ReceiveMessageAdvice in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/polling-consumer.html#smart-polling