Camel SFTP how to prevent processing the same file incase of multiple nodes?

Question

This is my route

    from("sftp://userName:password@ip:22/<my-folder>?move=.done")
        .routeId("my-route-1")
        .<processing-logic>

How to avoid processing the same files incase of multiple instances?

Answer 1

Camel recently introduced some interesting clustering capabilities - see here.

In your particular case, you could model a route which is taking the leadership when starting the directory polling, preventing thereby other nodes from picking the (same or other) files.

Set it up is very easy, and all you need is to prefix singleton endpoints according to the master component syntax: master:namespace:delegateUri

This would result in something like this:

from("master:mycluster:sftp://...")
        .routeId("clustered-route")
        .log("Clustered sftp polling !");

Answer 2

You can use File Based Idempotent Repository.

Using a file based idempotent repository

Here my example;

FileIdempotentRepository sftpIdempotentRepository = new FileIdempotentRepository();

sftpIdempotentRepository.setFileStore(new File("where to store it"));

exchange.getContext().getRegistry().bind("sftpIdempotentRepository", sftpIdempotentRepository);

After this configuration you have to add these parameters to your SFTP router.

 from("sftp://userName:password@ip:22/<my-folder>?move=.done&idempotent=true&idempotentKey=${file:name}-${file:size}&idempotentRepository=#sftpIdempotentRepository")
        .routeId("my-route-1")
        .<processing-logic>

It will store files read from SFTP in an idempotent file (filename + file size) and will prevent the same file from being processed again.

You can change the idempotentKey value according to your needs.