Spring integration FileTailingMessageProducer: Remember current line when restarting

Question

We are using the Spring integration FileTailingMessageProducer (Apache Commons) for remotely tailing files and sending messages to rabbitmq.

Obviously when the java process that contains the file tailer is restarted, the information which lines have already been processed is lost. We would like to be able to restart the process and continue tailing at last line we had previously processed.

I guess we will have to keep this state either in a file on the host or a small database. The information stored in this file or db will probably be a simple map mapping file ids (file names will not suffice, since files may be rotated) to line numbers:

file ids -> line number

I am thinking about subclassing the ApacheCommonsFileTailingMessageProducer.

The java process will need to continually update this file or db. Is there a method for updating this file when the JVM exits?

Has anyone done this before? Are there any recommendations on how to proceed?

Answer 1

Spring Integration has an an abstraction MetadataStore - it's a simple key/value abstraction so would be perfect for this use case.

There are several implementations. The PropertiesPersistingMetadataStore persists to a properties file and, by default, only persists on an ApplicationContext close() (destroy()).

It implements Flushable so it can be flush()ed more often.

The other implementations (Redis, MongoDB, Gemfire) don't need flushing because the data is written immediately.

A subclass would work, the file tailer is a simple bean and can be declared as a <bean/> - there's no other "magic" done by the XML parser.

But, if you'd be interested in contributing it to the framework, consider adding the code to the adapter directly. Ideally, it would go in the superclass (FileTailingMessageProducerSupport) but I don't think we will have the ability to look at the file creation timestamp in the OSDelegatingFileTailingMessageProducer because we just get the line data streamed to us.

In any case, please open a JIRA Issue for this feature.