Using Spring Integration to split a large XML file into individual smaller messages and process each individually

Question

I am using Spring Integration and have a large XML file containing a collection of child items, I want to split the file into a set of messages, the payload of each message will be one of the child XML fragments.

Using splitter is the obvious but this requires returning a collection of messages and this will exhaust the memory; I need to split the file into individual messages but process them one at a time (or more likely with a multi threaded task-executor).

Is there a standard way to do this without writing a custom component that writes the sub-messages to a channel programatically.

Answer 1

i have been looking for a similar solution and I have not found either any standard way of doing this. Here is a rather dirty fix, if anyone needs this behavior implemented:

1. Split the files manually using a Service Activator or a Splitter with a custom bean.

<int:splitter input-channel="rawChannel" output-channel="splitChannel" id="splitter" >
  <bean class="com.a.b.c.MYSplitter" />
</int:splitter>

2. Your custom bean should implement ApplicationContextAware so the application context can be injected by Spring.
3. Manually retrieve the output channel and send each sub-message

MessageChannel xsltChannel = (MessageChannel) applicationContext.getBean("splitChannel"); 
Message<String> message = new GenericMessage<String>(payload));

splitChannel.send(message);

Answer 2

For people coming across this very old question. Splitters can now handle results of type Iterable, Iterator, Stream, and Flux (project reactor). If any of these types are returned, messages are emitted one-at-a-time.

Iterator/Iterable since 4.0.4; Stream/Flux since 5.0.0.

There is also now a FileSplitter which emits file contents a line-at-a-time via an Interator - since 4.1.2.