Transferring big files in spring integration

Question

The spring integration flow I wrote has to get files (some of them are as big as 4G) from a rest service and transfer them to a remote shared drive. For downloading them from the rest service I configured this simple component:

@Bean
public HttpRequestExecutingMessagehandler httpDownloader (RestTemplate template){
  Expression expr = (new SpelExpressionParser()).parseExpression("payload.url");
  HttpRequestExecutingMessagehandler  handler = new HttpRequestExecutingMessagehandler (expr, template);
  handler.setExpectedResponseType(byte[].class);
  handler.setHttpMethod(GET);
  return handler;
}

Unfortunately this won't scale meaning for larger files it will eventually throw java.lang.OutOfMemoryError: Java heap space, even if i add more memory with -Xmx or -XXMaxPermSize

So my question is, what to do in order to avoid these problems no matter how big the files will be?

Answer 1

I think I have answered you in some other similar your question that Spring RestTemplate is not designed for streaming response body. It is explained in this SO thread: Getting InputStream with RestTemplate.

One of the solution which may work for your is to write a custom HttpMessageConverter which would return a File object containing data from HTTP response. This article explains how to do that with the ResponseExtractor, but something like FileHttpMessageConverter is not so hard to implement based on experience from that article. See StreamUtils.copy(InputStream in, OutputStream out)

Then you inject this FileHttpMessageConverter into your HttpRequestExecutingMessagehandler - setMessageConverters(List<HttpMessageConverter<?>> messageConverters).

Your service for remote shared drive should already deal with this local temporary file to get that large content without consuming memory.

See also this one about possible approach via WebFlux: https://www.amitph.com/spring-webclient-large-file-download/

Answer 2

Created this starting from ByteArrayHttpMessageConverter class and injected it into the custom RestTemplate I use. But this solution is based on using a File message, which is not quite the streaming I was hoping for.

public class FileCustomConverter extends AbstractHttpMessageConverter<File> {

    public FileCustomConverter() {
        super(new MediaType[]{MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL});
    }

    public boolean supports(Class<?> clazz) {
        return File.class == clazz;
    }

    public File readInternal(Class<? extends File> clazz, HttpInputMessage inputMessage) throws IOException {
        File outputFile = File.createTempFile(UUID.randomUUID().toString(), ".tmp");
        OutputStream outputStream = new FileOutputStream(outputFile);
        StreamUtils.copy(inputMessage.getBody(), outputStream);
        outputStream.close();
        return outputFile;
    }

    protected Long getContentLength(File bytes, @Nullable MediaType contentType) {
        return bytes.length();
    }

    protected void writeInternal(File file, HttpOutputMessage outputMessage) throws IOException {
        InputStream inputStream = new FileInputStream(file);
        StreamUtils.copy(inputStream, outputMessage.getBody());
        inputStream.close();
    }           
}