I am facing out of memory error on using the Apache camel Http component to download/stream a file from a public link to an external system (SharePoint). It seems the component internally is reading all of the files into memory at once. Can we do this without loading whole stream to memory? The route is quite simple.
/**
* Submit file to upload
*/
from("direct:fetch-content-data-and-upload")
.log("** Fetching file content from public link")
.setHeader(Exchange.HTTP_METHOD, constant("GET"))
.toD("${header.CamelHttpUri}")
.process(this.submitFileToUpload)
.end();
The processor internally uses Graph Client SDK to submit Input Stream to upload in chunks. Error is thrown before it can reach the processor.
Aug 3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onIdleExpired(HttpConnectionOverHTTP.java:160)
Aug 3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.ssl.SslConnection.onIdleExpired(SslConnection.java:391)
Aug 3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:402)
Aug 3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:171)
Aug 3 07:38:37 ip-172-31-9-176 web: at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113)
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
Aug 3 07:38:37 ip-172-31-9-176 web: ... 3 common frames omitted
Aug 3 07:38:37 ip-172-31-9-176 web: Caused by: java.lang.OutOfMemoryError: Java heap space
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
Aug 3 07:38:37 ip-172-31-9-176 web: at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:101) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:206) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:161) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:156) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.util.IOHelper.copy(IOHelper.java:152) ~[camel-util-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.doExtractResponseBodyAsStream(HttpProducer.java:547) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.extractResponseBody(HttpProducer.java:532) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.populateResponse(HttpProducer.java:332) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.component.http.HttpProducer.process(HttpProducer.java:291) ~[camel-http-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor.lambda$process$0(SendDynamicProcessor.java:197) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor$$Lambda$1001/0x000000d8016ec350.doInAsyncProducer(Unknown Source) ~[na:na]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.support.cache.DefaultProducerCache.doInAsyncProducer(DefaultProducerCache.java:327) ~[camel-support-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:182) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:477) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:181) ~[camel-base-engine-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.impl.engine.DefaultReactiveExecutor.schedule(DefaultReactiveExecutor.java:54) ~[camel-base-engine-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask.lambda$redeliver$2(RedeliveryErrorHandler.java:914) ~[camel-core-processor-3.21.0.jar!/:3.21.0]
Aug 3 07:38:37 ip-172-31-9-176 web: at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$RedeliveryTask$$Lambda$994/0x000000d8016c1360.done(Unknown Source) ~[na:na] ```