In my application we are reading more than 1 million rows from DB. Storing this data in Csv file in streaming manner by using Jdbc ResultSet. OpenCsv library is used to create CSV file which is zipped using ZipOutputStream. Now I want to send this file to another microService. I am using Spring restTemplate for this. Problem is while doing post() request entire file is loaded in memory and we start getting OutOfMemory issue. I have read RestTemplate has chunked transfer mode which will take more time to send data but Memory consumption should remain constant as it does not buffer entire data in memory. I am trying to use below code but looks like chunking is not working, I am still noticing OOM and memory consumption is spiked at same rate which I can verify on Grafana. Is there any way to manage this requirement ? I am sharing code tried by me Could anyone please help me. I am using springBoot 2.7 and Java17 with G1 GC enabled.
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(""));
HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
SimpleClientHttpRequestFactory httpRequestFactory = new SimpleClientHttpRequestFactory();
httpRequestFactory.setBufferRequestBody(false);
httpRequestFactory.setChunkSize(100);
RestTemplate restTemplate = new RestTemplate();
restTemplate.setRequestFactory(httpRequestFactory);
restTemplate.postForEntity(URI.create("myUrl"), requestEntity, String.class);