SpringBoot 3 Upload large file efficiently and performantly

68 Views Asked by At

I am using spring boot 3, java 21 for a rest api.

One of the API will handle file upload. Files can be pretty big, 2~3GB. Reading the spring documentation, I read that the MultipartFile request will always either save the data to memory or save it as a temporary file. Either way does not work. If possible I want a way to directly stream the MultipartFile data as it is being uploaded. I have read some SO posts, some of them is using Apache commons fileupload stream. Those posts are pretty old.

I will share two rest controller methods.

@PostMapping("/{fileId}/content")
@PreAuthorize("hasAuthority('SCOPE_write:files')")
public ResponseEntity<String> postFileFileId(@PathVariable Long fileId,
        @RequestParam("file") MultipartFile multipartFile) {
    Optional<FileResourceEntity> fileResourceEntityOptional = fileRepository.findByFileIdAndIsTrashFalse(fileId);
    if (!fileResourceEntityOptional.isPresent()) {
        return ResponseEntity.notFound().build();
    }
    try {
        FileResourceEntity fileResourceEntity = fileResourceEntityOptional.get();
        createFoldersAndSaveUploadFile(fileResourceEntity, multipartFile);
        fileRepository.save(fileResourceEntity);
        return ResponseEntity.noContent().build();
    } catch (Exception ex) {
        return ResponseEntity.badRequest().body(UPLOAD_FAILED_MESSAGE);
    }
}

@PostMapping(consumes = { "multipart/form-data" })
@PreAuthorize("hasAnyAuthority('SCOPE_write:files')")
public ResponseEntity<String> postFile(
        @RequestPart("filedetail") FileResourceDetail fileResource,
        @RequestPart("content") MultipartFile multipartFile,
        UriComponentsBuilder ubc) {

    FileResourceEntity savedFile = new FileResourceEntity();
    try {
        FileResourceEntity fileData = fileResourceDTOService.convertToEntity(fileResource);
        savedFile = fileRepository.save(fileData);

        if (!multipartFile.isEmpty()) {
            createFoldersAndSaveUploadFile(savedFile, multipartFile);
        }

        URI locationOfNewFile = ubc
                .path("/api/v4/files/{fileId}")
                .buildAndExpand(savedFile.getFileId())
                .toUri();
        return ResponseEntity.created(locationOfNewFile).build();
    } catch (Exception ex) {
        fileRepository.deleteById(savedFile.getFileId());
        return ResponseEntity.badRequest()
                .body(UPLOAD_FAILED_MESSAGE);
    }
}


private void createFoldersAndSaveUploadFile(FileResourceEntity fileResourceEntity, MultipartFile multipartFile)
        throws IOException {
    String createDate = fileResourceEntity.getCreatedAt().format(DATE_FORMATTER);
    createFolders(Path.of(rootDataFolder, createDate));
    Files.copy(multipartFile.getInputStream(), getFilePath(fileResourceEntity));
}

We are running the application as pod in Kubernetes. Files are saved to a PV, which is actually mounted FSX. Right now, after client finishes upload, server takes a long time to save the multipart file to actual file server, then returns response.

Is there any way to directly save/stream uploads to file using Spring Boot?

0

There are 0 best solutions below