I am unable to delete the directory due to file is being used by object

Question

I am using Java 11 and Spring Boot v2.3.4

Requirements:

1. Download the files from S3 bucket into local storage (temp directory)
2. Compress to Zip file
3. Upload the Zip files to OTC cloud
4. Delete the directories created in step 1. (Problem is here)

--> Main Problem

I am facing an issue where I am unable to delete the last directory. The root cause of the problem lies in the fact that the zip file, located inside the last directory, remains locked by the Java Virtual Machine (JVM) process even after the successful completion of the execution. (I handled the exception)

When I omit the invocation of the uploadObjectFromLocal() method, I can successfully delete all directories. This leads me to conclude that the locking issue is associated with the uploadObjectFromLocal() method.

Additionally, attempting to manually delete the directory while the project is still running proves unsuccessful due to the ongoing locking of the zip file by the JVM process. As a workaround, I must stop the project to release the lock on the zip file, allowing for manual deletion of the directory.

The primary objective is to address the file locking concern within the uploadObjectFromLocal() method, ensuring that directories can be deleted without the need to stop the project.

Folder structure of files downloaded from S3 bucket.(inside temp dir)

-> Folder1
    -> subfolder
        -> one.csv
        -> two.csv
    -> subfolder.zip
-> Folder2
    -> subfolder
        -> one.csv
        -> two.csv
    -> subfolder.zip

I am uploading zip files one by one. Once all the zip files are uploaded on OTC. I am deleting all the directories one by one.

compressToZip.zipFiles("subfolder path"); // inside Folder1 and so on..
log.info("successfully zipped");

otcOperations.uploadObjectFromLocal("Key", "subfolder.zip"); // zip of subfolder and so on..
log.info("successfully uploaded zip files");

// Adding main folder path in arraylist to delete the directory
// Second time -> Folder2 path
deleteDirectoryList.add("Folder1 path");  

Code to upload zip file on OTC (otcOperations.uploadObjectFromLocal() method)

// Necessary imports..
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import lombok.extern.slf4j.Slf4j;
import okhttp3.*;
import org.apache.logging.log4j.util.Strings;
import org.apache.tika.Tika;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

// Method
public void uploadObjectFromLocal(String key,String filePath) throws Exception {
    File file = new File(filePath);
    Tika tika = new Tika();
    String mimeType = tika.detect(file);
    Map<String, String[]> headers = new HashMap<String, String[]>();

    Map<String, String> queries = new HashMap<String, String>();
    queries.put("acl", null);

    String authSignature = headerSignature("POST", headers, queries, bucketName, key, filePath,null);
    //for policy
    String policy = "{ \"expiration\": \"2024-12-31T12:00:00.000Z\",\n" +
            "\"conditions\": [\n" +
            "{\"bucket\": \""+bucketName+"\"},\n" +
            "{\"key\": \""+key+"\"},\n" +
            "{\"content-type\": \""+mimeType+"\"},\n" +
            "{\"success_action_status\":\"201\"}\n" +
            "]\n" +
            "}";
    String base64EncodedPolicy =Base64.getEncoder().encodeToString(policy.getBytes("utf-8"));

    String policySignature = calculateHMACSHA1(otcSK, base64EncodedPolicy);
    OkHttpClient client = new OkHttpClient().newBuilder()
            .build();
    RequestBody body = new MultipartBody.Builder().setType(MultipartBody.FORM)
            .addFormDataPart("key",key)
            .addFormDataPart("AccessKeyId",otcAK)
            .addFormDataPart("Signature",policySignature)
            .addFormDataPart("content-type", mimeType)
            .addFormDataPart("policy",base64EncodedPolicy)
            .addFormDataPart("success_action_status","201")
            .addFormDataPart("bucket",bucketName)
            .addFormDataPart("file",key,
                    RequestBody.create(MediaType.parse("application/octet-stream"),
                            new File(filePath)))
            .build();

    Request request = new Request.Builder()
            .url("URL")
            .method("POST", body)
            .addHeader("x-obs-acl", "private")
            .addHeader("x-obs-storage-class", "STANDARD")
            .addHeader("Authorization", authSignature)
            .build();

    try (Response response = client.newCall(request).execute()){
        if(response.code() != 201){
            throw new Exception("Error creating object. Error: "+response.body().string());
        }
    }
}

Exception

2024-02-02 18:43:43.626  INFO 11636 --- [   scheduling-1] awsotcservice.AwsOtcServiceApplication   : successfully zipped
2024-02-02 18:43:56.861  INFO 11636 --- [   scheduling-1] awsotcservice.AwsOtcServiceApplication   : successfully uploaded zip files
2024-02-02 18:44:03.229 ERROR 11636 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

java.io.IOException: Unable to delete file: C:\Users\Rj\AppData\Local\Temp\FOLDER1\subfolder.zip
    at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2381) ~[commons-io-2.5.jar:2.5]
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1679) ~[commons-io-2.5.jar:2.5]
    at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1575) ~[commons-io-2.5.jar:2.5]
    at awsotcservice.AwsOtcServiceApplication.scheduledOperation(AwsOtcServiceApplication.java:111) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE]
    at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

When I try to manually delete the file (FYI: spring boot project is still running)

uploadObjectFromLocal() method isn't releasing the last zip file, hence I am unable to delete the last directory.

Any help is highly appreciated. Thank you.

I tried Thread.sleep(30000) but the last zip file isn't released by some object. (don't know where it stuck)

Answer 1

how do you compress your files?

Seems like there is an OutputStream or any kind of writer which is not closed.

On your zipping method

compressToZip.zipFiles("subfolder path");

take a look if there are any writers or streams which are created but never closed.