Send an upload in chunks to avoid OutOfMemoryError with httpclient 3.1 and URLConnection

Question

I'm trying to send a file to an IIS server using apache commons httpclient 3.1 but I get out of memory error.

    InputStream is = FileService.getInputStream(fileName, FileService.HDD);
    ByteArrayOutputStream fileToUpload = new ByteArrayOutputStream();
    ByteArrayPartSource file = null;

    try {
        byte[] buffer = new byte[1024];
        for (int len; (len = is.read(buffer)) != -1;)
            fileToUpload.write(buffer, 0, len);

        file = new ByteArrayPartSource(fileName, fileToUpload.toByteArray());

        post.setContentChunked(false); 
        Part[] part = new Part[] { new FilePart(fileName, file)}; 
        post.setRequestEntity(new MultipartRequestEntity(part, post.getParams())); 
    }
    catch (OutOfMemoryError e) {
    }
    finally {
        try {
            if (is != null) is.close();
            if (fileToUpload != null) fileToUpload.close();
        } catch (IOException e) {
        }
    }

OutOfMemoryError occurs at fileToUpload.toByteArray(). Is there any way to send file in chunks? I can't use file object because I get permission errors.

I also tried without using httpClient but I still get out of memory error

    InputStream fileToUpload = FileService.getInputStream(fileName, FileService.HDD);
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = null;

    try {
        writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

        writer.println("--" + boundary);
        writer.println("Content-Disposition: form-data; name=\"binaryFile\"; filename=\""+ fileName + "\"");
        writer.println("Content-Type: application/octet-stream");
        writer.println("Content-Transfer-Encoding: binary");
        writer.println();
        writer.flush();

        LogService.info("FileSender#setFileToUpload() Before");
        byte[] buffer = new byte[2048];
        int bytesRead;
        while ((bytesRead = fileToUpload.read(buffer)) != -1){
            output.write(buffer, 0, bytesRead);
        }

        LogService.info("FileSender#setFileToUpload() After");

        output.flush();
        writer.println();
        writer.flush();
        writer.println("--" + boundary + "--");

    } finally {
         if (writer != null) writer.close();
         if (output != null)  output.close();
         if (fileToUpload != null) fileToUpload.close();
    }

Answer 1

You need to stream the data, so it's not all in memory at once something like:

    InputStream is = FileService.getInputStream(fileName, FileService.HDD);
    try {
        InputStreamBody inputStreamBody = new InputStreamBody(is, fileName);
        MultipartEntity entity = new MultipartEntity();
        entity.addPart(fileName, inputStreamBody);
        post.setEntity(entity);

        httpClient.execute(post);
        ...
    } finally {
        is.close();
    }

Answer 2

In your non-httpclient version, try moving the output.flush() so you're sending the chunks you read in over the wire as you read them rather than buffering up the entire file in memory before flushing it (you may want to experiment with other chunk sizes):

    InputStream fileToUpload = FileService.getInputStream(fileName, FileService.HDD);
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = null;

    try {
        writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));

        writer.println("--" + boundary);
        writer.println("Content-Disposition: form-data; name=\"binaryFile\"; filename=\""+ fileName + "\"");
        writer.println("Content-Type: application/octet-stream");
        writer.println("Content-Transfer-Encoding: binary");
        writer.println();
        writer.flush();

        LogService.info("FileSender#setFileToUpload() Before");
        byte[] buffer = new byte[2048];
        int bytesRead;
        while ((bytesRead = fileToUpload.read(buffer)) != -1){
            output.write(buffer, 0, bytesRead);
            output.flush();// <-----------------------------------
        }//                                                      |
        //                                                       |
        LogService.info("FileSender#setFileToUpload() After");// |
        //                                                       |
        // -------------------------------------------------------
        writer.println();
        writer.flush();
        writer.println("--" + boundary + "--");

    } finally {
         if (writer != null) writer.close();
         if (output != null)  output.close();
         if (fileToUpload != null) fileToUpload.close();
    }