Very large SOAP response - Android- out of memory error

Question

I have an application where i need to download a large amount of data via a SOAP call to a webservice into the application when it is first run. The response is then sent to a function which converts the XML and stores the data in a db file.

The data is more than 16MB in size and i have a java.lang.OutOfMemoryError everytime.

Modifying the webservice to give out smaller amounts of data is not an option.

Is there a way to be able to download the large data? Something like an InputStream perhaps?

This is my code

public Protocol[] getProtocols() {

    String METHOD_NAME = "GetProtocols";
    String SOAP_ACTION = "urn:protocolpedia#GetProtocols";
    Log.d("service", "getProtocols");
    SoapObject response = invokeMethod(METHOD_NAME, SOAP_ACTION);
    return retrieveProtocolsFromSoap(response);
}

private SoapObject invokeMethod(String methodName, String soapAction) {
    Log.d(TAG, "invokeMethod");
    SoapObject request = GetSoapObject(methodName);
    SoapSerializationEnvelope envelope = getEnvelope(request);
    return makeCall(envelope, methodName, soapAction);

}

Can anyone suggest what should be done in this case?

Thanks and regards Mukul

Answer 1

Two strategies to help you solve this problem:

1. Save your SOAP XML stream directly to disk as you download it. Don't store it in memory.
2. Parse it using a SAX-style parser, where you don't load the whole DOM in memory, but rather parse it in chunks.

Depending on the kind of XML you are handling, using SAX parsers is usually harder in code; you will have to keep track of many things yourself, and you won't be able to "jump" from section to section of your DOM tree. But the memory consumption will be way lower.

Take note, however, that many "high-level" network communication libraries usually load the whole XML DOM in memory, which might be the case here. You will probably have to create and manage the HTTP connection yourself, and then manually parse the result.

Answer 2

Just an update, I found that the "call" method in AndroidHttpTransport was running out of memory at this line -

           if (debug) {
                    ByteArrayOutputStream bos = new ByteArrayOutputStream();
                    byte[] buf = new byte[256];
                    while (true) {
                        int rd = is.read(buf, 0, 256);
                        if (rd == -1)
                            break;
                        bos.write(buf, 0, rd);
                    }
                    bos.flush();
                    buf = bos.toByteArray(); //Goes out of memory here
                    responseDump = new String(buf);
                    is.close();
                    is = new ByteArrayInputStream(buf);

the call to toByteArray takes a lot of memory, so to overcome this, instead of converting the response to a byte array, i now directly write it to an XML file, and this is saved at a location of my choice. Here -

if (debug) {
    FileOutputStream bos = new FileOutputStream("/data/data/com.mypackage.myapp/response.xml");
    byte[] buf = new byte[1048576];
    int current = 0; int i=0; int newCurrent = 0;
    while ((current = inputStream.read(buf)) != -1) {
        newCurrent = newCurrent + current;
    Log.d("current", "Current = " + current + " total = "+newCurrent+" i = "+i++);
                    bos.write(buf, 0, current);
                }
                bos.flush();
}

The device no longer runs out of memory, and i have a custom parse method that takes this XML and writes it to the DB.

Answer 3

Fixed!

I downloaded/copied HttpTransportSE java class from here (after copied, some code errors can occur, but they are all quick fixable) and added to my package:

https://github.com/mosabua/ksoap2-android/blob/master/ksoap2-j2se/src/main/java/org/ksoap2/transport/HttpTransportSE.java

removed from my Connection class this row:

import org.ksoap2.transport.HttpsTransportSE;

and substituted this code in my new HttpTransportSE.java file:

if (debug) {
                    ByteArrayOutputStream bos = new ByteArrayOutputStream();
                    byte[] buf = new byte[256];
                    while (true) {
                        int rd = is.read(buf, 0, 256);
                        if (rd == -1)
                            break;
                        bos.write(buf, 0, rd);
                    }
                    bos.flush();
                    buf = bos.toByteArray(); //Goes out of memory here
                    responseDump = new String(buf);
                    is.close();
                    is = new ByteArrayInputStream(buf);
    }

with this

if (debug) {
    FileOutputStream bos = new FileOutputStream(file);

            byte[] buf = new byte[256];

            while (true) {
                int rd = is.read(buf, 0, 256);
                if (rd == -1) {
                    break;
                }
                bos.write(buf, 0, rd);
            }
            bos.flush();
}

where "file" is a simple file object like new File("/sdcard/","myFile.xml") for example