Curlpp, incomplete data from request

Question

I am using Curlpp to send requests to various webservices to send and receive data. So far this has worked fine since i have only used it for sending/receiving JSON data.

Now i have a situation where a webservice returns a zip file in binary form. This is where i encountered a problem where the data received is not complete.

I first had Curl set to write any data to a ostringstream by using the option WriteStream, but this proved not to be the correct approach since the data contained null characters, and thus the data stopped at the first null char.

After that, instead of using WriteStream i used WriteFunction with a callback function. The problem in this case is that this function is always called 2 or 3 times, regardless of the amount of data. This results in always having a few chunks of data that don't seem to be the first part of the file, although the data always contains PK as the first 2 characters, indicating a zip file.

I used several tools to verify that the data is entirely being sent to my application so this is not a problem of the webservice.

Here the code. Do note that the options like hostname, port, headers and postfields are set elsewhere.

string requestData;

size_t WriteStringCallback(char* ptr, size_t size, size_t nmemb)
{
    requestData += ptr;
    int totalSize= size*nmemb;
    return totalSize;
}

const string CurlRequest::Perform()
{
    curlpp::options::WriteFunction wf(WriteStringCallback);
    this->request.setOpt( wf );
    this->request.perform();

    return requestData;
}

I hope anyone can help me out with this issue because i've run dry of any leads on how to fix this, also because curlpp is poorly documented(and even worse since the curlpp website disappeared).

Answer 1

std::string is not a problem, but the concatenation is:

requestData += ptr;

C string (ptr) is terminated with zero, if the input contains any zero bytes, the input will be truncated. You should wrap it into a string which knows the length of its data:

requestData += std::string(ptr, size*nmemb);

Answer 2

The problem with the code is that the data is put into a std::string, despite having the data in binary (ZIP) format. I'd recommend to put the data into a stream (or a binary array). You can also register a callback to retrieve the response headers and act in the WriteCallback according to the "Content-type".

curlpp::options::HeaderFunction to register a callback to retrieve response-headers.