How to correctly use Keep-Alive header with libcurl C++ library

183 Views Asked by At

I have an application with makes millions of call to a backend server to fetch some data from the database. It uses HTTP API with TCP and the average time taken to execute a request is 150-200ms. The time spent on the backend server is merely 1-2ms. A lot of time is spent on making the DNS Lookup and the Three-way SSL handshake. I would like to utilize the TCP Keep-Alive functionality to not close the TCP Connections to the backend. I have noticed that merely adding the headers for the requests reduced the time taken to execute the requests. This was noticed via POSTMAN.

However, I don't see similar data when executed via our C++ application.

Below is the code that we use.

INT CCurlHTTP::HTTPSPost(const CString& endPointUrl, const CString& urlparam,const CString& cookie){
    
    CURL *curl;
    CURLcode res;
    struct curl_slist *headers=NULL;
    char errbuf[CURL_ERROR_SIZE];

    curl = curl_easy_init();

    CString KeepAlive = "Connection: keep-alive";
    CString KeepAliveSettings = "Keep-Alive: timeout=100, max=100";

    get_request req;
    req.buffer =0;
    req.len =0;
    req.buflen =0;

    if(curl) 
    {
        //add url, headers, and paramaters to the request
        curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "POST");
        curl_easy_setopt(curl, CURLOPT_URL, endPointUrl);
        curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
        headers = curl_slist_append(headers, m_httpHeadAccept);
        headers = curl_slist_append(headers, m_httpContentType);
        headers = curl_slist_append(headers, KeepAlive);
        headers = curl_slist_append(headers, KeepAliveSettings);
        //callback function used to save response
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_String);
        curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
        curl_easy_setopt(curl, CURLOPT_MAXCONNECTS, 5L);
        req.buffer = (unsigned char*) malloc(CHUNK_SIZE);
        req.buflen = CHUNK_SIZE;
        req.len = 0;
        curl_easy_setopt(curl,CURLOPT_WRITEDATA, (void *)&req);
        
        if (!cookie.IsEmpty())
        {
            headers = curl_slist_append(headers, m_DBAuthCertficate); //What is difference between this and line no 118?
            CString pCookie = "DBAuthTicket=" + cookie;
            curl_easy_setopt(curl,CURLOPT_COOKIE, pCookie);
        }
        else
        {
            headers = curl_slist_append(headers, m_OAuthToken);
        }
        curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
        curl_easy_setopt(curl, CURLOPT_POSTFIELDS, urlparam);
        curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
        errbuf[0] = 0;
        
        curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 512000);
        CFileTime start, end;
                CFileTimeSpan duration;

        start = CFileTime::GetCurrentTime();

        res = curl_easy_perform(curl);

        end = CFileTime::GetCurrentTime();
        duration = (end - start);

        CString elapsed, logMsg;
        elapsed.Format("%ld",duration.GetTimeSpan()/10000);
        logMsg.Format("[CCurlHTTP::HTTPSPost::%d]Respone time = %s ms",__LINE__, elapsed);
        UTHelper::Get().WriteLog(logMsg);

        long res_code;
        
        curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &res_code);

        if(res_code != 200 && res_code != 201)
        {
            res = CURLE_HTTP_RETURNED_ERROR;
        }

        m_response = (char*)req.buffer;
        m_errDescription.Format("%ld", res_code);
        len = req.len;
        buflen = req.buflen;

        //curl_easy_cleanup(curl);
        free(req.buffer);
    }
    return res;
}

This piece of code is used by Single threaded application as well as multi-threaded Web application (VBScript + C++ COM DLL) It is a very legacy codebase.

Questions

  1. We create a new CurlHTTP class everytime we want to execute a new request. Will curl_easy_init give us new curl handles everytime or will it give out the same ones.
  2. How do I make this work with Keep-Alive headers. The same code has not affect with these headers at all. The response times are still 150-200 ms.
  3. Please help in any way possible. I cannot make CurlHTTP singleton since this code is also used in a multi-threaded environment and it could cause bottleneck if I try synchronization.
  4. Where do I initialize curl_global_init and what params should I pass there.

LEGACY CODE - C++ 98 + VC7.

0

There are 0 best solutions below