Persistent Connection on a multiprocessed and multithreaded C server

Question

Third time I try to ask this question, maybe this time I'll be able to explain my problem better.

I have a multiprocess server with each process doing the accept() (avoiding the Thundering Herd problem with file locking, don't worry). Each process initialize a thread pool (excpet the main one the manage the others). When the accept() succeeds the file descriptor is passed to the thread pool and one of these threads is awakened by a pthread_cond_signal(). After this, the process returns on the file locking waiting to pass through it so it can wait again on the accept(). Meanwhile the thread reads the file descriptor and does its job: reading the HTTP request and serving it in an infinite loop of reading-serving (in order to obtain HTTP persistent-connection). The loop will be broken only if an error occurs or if the timeout expires.

So far so good. But something occurs right after a request is served correctly: in fact, the first request is read and served entirely but when the thread restarts the cycle and enters the read cycle it remains stuck because it reads only few letters like "GE" or "GET", insted of the entire request. If I remove the infinite cycle (for the persisten-connection), each request is served by a different thread and no error occurs!!

This is the reading cycle:

for (;;) {
 ssize_t readn, writen;
 size_t nleft;
 char buff[BUFF_SIZE];
 char *ptr = buff;

 errno = 0;

 nleft = BUFF_SIZE;

 while(nleft > 0) {                             
     //I will read as much as I can using the MSG_DONTWAIT flag making the call non-blocking
     //that means that or the call will succed or it will be closed by the other side
     if ((readn = recv(connsd, ptr, nleft, MSG_DONTWAIT)) < 0) { 

         //If the non-blocking recv fails, it could set errno with one of the following errorcode
         if (errno == EAGAIN || errno == EWOULDBLOCK) {

             //This check has been implemented due to an error that happened several times
             //The buffer was empty even if a new data was sent.
             //This check gives a sort of second chance to the recv.            
             if (strlen(buff) < 10) { 
                 errno = 0;                 //It is important to reset the errno!!
                 continue;
             //If other things occured then I will terminate the string and exit the cicle  
             } else {
                 break;
             }
         // If the conenction has been closed by the client
         } else if (errno == EINTR) readn = 0;
         // If other things occured I will simply shutdown the connection
         else {
             shutdown_sequence(connsd);
             return EXIT_FAILURE;
         }
     // If I read nothing
     } else if (readn == 0) break;

     nleft -= readn;
     ptr += readn;
 }
 buff[strlen(buff)-1] = '\0';
 //request parsing...
 //request serving...
 }

Thanks everyone for the patience!

EDIT1: Just tried using Wireshark in order to see what happen. The first request is read and served correctly, but then I receive "Continuation or non-HTTP Traffic" and [TCP Window Full]... I'm trying this server on a Virtual Machine in Ubuntu 14.04

EDIT2: I tried with a simple loop:

while(nleft > 0) {
        printf("Entering cylce and reading\n");
        fflush(stdout);
        if ((readn = recv(connsd, ptr, nleft, 0)) > 0) { 
            nleft -= readn;
            ptr += readn;
            printf("reading...\n");
            fflush(stdout);
        }
        if (readn == 0) {
            printf("connection closed or nothing more to read\n");
            fflush(stdout);
            break;
        }
        if (readn == -1) {
            printf("error occurred\n");
            fflush(stdout);
            break;
        }
    }

On the terminal I only read:

Entering cylce and reading
reading...
Entering cylce and reading

While Httperf (called with --num-calls=2 --num-conns=1) uses the 50% of the CPU. When I press Ctrl+C to terminate it, the terminal prints:

connection closed or nothing more to read 
buff =
GET /1262662405106.jpg HTTP/1.1
User-Agent: httperf/0.9.0
Host: localhost

EDIT3: In response to David:

while(nleft > 0) {
        printf("I'm going on the read\n");
        fflush(stdout);
        if ((readn = recv(connsd, ptr, nleft, 0)) > 0) { 
            nleft -= readn;
            ptr += readn;
            if (*(ptr-2) == '\r' && *(ptr-1) == '\n') {
                printf("It's an HTTP request\n");
                fflush(stdout);
                break;
            } else  continue;

        } else if (errno == EINTR || readn == 0) {
            break;
        }
    }

It perfectly recognise the first HTTP request beacuse it prints the message. But for the second one it prints "I'm going on the read" once. When I press Ctrl+C the cycle continues indefinitely printing the same message.

EDIT4: So... the problem was in the HTTP response... A mistake with the header and a bad allocation of the string. Thank you, Mr. David!

Answer 1

If you're going to do non-blocking I/O and don't want to burn the CPU at 100%, you have to have some place in your code where you wait for data to arrive. You have no such code, so you burn the CPU at 100% spinning tightly while you wait for data to arrive. It sounds like you want blocking I/O. (Start by getting rid of MSG_DONTWAIT.)

Also, don't use strlen to figure out the length of something that's not a string. If you need to know how many bytes were received, keep track of it yourself.

in fact, the first request is read and served entirely but when the thread restarts the cycle and enters the read cycle it remains stuck because it reads only few letters like "GE" or "GET", insted of the entire request.

If you haven't read the entire request, call the read function again until you have an entire request. Use a blocking read.

Basically:

1. Do a blocking read into our request buffer, after any data already in the buffer.
2. Did I get an error or is the connection closed? If so, stop.
3. Do I have a complete request according to the HTTP protocol? If not, go to step 1.
4. Process the request, send the response.
5. Go to step 1.