will tcp socket recv return "resource temporarily unavailable" when recv buf already have data

Question

I implement TCP socket communication using epoll to monitor all the client events, only one thread process all the client in a for loop. every socket is non-blocking.

now I just suffered a problem, when client send data more than MTU, means several fragment packet, server always can't read all the data completely. like below, I read the head first, get the pdu len from the head, then read the pdu part.

the problem is though I read the head successfully, closely following with the pdu recv(), but it always return EAGAIN several times. so my retry will break. Because server needs to process thousands of client event, so I think it is a great performance consumptionn to let the retry always continue which is not tolerable.

I use tcpdump capturing the packets from client, every packet is fragment to max 1448 bytes data, but head is only 5 bytes, why I can read the head successfully, but the following data recv() operation will return EAGAIN? is it possible recv return EAGAIN when data already arrive the recv buffer? In my opinion, I can read the first 5 bytes, so must have more data to read in the recv buffer.

maybe related with the assemble process in tcp/ip stack. code is like below, every pdu recv, need 10 or more retry, maybe success.

...
#define HDR_LEN    5
n = epoll(epfd, events, 1000, -1)
for(i =0; i < n; i++)
{
    uint8 pHdr[HDR_LEN] = {0};
    uint16 pdulen = 0, offset =0;
    infd = events[i].fd;
    nRead = recv(infd, pHdr, HDR_LEN);   // read the data head first
    pdulen = ntohs(*(uint16 *)(pHdr+2));   // get the pdu len from the head
    uint8 *pbuf = malloc(pdulen+HDR_LEN);

    memcpy(pbuf, pHdr, HDR_LEN);            // move the head to buf
    while(offset != pdulen)                 // then read the pdu data
    {
        nRead = recv(infd, pbuf+HDR_LEN+offset, pdulen-offset);
        if (nRead <=0)
        {
            if (nRead == -1 && errno == EAGAIN) // resource temporarily unavailable 
            {
                if (retry < 5)
                {
                    usleep(500);
                    retry++;
                    continue;
                }
                else
                     break;  // already try 5 times, should always continue?
            }
            else
                break;
        }
        else
        {
             offset += nRead;
             retry = 0;
        }
    }
    if (offset == pdulen)
        process(pbuf, pdulen+HDR_LEN); // process the complete data
    ...
}
...

Answer 1

epoll will tell you if you can recv without blocking, once. If recv consumes all data from the socket, then next recv will block until there's more data, or return EAGAIN (if non-blocking socket).

A common pattern is to:

1. Use select/poll/epoll to detect when a socket can be read.
2. Call recv once on a ready socket and append received data to a buffer.
3. Check if the buffer contains enough data for processing. If yes, then process. Otherwise let select/poll/epoll tell when you can read more.