C: Data forwarding server using epoll ET fills the send buffer

Question

I have the following situation. My server receives data from remote server (fd_server) and forwards it to the client (fd_client). I'm using edge triggered epoll so I can handle multiple clients and multiple server conncetions.

Procedure:

1. client connects to the server.
2. my server connects to the remote server and requests data.
3. remote server responds and my server forwards data to the client.

Details:

After my server connects to the remote server the fd_server is added to epoll control with EPOLLIN flag. Server waits for events.

When epoll_wait return the fd_server as readable I go in the following loop displayed bellow.

After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without loosing the data I have already read from the fd_server socket?

IS there a way of knowing before hand, how much data can I send, so I only read the right amount?

while(1){
    N = recv(fd_server,buf, sizeof buf,0);
    if (N == -1){
      /* If errno == EAGAIN, that means we have read all
         data. So go back to the main loop. */
      if (errno != EAGAIN){
          perror ("tcp_recv error");

        }
      break;
    }
    if(N == 0){
      /* End of file. The remote has closed the
         connection. */
         close(fd_server);      
         break;
    }
    pos = 0;
    while(pos < N){
        got = sctp_sendmsg(fd_client, &buf[pos], N-pos, to, tolen, 0, 0, stream, 0, 0);

        if(got<=0){
            if (errno == EAGAIN){
                //what to do?
            }else{
                perror("tcp to sctp send error");
            }
        }
        else{
        pos += got;}
    }
}

Answer 1

After some read/writes my sctp_sendmsg return EAGAIN, which means sctp send buffer is full. How should I handle this situation without losing the data I have already read from the fd_server socket?

You need to keep some sort of "context" (data structure) for each fd_client socket. For each new client socket that gets connected to your server, create an instance of a "connection state" struct and store it in a hash table. This will be something like the following:

struct ConnectionState
{
    int fd_client; // socket
    uint8_t buffer[MAX_CHUNK_SIZE];  // protocol buffer for this connection
    int buffer_length; // how many bytes received into this buffer
    int pos;           // how many bytes transmitted back out on fd_client from "buffer"
    int has_data;      // boolean to indicate protocol state (1 if there's still data in buffer to send)
};

If you can't send everything at once, toggle the fd_client socket from EPOLLIN to EPOLLOUT in your epoll mode. Change "has_data" to true in the ConnectionState structure. Then go back to waiting for socket events. When you are able to send again, you look at your ConnectionState struct for that socket to decide if you still need to keep sending or receive a new buffer.

Be careful with edge triggered sockets. When you do transition from EPOLLOUT back to EPOLLIN, you need to go ahead and recv() again just to make sure you don't lose any data. (Similarly for entering the send state, try an initial send).