EBADF while recv after epoll_wait

Question

i've got a following problem: i have a epoll code which receives connections:

while (1) {
    int nfds = epoll_wait(epollfd, events, 4096, -1);
    if (nfds == -1) {
        if (errno == EINTR)
            continue;
        perror("epoll_wait");
        exit(EXIT_FAILURE);
    }

    for (int i = 0; i < nfds; i++) {
        if (events[i].data.fd == server_sock) {
            client_sock = accept(server_sock,
                         (struct sockaddr *)&client_name,
                         (socklen_t *)(&client_name_len));

        if (client_sock == -1) //server overloaded
            continue;

        ev.events = EPOLLIN | EPOLLERR;

#ifdef CORE_NONBLOCKING_SOCKETS
        Arch::set_nonblocking(client_sock);
        ev.events |= EPOLLET; //input data and connection closing
#endif


#ifdef EPOLLRDHUP
        ev.events |= EPOLLRDHUP ;//
#else
        //for old libraries
        ev.events |= EPOLLHUP;//
#endif

        ev.data.fd = client_sock;

        if (epoll_ctl(epollfd, EPOLL_CTL_ADD, client_sock, &ev) == -1) {
            perror("epoll_ctl: client_socket");
            exit(EXIT_FAILURE);
        }

        accept_request(client_sock);

        } else {

#ifdef EPOLLRDHUP
            if (events[i].events & EPOLLRDHUP) {
                std::cout << "EPOLLRDHUP on " << events[i].data.fd << std::endl;
                listener->disconnectDriver(events[i].data.fd);
            }
#else
            if (events[i].events & EPOLLHUP) {
                std::cout << "EPOLLHUP on " << events[i].data.fd << std::endl;
                listener->disconnectDriver(events[i].data.fd);
            }
#endif
            if (events[i].events & EPOLLIN) {
                std::cout << "debug EPOLLIN on " << events[i].data.fd << std::endl;
                accept_request(events[i].data.fd);
            }

            if (events[i].events & EPOLLERR) {
                std::cout << "debug EPOLLERR on " << events[i].data.fd << std::endl;
                listener->disconnectDriver(events[i].data.fd);
            }


        }
    }

when i received input connection i trying to read all buff data:

void get_all_buf(int sock, std::string & inStr) {
int n = 1;
int total = 0;

char c;
char temp[1024*1024]; 

bzero(temp, sizeof(temp));

do {
#ifdef CORE_NONBLOCKING_SOCKETS
    timespec time_to_wait;
    time_to_wait.tv_nsec = 10000000;
    time_to_wait.tv_sec = 0;
    timespec tm;

    time_t begin = time(NULL);
    do {
#endif

        n = recv(sock, &temp[total], sizeof(temp), 0);

#ifdef CORE_NONBLOCKING_SOCKETS
        nanosleep(&time_to_wait, &tm); 
        time_t end = time(NULL); 
        if ((end - begin) > MAX_CLIENT_TIME) {
            inStr = std::string();
            return;
        }
    } while (n < 0 && errno == EAGAIN); //nonblocking sockets in edge-triggered mode
#endif

    if (n > 0) {
        total += n;
    } else if (n == 0) {
        //TODO: error handling
        //debug
        std::cout << "possibly no one byte was received" << std::endl;
        break;
    } else if (n < 0) {
        //TODO: error handling
        //debug
        std::cout << "error while receiving data" << std::endl;
        if (errno == EBADF) {
            std::cout << "recv returns with EBADF: " << strerror(errno) << std::endl;
        } else if (errno == EFAULT) {
            std::cout << "recv returns with EFAULT: " << strerror(errno) << std::endl;
        } else if (errno == EINTR) {
            std::cout << "recv returns with EINTR: " << strerror(errno) << std::endl;
        } else if (errno == EINVAL) {
            std::cout << "recv returns with EINVAL: " << strerror(errno) << std::endl;
        }
        //end debug
        break;
    }

} while (!strstr(temp, "</packet>")); 
inStr = temp;
};

within accept_request function. but sometime i get following in my debug output:

packet type='connect'
size of vector<Driver> in getDriversWithMoney is 1
epoll_wait detected activity of 164 counter i = 0 nfds = 1
EPOLLRDHUP on 164
disconnectDriver (fd = 164)
driver 1 disconnected
debug EPOLLIN on 164
error while receiving data
recv returns with EBADF: Invalid file descriptor

which means that someone was connected first than disconnected and when he's trying to connect again recv returns EBADF. what i did wrong? please help me.

P.S. on EPOLLRDHUP i just close file descriptor. epoll man says it's ok, because epoll removes closed fd from epoll_wait queue by itself.

Answer 1

When the remote host closes the socket, epoll() reports both a HUP and an EPOLLIN for the file descriptor.

You check for EPOLLRDHUP first, and close the socket; then you check for EPOLLIN, find that too, and try to call recv(). Since the socket has been closed, the file descriptor is no longer valid, and you get EBADF (the closed socket is removed from the epoll set, so subsequent epoll_wait() calls won't return it; but for the epoll_wait() that has already returned it's too late - the EPOLLIN is already waiting in your events[i].

You need to stop checking for events after you call disconnectDriver() (if it closed the file descriptor):

        if (events[i].events & EPOLLRDHUP) {
            std::cout << "EPOLLRDHUP on " << events[i].data.fd << std::endl;
            listener->disconnectDriver(events[i].data.fd);
            continue;
        }