From this question I know that I can call epoll_ctl(2)
while another thread is blocking on epoll_wait(2)
. I still have a question though.
When using epoll
with the EPOLLONESHOT
flag only one event is fired and the fd has to be rearmed using epoll_ctl(2)
. This is necessary so only one thread
will read from the fd and handle the result appropriately.
The following is a timeline that somewhat visualizes my supposed problem:
Thread1: Thread2: Kernel:
-----------------------------------------------------------------------
epoll_wait();
Receives chunk
dispatch chunk to thread 2
epoll_wait(); Handle chunk
Still handle chunk Receives chunk
Rearm fd for epoll
?
What happens on the question mark when the fd is rearmed after a chunk is received? Will epoll
fire an EPOLLIN
event, or will it block indefinitely although the socket is readable? Is my architecture at all sensible?
Your architecture is sensible, and it will work:
epoll
will mark the file descriptor as readable and fire anEPOLLIN
event.The documentation on this is scarce and subtle; the Q/A section of
man 7 epoll
briefly mentions this:The two operations that you can do on an existing file descriptor (an existing file descriptor is a file descriptor that has been added to the epoll set in the past - this includes file descriptors that are waiting to be rearmed) are delete and modify. As the manpage mentions, delete is meaningless here, and modify will re-evaluate the conditions in the file descriptor.
Nothing beats a real world experiment though. The following program tests this edge case:
It works as follows: the dispatcher thread adds
stdin
to an epoll set and then usesepoll_wait(2)
to fetch input fromstdin
whenever it becomes readable. When input arrives, the dispatcher wakes up the worker thread, who prints the input and simulates some processing time by sleeping 2 seconds. In the meantime, the dispatcher goes back to the main loop and blocks inepoll_wait(2)
again.The worker thread won't rearm
stdin
until you tell it to by sending itSIGUSR1
. So, we just write some more stuff intostdin
, and then sendSIGUSR1
to the process. The worker thread receives the signal, and only then it rearmsstdin
- which is already readable by that time, and the dispatcher was already waiting onepoll_wait(2)
.You can see from the output that the dispatcher is correctly awaken and everything works like a charm: