We have a Spring Boot REST application that takes requests from various sources. The web server is Netty Reactor. There are not a lot of clients, usually only two. The application uses Swagger UI for troubleshooting, so there may be connections for individual browsers, but there would be very little traffic.
We are seeing connections from one source that stay stuck in CLOSE_WAIT on the server side. When the number of these stuck connections exceeds the number of client processing threads, the application becomes unresponsive. We let Netty manage the connections and processing threads, so in our case there would be 20 threads available (the number of CPUs on the host).
The we have examined the headers coming from the problematic source. It does not send the connection close header, but it closes the connection after every request. Everything is using HTTP 1.1.
My questions are these:
If the Netty server does not receive the connection close header but the client closes the connection anyway, does this strand the connection and tie up a connection and a processing thread?
Is there a way at the application level to implement a method that would clean up these stranded connections? For example, set SO_KEEPALIVE on the underlying sockets. Note that we do not have any connection level code in the application. We only have the annotated Spring REST controller and we have selected web application type reactive in the application properties.
If we could set a keepalive on the connection and the OS closed it because the keepalive timed out, would this strand a resource at the application level?
We do not have access to the problematic application's code.
Thanks in advance for any help with this.