We use Jetty 9 as embedded web server in a Java 8 VM. On occasion, the following happens:
- The thread count of the (Windows) process starts to increase at a steady rate
- Some time after this, Jetty stops responding to HTTP (TLS) requests.
- After this, threads increase still for a while, then stops increasing at around 300 (around 100 non-jetty threads in the process).
At the same time the responses stops, message sequences like the following are starting to be logged repeatedly exactly every 2 minutes, and never stops:
2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:156 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120001/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout check, elapsed: 120001 ms, remaining: -1 ms
2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:163 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout expired
2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.WriteFlusher:473 - ignored: WriteFlusher@2527cb{IDLE} {}
2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.AbstractEndPoint:186 - Ignored idle endpoint SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120004/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1}
...
2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:156 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout check, elapsed: 120002 ms, remaining: -2 ms
2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:163 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout expired
2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.WriteFlusher:473 - ignored: WriteFlusher@2527cb{IDLE} {}
2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.AbstractEndPoint:186 - Ignored idle endpoint SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120003/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1}
Sequence like the above are logged for about 200 unique SelectChannelEndPoint objects (also around 200 unique client IP:port combos).
Versions: 9.3.22 and 9.3.25. These are TLS connections.
We do not change the default thread pool size.
It appears as if the scavenging of idle connections has a bug that on occasion kicks in - not from the start - there are many once-off clearing of idle connections before the problem sequence described above starts.
Please help! Is this a known bug (I could not find anything on the internet)?
What other diagnostic info could help pinpoint the issue?