Jetty 9 leaks threads and connections and stops responding

854 Views Asked by At

We use Jetty 9 as embedded web server in a Java 8 VM. On occasion, the following happens:

  1. The thread count of the (Windows) process starts to increase at a steady rate
  2. Some time after this, Jetty stops responding to HTTP (TLS) requests.
  3. After this, threads increase still for a while, then stops increasing at around 300 (around 100 non-jetty threads in the process).
  4. At the same time the responses stops, message sequences like the following are starting to be logged repeatedly exactly every 2 minutes, and never stops:

    2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:156 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120001/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout check, elapsed: 120001 ms, remaining: -1 ms

    2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:163 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout expired

    2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.WriteFlusher:473 - ignored: WriteFlusher@2527cb{IDLE} {}

    2018-09-22 00:00:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.AbstractEndPoint:186 - Ignored idle endpoint SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120004/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1}

    ...

    2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:156 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout check, elapsed: 120002 ms, remaining: -2 ms

    2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.IdleTimeout:163 - SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120002/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1} idle timeout expired

    2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.WriteFlusher:473 - ignored: WriteFlusher@2527cb{IDLE} {}

    2018-09-22 00:02:47 DEBUG [Scheduler-7519289] org.eclipse.jetty.io.AbstractEndPoint:186 - Ignored idle endpoint SelectChannelEndPoint@be1c72{/192.168.37.61:12421<->444,Open,in,out,-,-,120003/120000,SslConnection@16ad018}{io=0/0,kio=0,kro=1}

Sequence like the above are logged for about 200 unique SelectChannelEndPoint objects (also around 200 unique client IP:port combos).

Versions: 9.3.22 and 9.3.25. These are TLS connections.

We do not change the default thread pool size.

It appears as if the scavenging of idle connections has a bug that on occasion kicks in - not from the start - there are many once-off clearing of idle connections before the problem sequence described above starts.

Please help! Is this a known bug (I could not find anything on the internet)?

What other diagnostic info could help pinpoint the issue?

0

There are 0 best solutions below