Tomcat 7 Freeze after few days

Question

Tomcat server freezes and stops responding completely. Restart seem to be the only way to bring it back online. Thread dump showed that server maxed out on AJP threads. I have default value (200) set for maxThread, and I saw exactly 200 AJP threads in "RUNNABLE" and "locked on monitor" state when my server hung. This issue typically happens once every 3 to 4 days. The server is not loaded heavily, it's serving one request every 5 min or so. Is this some kind of ajp-thread leak issue?

Server is unable to create new AJP threads once it reaches max-limit on number AJP threads, I can understand that's the expected behavior. But what I am struggling to understand is

• Why the server keep creating new AJP thread for serving new requests when "RUNNABLE" ajp threads are available in the pool?

Did anyone experience similar issue? Is this a know issue/bug in Tomcat? Appreciate your help!

Thread Dump snippet of AJP-Thread: (I can post complete thread dump if anyone like to look at it... please let me know)

    "ajp-bio-6109-exec-307" daemon prio=10 tid=0x00007f846d3bc800 nid=0x51c9 runnable [0x00007f842e4e3000]
   java.lang.Thread.State: RUNNABLE   
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:316)
    at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:371)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:128)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
    - locked <0x00000007b4f6f9a8> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Thanks

Answer 1

I cant seem to post comments yet...

in your server.xml file do you have connectionTimeout values set?

<!-- A "Connector" represents an endpoint by which requests are received
     and responses are returned. Documentation at :
     Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
     Java AJP  Connector: /docs/config/ajp.html
     APR (HTTP/AJP) Connector: /docs/apr.html
     Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1" 
           maxThreads="660" connectionTimeout="20000" 
           redirectPort="8443" URIEncoding="WINDOWS-1256" />

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="20000" URIEncoding="WINDOWS-1256"  />

hope this helps regards,

Answer 2

If you are using a web server with a keepAlive option enabled, your AJP threads may be getting stuck in keepAlive state.

To verify the state of the AJP threads, you can log into the tomcat manager app (e.g. http://localhost:8080/manager) and go to the server status tab. You should see the state of all your AJP threads there. You will see a "K" (keepalive state) in the left most column for all threads in the AJP thread pool.

If the AJP thread pool is maxed out with these kinds of threads, and the timestamps are really old on these threads, then you probably have a web server which is sending Keep-Alive headers. On average a browser sends 6-8 parallel requests per site, so each browser session can cause up to 8 AJP threads to get stuck in keepAlive state. After several users visit your site you will see the AJP thread pool get maxed out with connections in keepAlive state.

You can set the keepAliveTimeout in your server.xml to be the same as the web server keepAliveTimeout to fix the issue. The connectionTimeout parameter can also be used as an extra layer of defense against the AJP thread pool getting maxed out.

Note: According to the tomcat 7 documentation, if you set the connectionTimeout but not the keepAliveTimeout in your server.xml, then the keepAliveTimeout will be set to whatever the connectionTimeout is.