I am facing high CPU utilization issue with tomcat7. A web app developed using JDK7 is deployed on the tomcat server. This web app contacts a MySQL server that is running on a different instance, for CRUD functions.
CPU usage of tomcat becomes abnormally high once in a week. To analyse it, I had taken thread dump and found that are some blocked threads.
Most of the blocked threads have the following stack trace,
ajp-bio-8009-exec-20604 - priority:10 - threadId:0x0000000001aac000 - nativeId:0x4143 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.log4j.Category.callAppenders(Category.java:204) - waiting to lock <0x000000008bf42fc8> (a org.apache.log4j.Logger) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:838) at com.myUtil.filter.XSSFilter.doFilter(XSSFilter.java:37) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked <0x00000000db84e148> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Locked ownable synchronizers: - <0x00000000db84e190> (a java.util.concurrent.ThreadPoolExecutor$Worker)
And another stack trace related to JDBC connection,
ajp-bio-8009-exec-20585 - priority:10 - threadId:0x000000000190f000 - nativeId:0x40d6 - state:BLOCKED stackTrace: java.lang.Thread.State: BLOCKED (on object monitor) at com.mysql.jdbc.JDBC4Connection.isValid(JDBC4Connection.java:93) - waiting to lock <0x000000008bdf9890> (a com.mysql.jdbc.JDBC4Connection) at com.myUtil.dbconnection.DBAccessObject.IsConnected(DBAccessObject.java:751) at com.myUtil.dbhandler.MyDBHandler.InitOrGetConnection(MyDBHandler.java:119) at com.myUtil.dbhandler.MyDBHandler.GetNotification(MyDBHandler.java:8585) at com.myUtil.util.ProcessRequest.myPost(ProcessRequest.java:1539) at com.myUtil.util.ProcessRequest.doPost(ProcessRequest.java:104) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.myUtil.filter.XSSFilter.doFilter(XSSFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) - locked <0x00000000db7c38f8> (a org.apache.tomcat.util.net.SocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Locked ownable synchronizers: - <0x00000000db7c3940> (a java.util.concurrent.ThreadPoolExecutor$Worker)
I checked the status of MySQL server. But everything (No. of connections/threads) was normal there.
This application was working fine for the past 4 years. Only for the past 3 months, I am facing this issue. And no changes were done to the server code. I had to restart the server, every time this issue occurred.
How can I find the root cause of this problem and fix this issue permanently?