Tomcat crashes over a period of time when run 24/7

71 Views Asked by At

I am working on a billing module of an application and my client runs a 24/7 outlet. Over a period of time (1 week and 2 days), the Tomcat application crashes and a crash log is generated.

I need to understand how to address the issues, since we run the same application to even higher billing hour clients and no crash were observed.

Our client runs a 16 GB RAM PC with HDD disk of 256 GB with i3 3.3GHz processor.

Fixes tried by me

  1. Set heap size for JVM in Tomcat inside a setenv file (minimum 2 GB and maximum 8 GB)
  2. Reduced long_query_time to 3 and increased key_buffer_size to 128M
  3. Crash did not occur at peak billing hours
  4. Memory reaching limit during MySQL FTP backup, This was causing memory overload during peak time. So shifted the backup time to non-peak hour
  5. Reduced buffer time to 4GB (which is default), it was set to 7 GB which caused CPU utilization to peak which has caused Tomcat crash in previous instance
  6. Tested same application on a local instance in a comparatively low spec PC and pushed same load of orders. No crash was observed
  7. Every crash occurred at an interval of/between (1 week and 2 days). The time difference between the last two crashes was 2 weeks. This restart was skipped to observe changes in MySQL. When crash occurs on scheduled restarts improvements can be seen.

Steps taken now

  1. Scheduled Tomcat restart before business hours and monitor

Last exception caught in Tomcat crash log

*Internal exceptions (10 events):*
Event: 12273.738 Thread 0x000000002bdc5000 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000741efb0c8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12335.148 Thread 0x000000002bdc3800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException': 1> (0x0000000753a3d7f0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\interpreter\interpreterRuntime.cpp, line 368]
Event: 12341.253 Thread 0x0000000029b54800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x0000000757fa4660) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12341.253 Thread 0x0000000029b54800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000757fa5ab8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12352.115 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x0000000739ad7788) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12352.115 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000739ad9198) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12356.532 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x00000007456aac98) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12356.532 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x00000007456ac6a8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12357.730 Thread 0x000000002b9ce800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x00000007494dc708) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12357.730 Thread 0x000000002b9ce800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x00000007494de118) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
0

There are 0 best solutions below