I am working on a billing module of an application and my client runs a 24/7 outlet. Over a period of time (1 week and 2 days), the Tomcat application crashes and a crash log is generated.
I need to understand how to address the issues, since we run the same application to even higher billing hour clients and no crash were observed.
Our client runs a 16 GB RAM PC with HDD disk of 256 GB with i3 3.3GHz processor.
Fixes tried by me
- Set heap size for JVM in Tomcat inside a setenv file (minimum 2 GB and maximum 8 GB)
- Reduced long_query_time to 3 and increased key_buffer_size to 128M
- Crash did not occur at peak billing hours
- Memory reaching limit during MySQL FTP backup, This was causing memory overload during peak time. So shifted the backup time to non-peak hour
- Reduced buffer time to 4GB (which is default), it was set to 7 GB which caused CPU utilization to peak which has caused Tomcat crash in previous instance
- Tested same application on a local instance in a comparatively low spec PC and pushed same load of orders. No crash was observed
- Every crash occurred at an interval of/between (1 week and 2 days). The time difference between the last two crashes was 2 weeks. This restart was skipped to observe changes in MySQL. When crash occurs on scheduled restarts improvements can be seen.
Steps taken now
- Scheduled Tomcat restart before business hours and monitor
Last exception caught in Tomcat crash log
*Internal exceptions (10 events):*
Event: 12273.738 Thread 0x000000002bdc5000 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000741efb0c8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12335.148 Thread 0x000000002bdc3800 Exception <a 'java/lang/ArrayIndexOutOfBoundsException': 1> (0x0000000753a3d7f0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\interpreter\interpreterRuntime.cpp, line 368]
Event: 12341.253 Thread 0x0000000029b54800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x0000000757fa4660) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12341.253 Thread 0x0000000029b54800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000757fa5ab8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12352.115 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x0000000739ad7788) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12352.115 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x0000000739ad9198) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12356.532 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x00000007456aac98) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12356.532 Thread 0x000000002bdc3800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x00000007456ac6a8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12357.730 Thread 0x000000002b9ce800 Exception <a 'java/lang/ClassNotFoundException': org/apache/catalina/webresources/d/Handler> (0x00000007494dc708) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]
Event: 12357.730 Thread 0x000000002b9ce800 Exception <a 'java/lang/ClassNotFoundException': sun/net/www/protocol/d/Handler> (0x00000007494de118) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u202\12323\hotspot\src\share\vm\classfile\systemDictionary.cpp, line 210]