WebSphere 7.0.0.19 JVM Crash Load Testing RHEL Linux

Question

WebSphere JVM crashes while performing load testing and generates .dmp, .txt and .trc files.

Environment details:

HW: Architecture x86_64

CPU op-mode(s) 64-bit

CPU(s) 5

RAM 16 GB

OS: Operating System Red Hat Enterprise Linux Server release 5.6 (Tikanga)

Kernel 2.6.18-238.el5

File Limit 8000

Max User Processes 1024

WebSphere WebSphere Version 7.0.0.19

Initial Heap Size 2048 MB

Maximum Heap Size 4096 MB

Generic JVM arguments -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Djavax.management.builder.initial= -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099

Minimum DB Connections 1

Maximum DB Connection 180

DB Connection Time Out 180

Minimum Web Container Thread Size 50

Minimum Web Container Thread Size 75

JVM Details

NULL           ------------------------------------------------------------------------
0SECTION       ENVINFO subcomponent dump routine
NULL           =================================
1CIJAVAVERSION J2RE 6.0 IBM J9 2.4 Linux amd64-64 build jvmxa6460-20080816_22093
1CIVMVERSION   VM build 20080816_022093_LHdSMr
1CIJITVERSION  JIT enabled, AOT enabled - r9_20080721_1330ifx2
1CIGCVERSION   GC - 20080724_AA_CMPRSS
1CIRUNNINGAS   Running as a standalone JVM

Below is the snapshot of the GC

<af type="nursery" id="21" timestamp="Dec 23 19:47:03 2013" intervalms="8681.336">
  <minimum requested_bytes="1160" />
  <time exclusiveaccessms="0.025" meanexclusiveaccessms="0.025" threads="0" lastthreadtid="0x00000000029DF700" />
  <refs soft="37815" weak="31295" phantom="1064" dynamicSoftReferenceThreshold="28" maxSoftReferenceThreshold="32" />
  <nursery freebytes="0" totalbytes="502379520" percent="0" />
  <tenured freebytes="1445056240" totalbytes="1610612736" percent="89" >
    <soa freebytes="1445056240" totalbytes="1610612736" percent="89" />
    <loa freebytes="0" totalbytes="0" percent="0" />
  </tenured>
  <gc type="scavenger" id="21" totalid="39" intervalms="8695.837">
    <flipped objectcount="772517" bytes="49361504" />
    <tenured objectcount="0" bytes="0" />
    <finalization objectsqueued="397" />
    <scavenger tiltratio="82" />
   <nursery freebytes="466161688" totalbytes="516178944" percent="90" tenureage="10" />
   <tenured freebytes="1445056240" totalbytes="1610612736" percent="89" >
     <soa freebytes="1445056240" totalbytes="1610612736" percent="89" />
     <loa freebytes="0" totalbytes="0" percent="0" />
   </tenured>
   <time totalms="37.615" />
 </gc>
 <nursery freebytes="466160528" totalbytes="516178944" percent="90" />
 <tenured freebytes="1445056240" totalbytes="1610612736" percent="89" >
   <soa freebytes="1445056240" totalbytes="1610612736" percent="89" />
   <loa freebytes="0" totalbytes="0" percent="0" />
 </tenured>
 <refs soft="37598" weak="14193" phantom="1064" dynamicSoftReferenceThreshold="28" maxSoftReferenceThreshold="32" />
 <time totalms="57.603" />
  </af>

  <con event="kickoff" timestamp="Dec 23 19:47:03 2013">
    <kickoff reason="Unloading of classes requested" />
    <stats tenurefreebytes="1445056240" nurseryfreebytes="466094992" tracetarget="65473953" kickoff="9082810"  />
  </con>

Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F5FD09EC780 Handler2=00007F5FD06AFF20 InaccessibleAddress=0000000000000020
RDI=00007F5FCC077980 RSI=000000000157DDA0 RAX=00007F5FBF6F1300 RBX=0000000000000000
RCX=0000000000001555 RDX=00007F5FBF6F1300 R8=00007F5F1E227E80 R9=0000000000000000
R10=0000000000000384 R11=00000001E1979348 R12=00007F5FCC077980 R13=0000000000000000
R14=00007F5F1E227E80 R15=000000000157DDA0
RIP=00007F5FC5F738E9 GS=0006000000000000 FS=0000000600000000 RSP=00007F5FBC2F67E0
EFlags=0000000000210206 CS=0000000000000033 RBP=00007F5FBF6F1300 ERR=0000000000000006
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000020
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 0000000040800000 (f: 1082130432.000000, d: 5.346435e-315)
xmm2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm3 00ff000000000000 (f: 0.000000, d: 7.063274e-304)
xmm4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm5 00007f5fb42807a0 (f: 3022522368.000000, d: 6.919341e-310)
xmm6 00007f5fd0b4cca0 (f: 3501509888.000000, d: 6.919365e-310)
xmm7 000000000002010c (f: 131340.000000, d: 6.489058e-319)
xmm8 4350e0198eaee000 (f: 2393825280.000000, d: 1.900000e+16)
xmm9 4024000000000000 (f: 0.000000, d: 1.000000e+01)
xmm10 4297fffffffffd2c (f: 4294966528.000000, d: 6.597070e+12)
xmm11 3e90affee0cce3d1 (f: 3771524096.000000, d: 2.486629e-07)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home0/was7/IBM/WebSphere/AppServer/java/jre/lib/amd64/compressedrefs/libj9jit24.so
Module_base_address=00007F5FC5BC9000
Target=2_40_20080816_022093_LHdSMr (Linux 2.6.32-71.el6.x86_64)
CPU=amd64 (5 logical CPUs) (0x3ed11e000 RAM)
----------- Stack Backtrace -----------

On searching the internet found this link

http://www-01.ibm.com/support/docview.wss?uid=swg21499624

which says the issue could be because of page cache, but WAS version is different as mentioned in the link above. Require some help on this.

Thanks.

Answer 1

I Disabled compressed references in JVM, and and solved the problem

http://www-01.ibm.com/support/docview.wss?uid=swg21499624