High CPU usage by unknown process in multithreading

Question

We have server with 35gb memory and Intel® Xeon(R) E5-1620 0 @ 3.60GHz × 8 CPU. I am running a multithreaded program designed with akka actors and written in scala. In the program, there are 4 actors with tasks:

1) Lazy reading from file with Scala's BufferedSource and iterator,

2) Tokenizing sentences,

3) Calculating single and bigram words frequency for a given window size, and putting them into a map (one map for single words [String, Int], one for tuple words[WordTuple,Int),

4) Merging returned hasmaps into one hashmap and when all lines read from file and write them into a file.

My custom jvm settings is as follows:

-Xms34g

-Xmx34g

-XX:ReservedCodeCacheSize=240m

-XX:+UseParallelGC

-XX:ParallelGCThreads=4

-XX:NewSize=12g

-XX:SoftRefLRUPolicyMSPerMB=50

-ea

-Dsun.io.useCanonCaches=false

-Djava.net.preferIPv4Stack=true

-XX:+HeapDumpOnOutOfMemoryError

-XX:-OmitStackTraceInFastThrow

-Dawt.useSystemAAFontSettings=lcd

-Dsun.java2d.renderer=sun.java2d.marlin.MarlinRenderingEngine

-verbose:gc

-XX:+PrintGCDetails

-Xloggc:gc.log

My application.conf is as follows:

systemParameters {
  linesPerActor = 5
  windowSize = 6
  threadPoolSize = 5
}


akka.actor.deployment {

  /wordTokenizerRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }

  /frequencyCalculatorRouter {
    router = round-robin-pool
    nr-of-instances = 5
  }
}

The problem:

I am processing a text file with size 15gb. Program starts working, after a while, say 2 hours, those tokenizing, calculating operations is almost not working, no operations can perform. The operation that takes 300 milliseconds starts taking 100000 seconds. But the cpu usage is %100 for all processors. I've tried using jvisualvm to motinor it but sampler is not working with this high cpu usage, so I could not identify which process is making cpu %100. I check gc activity from jvisualvm and most of the time it is using about %10 cpu. So, what could be the problem with my program, which process is possibly using all the cpu?

Here some screenshots from jvisualvm when operations in the program is stop but cpu usage is %100:

Garbage collector status screenshot

Overall status screenshot

Hope I explained it clear. Thanks in advance for your answers.

Answer 1

I would look into a few areas.

1. Your heap looks full, including the old generation. Another hint: Of the 8 hours 20 minutes runtime, your application spent 5 hours and 45 minutes in olg generation GC. As you're heap is getting full, it triggers full GCs, one after another. With the Parallel GC, this will use all cores during the full GC. Look at you're gc.log, and see how many times the full GC gets triggered.
2. During the CPU loads, create a few thread dumps. You can use VisualVM' or the `jstack' command. In Visual VM its on the 'Threads' tab, 'Thread Dump'. Look at the stack dumps and look for the 'Runnable' threads, which are not at some blocking / IO API. See what they are doing.

If it's spending time in garbage collection. I would take a heap dump and analyze what memory is held. You can take a heap dump also in VisualVm in the Monitor tab and do a rough analysis there.