Thread "WebContainer : 0" (00000029) has been active for 647279 milliseconds and may be hung

Question

I am getting the following exception in WebSphere while trying to generate an excel report using jasper.

 ThreadMonitor W   WSVR0605W: Thread "WebContainer : 0" (00000029) has been active for 647279 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

Tried increasing the default thread pool size from 20 to 50 and also increased WebContainer pool size from 50 to 150. Not worked.

Any solution for this.

Answer 1

Your error message tells that the thread named “ WebContainer : 0” has been doing something for 647 seconds or 10.7 minutes and 1 thread active in the JVM that my also hung (been active for longer than the threshold time).

A hung thread is a thread in Java EE apps which is being blocked by a call or is waiting on a monitor for a locked object to be released, so that the thread can use it. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). CPU utilization for that server's java process may become high and unresponsive if number of threads grows.

There are ways to increase the hangtime (default 10min), but not recommended. You have optimize or limit your excel report generation process. I think, Report you are extracting is very big or hitting the db more for generation, basically might be due to code bug. Increasing the number of thread pool has nothing to do for your issue.

In sysout.log: Below lines to the posted warning message, should give some idea about the Java EE class causing this issue. FFDC also helps this to figure it out.

The hang detection option is turned on by default. You can configure it to accommodate the environment, so that potential hangs can be reported. When a hung thread is detected, WAS notifies you so that you can troubleshoot the problem.

Using the hang detection policy, you can specify a time that is too long for a unit of work to complete. The thread monitor checks all managed threads including Web container threads.

1. From the administrative console, click Servers > Application Servers > server_name
2. Under Server Infrastructure, click Administration > Custom Properties & Click New.
3. Add the following properties:

Name: com.ibm.websphere.threadmonitor.interval
Value: The frequency (in seconds) at which managed threads in the selected application server will be interrogated.
Default: 180 seconds (three minutes).

Name: com.ibm.websphere.threadmonitor.dump.java
Value: Set to true to execute the dumpThreads function when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore dump can be analyzed to determine what the reported thread
Default: false (0).

Name: com.ibm.websphere.threadmonitor.threshold
Value: The length of time (in seconds) in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung.
Default: The default value is 600 seconds (ten minutes).

Name: com.ibm.websphere.threadmonitor.false.alarm.threshold
Value: The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment. (Link to Detecting hung threads in J2EE applications for information on false alarms)
Default: 100

To disable the hang detection option, set the com.ibm.websphere.threadmonitor.interval property to less than or equal to zero.

4. Save & Restart the WAS instance

Answer 2

This is not an Exception. As you can see it has W level, which is Warning (ThreadMonitor W WSVR0605W). It just informs you that your report generation is extremely long. Longer than 647279 milliseconds. When you will look further down the log, you will most likely notice similar message telling that your thread has finished the job, and is no longer hanged.

Increasing WebContainer pool has nothing to do with that. You have to look at class creating your report and see, if you can make it quicker.

Another solution would be to just handle the report creation as async task using either Async Beans or JMS (you unfortunately are on v7, so no EJB 3.1 or async servlets are available), store it into database or file system and later retrieve using separate request, when the report is done.

Answer 3

In my team we found that Websphere 8.5.5.13 never shuts down the hung threads, so our team came with a solution for a transactional system which process high number of concurrent threads, so instead of hung forever we found a solution using Java 8 that times out after a parameter setting. Below code shows a demo of how to write a Function that executes critical arbitrary code with timeout.

public class MyFactory
{
TimeUnit timeUnit=TimeUnit.MILLISECONDS;
int REQUEST_TIMEOUT=50;
int MAX_THREADS=1;
ExecutorService executor = Executors.newFixedThreadPool(MAX_THREADS);

public Function<Supplier<?>, ?> executeWithTimeout =  typeSupplier-> {
    final ExecutorService singleCallableExec = 
    Executors.newSingleThreadExecutor();
    try
        {
            return executor.submit(()-> {
                try {   return singleCallableExec.submit(()->{
                    return typeSupplier.get();
                    }).get(REQUEST_TIMEOUT, timeUnit);  
                }
                catch (Exception t2) { singleCallableExec.shutdownNow(); throw t2; }
                finally {
                    Optional.of(singleCallableExec)
                    .filter(e -> !e.isShutdown())
                    .ifPresent(ExecutorService::shutdown);
                }
            }).get();
        }
        catch (InterruptedException | ExecutionException e)
        {
            throw new RuntimeException(e);
        }
};

so this can be put into a factory class and call this by every thread requiring to execute critical code which may timeout forever, this will be useful to send a timeout exception which raises an alerts instead of waiting for system to go out of resources

    public static void main (String args[])
    {
        MyFactory mf=new MyFactory();
        try
        {
            long result = (long) mf.executeWithTimeout.apply(()->mf.longCalculation());
            System.out.println("Result:" + result);
        }
        catch (Exception te)
        {
            if (te instanceof RuntimeException && 
                te.getCause() != null && te.getCause() instanceof ExecutionException 
                && te.getCause().getCause() != null && te.getCause().getCause() 
                instanceof TimeoutException)
            {
                System.out.println(new Date() + " Raising alarm, TimeoutException");
            }
        }
        finally
        {
            mf.executor.shutdownNow();
        }
    }

    public long longCalculation()
    {
        long result=LongStream.range(0, (long) Math.pow(2,32)).max().getAsLong();
        System.out.println(new Date() + " Calculation result " + result);
        return result;
    }
}