analysis of a java process "not responding"?

Question

My java process stopped responding. I tried to jstack but failed with below error.

21039: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Then I used -F option, but "No deadlocks found."

Other info:

java version: java version

jmap: jmap

jstat: jstat

jinfo: jinfo

Can anyone help have a look and share some links on troubleshooting this kind of java "not responding" issue?

Answer 1

Possible reasons of Unable to open socket file problem:

1. The target PID is not a HotSpot JVM process.
   This is obviously not your case, since jinfo PID works fine.
2. JVM is started with -XX:+DisableAttachMechanism option.
   This can be also verified by jinfo PID.

3. Attach socket /tmp/.java_pidNNN has been deleted.
   There is a common practice to clean /tmp automatically with some scheduled script. In this case you should configure the cleanup software not to delete .java_pid* files.

   How to check: run lsof -p PID | grep java_pid
   If it lists a socket file, but the file does not exist, then this is exactly the described problem.

4. Credentials of the current user (euid/egid) do not match the owner of the attach socket. Make sure you run jstack by the same user as JVM. Attach won't work if you run jstack by a different user, even if this user is root.

5. /tmp directory of the target process is not the same as /tmp of your shell. This may happen in the following cases:
   • JVM is started in a different mount namespace. Typically this happens when JVM runs in a Docker container. Running jstack from within the same container will help.
   • JVM is started in chroot environment. For example, LXC containers may use chroot.
     How to check: run readlink -f /proc/PID/root/tmp to see if it points to /tmp or to some other directory.

6. Current working directory of the target JVM belongs to a file system that does not allow to change permissions. CIFS and DrvFs (WSL) are examples of such file systems.

   How to check: run umask 077; touch /proc/PID/cwd/somefile.tmp, then verify that file owner is yourself, and file permissions are 600.

7. JVM is busy and cannot reach a safepoint. For instance, JVM is in the middle of long-running garbage collection.

   How to check: run kill -3 PID. JVM should print a thread dump and heap info in its console. If JVM does not dump anything, but the process consumes almost 100% CPU or shows high I/O utilization, then this looks like the described problem.

8. JVM process is suspended.

   How to check: run ps PID. The STAT column of alive JVM process should be Sl.

More about the internals of jstack.

There is also jattach project which is a better alternative to jstack / jmap. It can automatically handle credentials issue, it works with Docker containers, supports chroot'ed JVMs and handles uncommon file systems.