How to obtain threaddump of a java application on Cloud foundry?

Question

I tried a couple of options

a. cf java thread-dump myapp -i0 (Only 1 instance is there so 0)
Response is nothing. logs of myapp show 2019-12-13T14:52:41.15+0800 [SSH/0] OUT Successful remote access by 10.x.x.x:35764 2019-12-13T14:52:41.84+0800 [SSH/0] OUT Remote access ended for 10.x.x.x:35764

b. I did cf ssh myapp

/home/vcap/app/.java-buildpack/open_jdk_jre/bin/ has no jstack and jmap command doesn't work either /home/vcap/app/.java-buildpack/open_jdk_jre/bin/ has the following components

java
orbd servertool
java-buildpack-memory-calculator-3.13.0_RELEASE
pack200
tnameserv
jjs
policytool
unpack200 jvmkill-1.16.0_RELEASE
rmid
keytool
rmiregistry

Question is-> How to get the java threaddump?

Answer 1

I believe the easiest option is to run cf logs to watch your app logs. Then in a second terminal, cf ssh to the app and run ps aux and look for the process id of the Java process. Then run kill -3 <pid>. Your Java process will catch this signal and issue a thread dump. It will go to STDOUT, which conveniently goes to your app's log stream so it should show up in the first terminal.

The crummy part about this is that cf logs will inject some additional text at the beginning of every line, which makes it hard to load the thread dump into tools. You can usually strip that off with cut or awk, or you can redirect cf logs to a file, open with a text editor and find/replace the leading text.

Besides that you have a couple other options:

• If you're using Spring, enable Spring Boot Actuators. You can then use the /actuator/threaddump endpoint to generate and download a thread dump (note the format for this is JSON, not the standard format)

• Use an APM tool or profiler like NewRelic, AppDynamics, Dynatrace or YourKit. These are well supported by the Java buildpack and most will just work.

• Use JMX. It's easy enough to enable with the Java buildpack, just set the env variable JBP_CONFIG_JMX to '{enabled: true}' and restage your app. Once enabled, you can open a tunnel with cf ssh -N -T -L 5000:localhost:5000 <APP_NAME>. Then open jvisualvm, make a new "local" JMX connection to "localhost:5000" and connect. Click the "Threads" tab and click the "Thread Dump" button. It might be a little slow as it's going across the tunnel, but give it a few seconds to finish and you should have a thread dump.

What won't work on Cloud Foundry are jstack and jcmd. The Java buildpack, at the time of writing, installs a JRE and these tools are no longer shipped with the OpenJDK JRE and for legal reasons can't be bundled with it. If you were inclined to do so, you could scp the tools and required shared libraries from a JDK up to your app container and run them, but it's a lot of work and not fun. I'm not super familiar with the cf java plugin, but I suspect it tries to use these tools, which used to be available, and no longer are there.

Answer 2

You should be able to get ThreadDumps with kill -3 <PID> and open that in heap analyzer such as eclipse memory analyzer (MAT) and select Thread Stacks