My OS
is RHEL 7
, and I run a simple Go
program:
package main
import (
"time"
)
func main() {
time.Sleep(1000 * time.Second)
}
During its running, I check the thread count of process:
# cat /proc/13858/status | grep Thread
Threads: 5
While using the pstack
command shipped on RHEL
, it only prints one thread's stack:
# pstack 13858
Thread 1 (process 13858):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:307
#1 0x0000000000422580 in runtime.futexsleep (addr=0x4c7af8 <runtime.timers+24>, val=0, ns=999999997446) at /usr/local/go/src/runtime/os1_linux.go:57
#2 0x000000000040b07b in runtime.notetsleep_internal (n=0x4c7af8 <runtime.timers+24>, ns=999999997446, ~r2=255) at /usr/local/go/src/runtime/lock_futex.go:174
#3 0x000000000040b1e6 in runtime.notetsleepg (n=0x4c7af8 <runtime.timers+24>, ns=999999997446, ~r2=false) at /usr/local/go/src/runtime/lock_futex.go:206
#4 0x000000000043e5de in runtime.timerproc () at /usr/local/go/src/runtime/time.go:209
#5 0x0000000000451001 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1998
#6 0x0000000000000000 in ?? ()
Why does pstack
only print one thread's content?
P.S.: The pstack
script is here:
#!/bin/sh
if test $# -ne 1; then
echo "Usage: `basename $0 .sh` <process-id>" 1>&2
exit 1
fi
if test ! -r /proc/$1; then
echo "Process $1 not found." 1>&2
exit 1
fi
# GDB doesn't allow "thread apply all bt" when the process isn't
# threaded; need to peek at the process to determine if that or the
# simpler "bt" should be used.
backtrace="bt"
if test -d /proc/$1/task ; then
# Newer kernel; has a task/ directory.
if test `/bin/ls /proc/$1/task | /usr/bin/wc -l` -gt 1 2>/dev/null ; then
backtrace="thread apply all bt"
fi
elif test -f /proc/$1/maps ; then
# Older kernel; go by it loading libpthread.
if /bin/grep -e libpthread /proc/$1/maps > /dev/null 2>&1 ; then
backtrace="thread apply all bt"
fi
fi
GDB=${GDB:-/usr/bin/gdb}
# Run GDB, strip out unwanted noise.
# --readnever is no longer used since .gdb_index is now in use.
$GDB --quiet -nx $GDBARGS /proc/$1/exe $1 <<EOF 2>&1 |
set width 0
set height 0
set pagination no
$backtrace
EOF
/bin/sed -n \
-e 's/^\((gdb) \)*//' \
-e '/^#/p' \
-e '/^Thread/p'
pstack uses gdb. This is a quote from golang doc (https://golang.org/doc/gdb):
4 out of 5 threads that you see in /proc are created even before you program enters main. I assume that golang runtime creates them.
Judging from output of strace for gdb I see that
gdb
actually tries to attach to them but after something goes wrong andgdb
does not try to inspect these threads. These are syscalls that gdb issued for these runtime threads but due to unknown reason decided stopped investigating them immediately:However you can inspect them yourself. It seems that these threads belong to golang runtime
Update for gdb 8.0
pstack that uses gdb 8.0 correctly prints backtraces for all threas. The command looks like:
And here is its output (shortened):