I am using stress-ng docker image from https://hub.docker.com/r/polinux/stress-ng/dockerfile to stress my system. I want to use perf tool to monitor metrics.
perf stat -- stress-ng --cpu 2 --timeout 10
runs stress-ng for 10 seconds and returns performance metrics. I tried to do the same with the docker image by using perf stat -- docker run -ti --rm polinux/stress-ng --cpu 2 --timeout 10
. This returns metrics but not the metrics of stress-ng.
The output I got when using 'perf stat' on stress-ng:
Performance counter stats for 'stress-ng --cpu 2 --timeout 10':
19975.863889 task-clock (msec) # 1.992 CPUs utilized
2,057 context-switches # 0.103 K/sec
7 cpu-migrations # 0.000 K/sec
8,783 page-faults # 0.440 K/sec
52,568,560,651 cycles # 2.632 GHz
89,424,109,426 instructions # 1.70 insn per cycle
17,496,929,762 branches # 875.904 M/sec
97,910,697 branch-misses # 0.56% of all branches
10.025825765 seconds time elapsed
The output I got when using perf tool on docker image:
Performance counter stats for 'docker run -ti --rm polinux/stress-ng --cpu 2 --timeout 10':
154.613610 task-clock (msec) # 0.014 CPUs utilized
858 context-switches # 0.006 M/sec
113 cpu-migrations # 0.731 K/sec
4,989 page-faults # 0.032 M/sec
252,242,504 cycles # 1.631 GHz
375,927,959 instructions # 1.49 insn per cycle
84,847,109 branches # 548.769 M/sec
1,127,634 branch-misses # 1.33% of all branches
10.704752134 seconds time elapsed
Can someone please help me with how to get metrics of stress-ng when run using docker?
Carrying on from comments by @osgx,
As is mentioned here, by default, the
perf stat
command will monitor not only all the threads of the process to be monitored, but also its child processes and threads.The problem in this situation is that by running
perf stat
and monitoring thedocker run stress-ng
command, you are not monitoring the actualstress-ng
process. It is important to note that, the processes running as part of the container, will actually not be started by thedocker
client, but rather by thedocker-containerd-shim
process (which is a grandchild process of thedockerd
process).If you run the docker command to run
stress-ng
inside the container and observe the process-tree, it becomes evident.The PPID of the first
stress-ng
process is 26431, which is not thedocker run
command, but actually thedocker-containerd-shim
process. Monitoring thedocker run
command will never reflect correct values, because thedocker
client is completely detached from the process of starting thestress-ng
commands.perf stat
command to the PIDs of the stress-ng processes that are started by the docker runtime.eg, as in the above case, once the
docker run
command is started, you can immediately start doing this -You may increase the
--timeout
a little bit so that the command runs longer, since you are now startingperf stat
post startingstress-ng
. Also you have to account for a small fraction of the initial measuring time lost.perf stat
inside the docker container, something like adocker run perf stat ...
, but for that you would have to start providingprivileges
to your container, since, by default, theperf_event_open
system call is blacklisted indocker
. You can read this answer here.