blktrace output error in docker container

Question

my docker container base image is ubuntu, and I ran it with full privilege options that means in run command I use these switches:

--cap-add=SYS_ADMIN --security-opt apparmor:unconfined

I wanna use blktrce using below command:

sudo blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out

but, first time when is use this command I get this error:

Debugfs is not mounted at /sys/kernel/debug

i searched and found this solution which led to use this command:

mount -t debugfs none /sys/kernel/debug

after that, when I use the blktrace command again, i get this error:

BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device
Thread 1 failed open /sys/kernel/debug/block/(null)/trace1: 2/No such file or directory
Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory
Thread 2 failed open /sys/kernel/debug/block/(null)/trace2: 2/No such file or directory
Thread 4 failed open /sys/kernel/debug/block/(null)/trace4: 2/No such file or directory
Thread 3 failed open /sys/kernel/debug/block/(null)/trace3: 2/No such file or directory
Thread 5 failed open /sys/kernel/debug/block/(null)/trace5: 2/No such file or directory
Thread 7 failed open /sys/kernel/debug/block/(null)/trace7: 2/No such file or directory
Thread 8 failed open /sys/kernel/debug/block/(null)/trace8: 2/No such file or directory
Thread 6 failed open /sys/kernel/debug/block/(null)/trace6: 2/No such file or directory
Thread 12 failed open /sys/kernel/debug/block/(null)/trace12: 2/No such file or directory
Thread 10 failed open /sys/kernel/debug/block/(null)/trace10: 2/No such file or directory
Thread 13 failed open /sys/kernel/debug/block/(null)/trace13: 2/No such file or directory
Thread 15 failed open /sys/kernel/debug/block/(null)/trace15: 2/No such file or directory
Thread 14 failed open /sys/kernel/debug/block/(null)/trace14: 2/No such file or directory
Thread 17 failed open /sys/kernel/debug/block/(null)/trace17: 2/No such file or directory
Thread 16 failed open /sys/kernel/debug/block/(null)/trace16: 2/No such file or directory
Thread 18 failed open /sys/kernel/debug/block/(null)/trace18: 2/No such file or directory
Thread 11 failed open /sys/kernel/debug/block/(null)/trace11: 2/No such file or directory
Thread 19 failed open /sys/kernel/debug/block/(null)/trace19: 2/No such file or directory
Thread 20 failed open /sys/kernel/debug/block/(null)/trace20: 2/No such file or directory
Thread 9 failed open /sys/kernel/debug/block/(null)/trace9: 2/No such file or directory
Thread 21 failed open /sys/kernel/debug/block/(null)/trace21: 2/No such file or directory
Thread 22 failed open /sys/kernel/debug/block/(null)/trace22: 2/No such file or directory
Thread 23 failed open /sys/kernel/debug/block/(null)/trace23: 2/No such file or directory
FAILED to start thread on CPU 0: 1/Operation not permitted
FAILED to start thread on CPU 1: 1/Operation not permitted
FAILED to start thread on CPU 2: 1/Operation not permitted
FAILED to start thread on CPU 3: 1/Operation not permitted
FAILED to start thread on CPU 4: 1/Operation not permitted
FAILED to start thread on CPU 5: 1/Operation not permitted
FAILED to start thread on CPU 6: 1/Operation not permitted
FAILED to start thread on CPU 7: 1/Operation not permitted
FAILED to start thread on CPU 8: 1/Operation not permitted
FAILED to start thread on CPU 9: 1/Operation not permitted
FAILED to start thread on CPU 10: 1/Operation not permitted
FAILED to start thread on CPU 11: 1/Operation not permitted
FAILED to start thread on CPU 12: 1/Operation not permitted
FAILED to start thread on CPU 13: 1/Operation not permitted
FAILED to start thread on CPU 14: 1/Operation not permitted
FAILED to start thread on CPU 15: 1/Operation not permitted
FAILED to start thread on CPU 16: 1/Operation not permitted
FAILED to start thread on CPU 17: 1/Operation not permitted
FAILED to start thread on CPU 18: 1/Operation not permitted
FAILED to start thread on CPU 19: 1/Operation not permitted
FAILED to start thread on CPU 20: 1/Operation not permitted
FAILED to start thread on CPU 21: 1/Operation not permitted
FAILED to start thread on CPU 22: 1/Operation not permitted
FAILED to start thread on CPU 23: 1/Operation not permitted    

how can i solve that?

Update1:

there is sda folder in /sys/kernel/debug/block/ and into this folder there are these files:

trace0 trace1 trace2 etc.

Update2:

@abligh thank for your answer, but did not help. The strace output about ioctl is:

ioctl(3, BLKTRACESETUP, {act_mask=64, buf_size=524288, buf_nr=4, start_lba=0, end_lba=0, pid=0}, 0x7ffe8a4ceac0) = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device\n", 61) = 61
ioctl(3, BLKTRACESTOP, 0x7f6fd19789d0) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKTRACESTOP, 0x7ffe8a4ce540) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKTRACESTOP, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)`
ioctl(3, BLKTRACETEARDOWN, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)

to answer your question about why i run blktrace in container, i would say I'm using containers as cluster, so i need the trace of each node.

Answer 1

I don't have enough information to debug this, but you asked for an answer to be posted.

The root of the problem seems to be here:

BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device

This is indicating that the ioctl call to set up block tracing is failing. It is failing inside the container, but according to the comments works outside the container. This would indicate that the problem is with the container setup or a limitation in the kernel that prevents that ioctl from being used in a container at all.

The error inappropriate ioctl for device is errno 25, i.e. ENOTTY. I can't immediately see what would be returning that unless it can't find the device node at all (given you've already demonstrated from the fact it works outside the container that the block trace code is compiled in). I can't remember whether this is in a module, but it would be worth trying block tracing outside the container first (then check it inside the container), just to check this isn't a module loading issue.

The first step in debugging this would be to use strace tool (as suggested above) so you know exactly which system call is being made with what parameters. EG run:

sudo strace -f -s2048 -o/tmp/trace blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out

and look at /tmp/trace afterwards. strace will list all the system calls made. See if you can determine from that which ioctl is failing.

Secondly, I'd ensure that the block device that you are trying to trace actually appears within the correct place(s) in /proc/ and /sys/. Something wrong is happening here:

Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory

Note the (null) in the debug line, which clearly should not be there - that should be the name of the block device. This is possibly a consequence of the failed ioctl, or possibly indicative of a problem within the /sys/ hierarchy.

BLKTRACESETUP is handled in the kernel here. This eventually calls doblktracesetup here. I cannot immediately see any reason why this would not work from an appropriately permissioned container, which makes me suspect your /sys hierarchy might not be set up right. But the output of strace would be helpful.

Also, the inevitable comment: why are you doing this in a container? Why not run it outside the container?

EDIT: Looks like it might be a kernel thing. See https://github.com/scaleway/kernel-tools/issues/107 - this suggests you (a) need to modprobe the specific modules first, and (b) may need a specific kernel.