How to determine LSM hook from a syscall?

Question

From list of LSM hook definitions https://github.com/torvalds/linux/blob/v5.18/include/linux/lsm_hook_defs.h#L179 I can see file_open hook.

I looked into definition of openat syscall https://elixir.bootlin.com/linux/v5.18/source/fs/open.c#L1233

Then I tried looked into functions that it calls, do_sys_op -> do_sys_openat. From there, it becomes harder to determine where the hook is actually called. And I'm actually not sure how security_file_open is called.

Is there a general way to determine if a syscall has an LSM hook?

Answer 1

Probably the easiest way is to use in kernel function graph profiler. It can be used via trace-cmd wrapper, or directly via tracefs.

trace-cmd

$ trace-cmd record -p function_graph -g '*openat*' -F head /etc/mtab
$ trace-cmd report | less

            head-157791 [001]  2350.203452: funcgraph_entry:                   |  __x64_sys_openat() {
            head-157791 [001]  2350.203453: funcgraph_entry:                   |    do_sys_openat2() {
            head-157791 [001]  2350.203453: funcgraph_entry:                   |      getname() {
...................................................................................
            head-157791 [001]  2350.201689: funcgraph_entry:                   |    may_open() {
            head-157791 [001]  2350.201689: funcgraph_entry:                   |      path_noexec() {
            head-157791 [001]  2350.201689: funcgraph_exit:         0.110 us   |      }
            head-157791 [001]  2350.201689: funcgraph_entry:                   |      inode_permission() {
            head-157791 [001]  2350.201689: funcgraph_entry:                   |        generic_permission() {
            head-157791 [001]  2350.201689: funcgraph_exit:         0.120 us   |        }
            head-157791 [001]  2350.201689: funcgraph_entry:                   |        security_inode_permission() {
            head-157791 [001]  2350.201689: funcgraph_exit:         0.110 us   |        }
            head-157791 [001]  2350.201689: funcgraph_exit:         0.520 us   |      }
            head-157791 [001]  2350.201690: funcgraph_exit:         0.970 us   |    }
            head-157791 [001]  2350.201690: funcgraph_entry:                   |    vfs_open() {
            head-157791 [001]  2350.201690: funcgraph_entry:                   |      do_dentry_open() {
            head-157791 [001]  2350.201690: funcgraph_entry:                   |        path_get() {
            head-157791 [001]  2350.201690: funcgraph_entry:                   |          mntget() {
            head-157791 [001]  2350.201690: funcgraph_exit:         0.130 us   |          }
            head-157791 [001]  2350.201690: funcgraph_exit:         0.370 us   |        }
            head-157791 [001]  2350.201691: funcgraph_entry:                   |        try_module_get() {
            head-157791 [001]  2350.201691: funcgraph_exit:         0.120 us   |        }
            head-157791 [001]  2350.201691: funcgraph_entry:                   |        security_file_open() {
            head-157791 [001]  2350.201691: funcgraph_entry:                   |          __fsnotify_parent() {
            head-157791 [001]  2350.201691: funcgraph_exit:         0.120 us   |          }
            head-157791 [001]  2350.201691: funcgraph_entry:                   |          __fsnotify_parent() {
            head-157791 [001]  2350.201691: funcgraph_exit:         0.110 us   |          }
            head-157791 [001]  2350.201691: funcgraph_exit:         0.540 us   |        }
            head-157791 [001]  2350.201691: funcgraph_entry:                   |        xfs_file_open() {
            head-157791 [001]  2350.201692: funcgraph_entry:                   |          generic_file_open() {
            head-157791 [001]  2350.201692: funcgraph_exit:         0.110 us   |          }
            head-157791 [001]  2350.201692: funcgraph_exit:         0.300 us   |        }
            head-157791 [001]  2350.201692: funcgraph_entry:                   |        file_ra_state_init() {
            head-157791 [001]  2350.201692: funcgraph_entry:                   |          inode_to_bdi() {
            head-157791 [001]  2350.201692: funcgraph_exit:         0.090 us   |          }
            head-157791 [001]  2350.201692: funcgraph_exit:         0.260 us   |        }
            head-157791 [001]  2350.201692: funcgraph_exit:         2.410 us   |      }
            head-157791 [001]  2350.201693: funcgraph_exit:         2.600 us   |    }

Directly via tracefs

trace-cmd is just a wrapper over ftrace API, and the same could be achieved without it, only less convenient.

tracefs /sys/kernel/debug/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
cd /sys/kernel/debug/tracing
echo "do_sys_openat2" > set_graph_function # optional, only trace this function
echo 1 > /proc/sys/kernel/ftrace_enabled # enable ftrace
echo function_graph > current_tracer
cat trace_pipe # display trace events

echo nop > current_tracer
echo "" > set_graph_function
echo 0 > /proc/sys/kernel/ftrace_enabled # disable ftrace

Although, I suggest you use trace-cmd since it's much more convenient.

perf ftrace

Another option would be perf ftrace command, which also uses ftrace with function_graph profiler.

$ perf ftrace -t function_graph cat /dev/null

....
10)   0.420 us    |            }
 10)   0.560 us    |          }
 10)               |          vfs_open() {
 10)               |            do_dentry_open() {
 10)               |              path_get() {
 10)   0.060 us    |                mntget();
 10)   0.220 us    |              }
 10)   0.070 us    |              try_module_get();
 10)               |              security_file_open() {
 10)   0.080 us    |                __fsnotify_parent();
 10)   0.200 us    |              }
 10)               |              xfs_file_open() {
 10)   0.080 us    |                generic_file_open();
 10)   0.190 us    |              }
 10)               |              file_ra_state_init() {
 10)   0.070 us    |                inode_to_bdi();
 10)   0.200 us    |              }
 10)   1.410 us    |            }
 10)   1.540 us    |          }
....

As an output, you'll get a function call graph from a kernel. Obviously, you will not see inlined or static functions in this graph.

Then you can just search functions that start with security_ prefix and in this call graph you'll be able to determine by which function it were called and in what order, since many syscalls trigger more than a single LSM hook.