gprof shows every function as 100%

Question

I'm using a modified auditd package, and linked gprof with it.

But from the output of gprof XX | gprof2dot | dot -Tpng output.png, all functions are 100% usage, so I cannot tell who's consuming most of the CPU resources.

(Right click on the image to have a full view)

Any ideas? Is that normal, and most importantly, how should I know which function consumes most CPU?

P.S: original gprof output http://pastie.org/8546568

Answer 1

At the top of the flat profile (in the pastie.org link) is the immediate answer:

Flat profile:

Each sample counts as 0.01 seconds.
 no time accumulated

Since everything took exactly no time at all, everything took the same amount of time as everything else.

The more interesting question is, why did everything take no time at all? The problem here is that auditd is setuid, and (at least in general) you cannot profile setuid programs.

(To get a profile of a program that would be setuid, make a non-suid version of it, and profile that. Sometimes you need to tweak the internals a bit to make it behave well now that it's not setuid.)

After poking around some more, I don't know why it fails. Linux uses a libc compatibility routine to fake the profil() system call for gprof. This needs the SIGPROF-based ITIMER_PROF interval timer but should generally work.

An alternative is to use OProfile, which is more general-purpose (it can profile system calls) and probably better-supported these days.