How to properly handle signals, so that gperftools CPU profiler still works?

Question

I want to profile my daemon program, that pauses the main thread:

sigset_t signal_mask;
sigemptyset(&signal_mask);
sigaddset(&signal_mask, SIGTERM);
sigaddset(&signal_mask, SIGINT);

int sig;
sigwait(&signal_mask, &sig);

All other threads simply block all signals. As far as I know the profiler uses SIGPROF signal for its operations. If I start profiling with such a code, the output .prof file is empty:

env CPUPROFILE=daemon.prof ./daemon

How should I properly handle signals in main thread and other threads to enable profiling? Or may be an issue is somewhere else?

Answer 1

I need to see more of your code, but your statement that "All other threads simply block all signals" raises...signals.

You have to remember that most system calls were created before the concept of threads existed. Signal handling is one of them. Thus, when you block a signal on ANY thread, it's likely blocked for ALL threads.

In fact, check out the signal(2) manpage:

The effects of signal() in a multithreaded process are unspecified.

Yes, this is sad, but it is the price you must pay for using a low-overhead statistical sampling profiler. And working around it is very easy: just remove SIGPROF (or SIGALRM if you are using the REAL mode) from your signal mask set and you should be fine.

And in general, unless you absolutely have to, you should not be doing process-level signal masking in anything other than the main thread...where "main" doesn't necessarily mean the thread that MAIN() is running in, but rather, the thread you consider the "boss" of all the others, for reasons you have made all too clear already. :)

You can also try using the pthread library's sigmask wrapper pthread_sigmask, but it is unclear to me how well it works in situations such as a child thread REMOVING an entry from a sigmask (pthreads inherit their parent's pthread sigmask).

Answer 2

All other threads simply block all signals.

You simply need to unblock SIGPROF in all threads (or in those that you want to profile). We were just solving exactly the same problem in a multi-threaded daemon.