The LLVM projects MemorySanitizer has a somewhat difficult to satisfy constraint that all system libraries must also be built with MemorySanitizer.
I've been thinking about some ways to enable MSAN for a project I'm working on (it already uses ASAN and UBSAN), specifically with respect to this constraint. One possible solution I've been considering would result in a situation where when the project was compiled with -fsanitize=memory, it would link against non-instrumented versions of dynamic system libraries like libc, and libstdc++, and, say libcrypto (to add one that isn't really part of the 'toolchain'). But then at runtime, the instrumented binary would run in a special environment (basically a different userspace) where all of the system dynamic libraries had been built with -fsanitize=memory.
Is such a scheme viable? It seems to me that it depends on whether building a library with -fsanitize=memory alters its ABI, because if the library ABI differs between the instrumented and non-instrumented versions, then the image built against the non-instrumented system dynamic libraries is going to fail catastrophically at runtime when started up in the environment with the instrumented system libraries.
First of all, compiling with
-fsanitize=memorydoes change ABI e.g. by adding additional (shadow) operands to function calls. This means that combining sanitized and non-sanitized code at runtime will likely crash.On the other hand, Msan does not change link-time library ABI i.e. it does not change symbol sizes or names so external interface of library (i.e. contents of it's ELF symbol table) compiled with
-fsanitize=memorydoes not change.This means that linking against non-sanitized version and later running with sanitized one is ok.