Error finding standard library headers when parsing C++ code with libClang on macOS

Question

I am working on a project using the Clang library to parse C++ source code and further statically analyze its AST. Everything is fine on Linux (the project especially supports the Ubuntu distribution), however supporting macOS was also added to the development plans recently.

However, after building our tool on macOS and linking it against the Clang library (installed from HomeBrew), parsing C++ source fails due to some strange issues resolving the include statements for standard library headers.

As a simplistic test let's say we would like to parse the tinyxml2 project, which is a simple XML parser written in C++, contains only 2 translation units. Since it's uses CMake as its build system, a compile_commands.json file can be easily generated, with a content like this:

[
{
  "directory": "<path to project dir>/tinyxml2/build",
  "command": "/Library/Developer/CommandLineTools/usr/bin/c++  -I<path to project dir>/tinyxml2  -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk -mmacosx-version-min=13.5 -fvisibility=hidden -fvisibility-inlines-hidden -o CMakeFiles/tinyxml2.dir/tinyxml2.cpp.o -c /Users/mate/Documents/cc/projects/tinyxml2/tinyxml2.cpp",
  "file": "<path to project dir>/tinyxml2/tinyxml2.cpp",
  "output": "CMakeFiles/tinyxml2.dir/tinyxml2.cpp.o"
},
{
  "directory": "<path to project dir>/tinyxml2/build",
  "command": "/Library/Developer/CommandLineTools/usr/bin/c++  -I<path to project dir>/tinyxml2  -arch arm64 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk -mmacosx-version-min=13.5 -fvisibility=hidden -fvisibility-inlines-hidden -o CMakeFiles/xmltest.dir/xmltest.cpp.o -c <path to project dir>tinyxml2/xmltest.cpp",
  "file": "<path to project dir>/tinyxml2/xmltest.cpp",
  "output": "CMakeFiles/xmltest.dir/xmltest.cpp.o"
}
]

Executing the commands directly from this file succeeds and builds the respective object files properly.

However, when using this very same compile_commands.json to parameterize libClang, the following strange error occurrs:

In file included from <path to project dir>/tinyxml2/tinyxml2.cpp:24:
<path to project dir>/tinyxml2/tinyxml2.h:37:13: fatal error: cannot open file '<path to project dir>/tinyxml2/cctype': No such file or directory
#   include <cctype>

For some reason the cctype system header is searched in the folder of the project, although the -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk flag was passed and it contains the proper path.

To make it even a bit more strange, when I removed the -I<path to project dir>/tinyxml2 flag, the cctype system header was found, but a different error was raised instead:

In file included from <path to project dir>/tinyxml2/tinyxml2.cpp:24:
In file included from <path to project dir>/tinyxml2/tinyxml2.h:37:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/cctype:37:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__assert:13:
/Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/__config:76:21: fatal error: cannot open file '/Library/Developer/CommandLineTools/SDKs/MacOSX14.0.sdk/usr/include/c++/v1/RTK_device_methods.h': No such file or directory
#  if __has_include(<RTK_device_methods.h>) && defined(_LIBCPP_HAS_NO_RANDOM_DEVICE) && defined(_LIBCPP_HAS_NO_LOCALIZATION)

I have limited experience working on macOS, therefore any help or suggestions are welcome! No such issue arises when working on Linux. I also tried to make sure that the same version of Clang is used in the compile_commands.json, which is linked against our tool, but the same issue remained. (So using /opt/homebrew/opt/llvm@11/bin/clang++ instead of /Library/Developer/CommandLineTools/usr/bin/c++.)

I have managed to reproduce this problem on 2 different Macs - both having an ARM processor if that matters.

I made an MWE from our project (see below), which can reproduce this behaviour. It is a simple CMake project, which will produce an executable binary named parse_test. Then you can test it like ./parse_test <path>/compile_commands.json.

Content of CMakeLists.txt:

cmake_minimum_required(VERSION 3.16.3)
project(mwe-clang-apple)

# Set CXX standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

# Find LLVM
find_package(LLVM REQUIRED CONFIG)
message(STATUS "Found LLVM ${LLVM_PACKAGE_VERSION}")
message(STATUS "Using LLVMConfig.cmake in: ${LLVM_DIR}")

# Check whether RTTI is on for LLVM
message(STATUS "Using RTTI for LLVM: ${LLVM_ENABLE_RTTI}")
if(NOT LLVM_ENABLE_RTTI)
    message(SEND_ERROR "RTTI is required for LLVM")
endif()

# Check LLVM version
if (${LLVM_PACKAGE_VERSION} VERSION_LESS "10")
    message(SEND_ERROR "Loaded LLVM version must be at least 10.0, ${LLVM_PACKAGE_VERSION} found.")
endif()

# Find Clang
find_package(Clang REQUIRED CONFIG)
message(STATUS "Using ClangConfig.cmake in: ${Clang_DIR}")

include_directories(SYSTEM
  ${LLVM_INCLUDE_DIRS}
  ${CLANG_INCLUDE_DIRS})

link_directories(${LLVM_LIBRARY_DIRS})

add_definitions(${LLVM_DEFINITIONS})

add_executable(parse_test
    main.cpp)

target_link_libraries(parse_test
        clangTooling
        clangFrontend
        clang)

if(APPLE)
    # Use Linux-like linking behaviour, as we reference headers from Clang without implementation
    set_target_properties(parse_test PROPERTIES LINK_FLAGS
            "-undefined dynamic_lookup")
endif(APPLE)

Content of main.cpp:

#include <iostream>
#include <string>
#include <vector>
#include <memory>
#include <filesystem>

#include <clang/Tooling/JSONCompilationDatabase.h>
#include <clang/Tooling/Tooling.h>
#include <clang/Frontend/CompilerInstance.h>
#include <clang/Frontend/FrontendActions.h>

namespace fs = std::filesystem;

int parseWorker(const clang::tooling::CompileCommand& command_);

int main(int argc, char* argv[])
{
    if (argc < 2) {
        std::cout << "Pass the path to the JSON Compilation Database as an argument!" << std::endl;
        return 1;
    }

    std::string jsonFile_ = std::string(argv[1]);
    std::string errorMsg;

    std::unique_ptr<clang::tooling::JSONCompilationDatabase> compDb
            = clang::tooling::JSONCompilationDatabase::loadFromFile(
                    jsonFile_, errorMsg,
                    clang::tooling::JSONCommandLineSyntax::Gnu);

    if (!errorMsg.empty())
    {
        std::cout << "Error: " << errorMsg << std::endl;
        return 1;
    }

    //--- Read the compilation commands compile database ---//

    std::vector<clang::tooling::CompileCommand> compileCommands =
            compDb->getAllCompileCommands();

    for (const auto& command : compileCommands)
    {
        std::cout << " Parsing " << command.Filename << std::endl;

        int error = parseWorker(command);

        if (error)
            std::cout << " Parsing " << command.Filename << " has been failed." << std::endl;
    }

   return 0;
}

int parseWorker(const clang::tooling::CompileCommand& command_)
{
    //--- Assemble compiler command line ---//

    std::vector<const char*> commandLine;
    commandLine.reserve(command_.CommandLine.size());
    commandLine.push_back("--");
    std::transform(
            command_.CommandLine.begin() + 1, // Skip compiler name
            command_.CommandLine.end(),
            std::back_inserter(commandLine),
            [](const std::string& s){ return s.c_str(); });

    int argc = commandLine.size();

    std::string compilationDbLoadError;
    std::unique_ptr<clang::tooling::FixedCompilationDatabase> compilationDb(
            clang::tooling::FixedCompilationDatabase::loadFromCommandLine(
                    argc,
                    commandLine.data(),
                    compilationDbLoadError));

    if (!compilationDb)
    {
        std::cout
                << "Failed to create compilation database from command-line. "
                << compilationDbLoadError
                << std::endl;
        return 1;
    }

    //--- Start the tool ---//

    fs::path sourceFullPath(command_.Filename);
    if (!sourceFullPath.is_absolute())
        sourceFullPath = fs::path(command_.Directory) / command_.Filename;

    clang::tooling::ClangTool tool(*compilationDb, sourceFullPath.string());

    int error = tool.run(clang::tooling::newFrontendActionFactory<clang::SyntaxOnlyAction>().get());

    return error;
}

Answer 1

Unfortunately, it seems to me that apple has started an all-out war with anyone who wants to develop with anything other than XCode and stuff they ship.

I have been an apple user for close to 30 years, and they have systematically, in the name of so-called security, made it more and more difficult to do things that are so easy with normal *nix systems.

I ended up compiling a source file with -v...

clang++ -v -g3 -std=c++20 foo.cpp -c

And then took that output and added all the include files to my command line, then systematically removed them until it worked with the minimum I needed.

I ended up with this as extra arguments I passed to my application, which then forwarded them as arguments to the libclang function to parse the file.

 xtra_incs=-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk \
    -isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1 \
    -isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/15.0.0/include