Using IWYU fix_includes.py with CMake - more friendly instuctions

Question

I have problem running tool fix_includes.py that automatically fixes up source files based on the include-what-you-use recommendations.

I have configured IWYU tool accordingly to the documentation:

• downloaded IWYU version matching my Clang version

• located it following the issue 100 So IWYU is located in /usr/bin/iwyu/include-what-you-use and clang in /usr/lib/llvm-10/lib/clang/10.0.0

• in /usr/bin/iwyu/ built it with following commands:

mkdir build && cd build
cmake -G "Unix Makefiles" -DCMAKE_PREFIX_PATH=/usr/lib/llvm 10/lib/clang/10.0.0
make
sudo make install

• set CXX_INCLUDE_WHAT_YOU_USE property in CMakefile.txt in my project root /home/kpolok/worksapce/projects/using_iwyu_proj:

cmake_minimum_required(VERSION 3.16.3)
set(CMAKE_CXX_STANDARD 17)
project(using_mocks_with_di_proj LANGUAGES CXX)

option(USE_IWYU "Enable include-what-you-use reports during build" ON)

if(USE_IWYU)
    find_program(IWYU_PATH NAMES include-what-you-use iwyu)
    if(NOT IWYU_PATH)
        message(FATAL_ERROR "Could not find the program include-what-you-use")
    endif()
    message("Found IWYU ${IWYU_PATH}")
    set(CMAKE_CXX_INCLUDE_WHAT_YOU_USE ${IWYU_PATH})
    set(CMAKE_CXX_INCLUDE_WHAT_YOU_USE ${IWYU_PATH})
endif()

set(PROJ_DIR ${PROJECT_SOURCE_DIR})

add_subdirectory(app)
add_subdirectory(src)

And it produces report while building:

[ 20%] Building CXX object src/Car/CMakeFiles/Car.dir/src/Car.cpp.o
Warning: include-what-you-use reported diagnostics:

/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/inc/Car.h should add these lines:
class MotorIf;

/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/inc/Car.h should remove these lines:
- #include "MotorIf.h"  // lines 2-2

The full include-list for /home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/inc/Car.h:
class MotorIf;
---

/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/src/Car.cpp should add these lines:
#include "inc/MotorIf.h"  // for MotorIf

/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/src/Car.cpp should remove these lines:

The full include-list for /home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/src/Car.cpp:
#include "inc/Car.h"
#include <iostream>       // for basic_ostream::operator<<, operator<<, endl
#include "inc/MotorIf.h"  // for MotorIf
---

[ 40%] Building CXX object src/Car/CMakeFiles/Car.dir/src/CarMotor.cpp.o
[ 60%] Linking CXX static library libCar.a
make[3]: Leaving directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build'
[ 60%] Built target Car
make[3]: Entering directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build'
Scanning dependencies of target Executable
make[3]: Leaving directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build'
make[3]: Entering directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build'
[ 80%] Building CXX object app/CMakeFiles/Executable.dir/main.cpp.o
Warning: include-what-you-use reported diagnostics:

/home/kpolok/worksapce/projects/using_iwyu_proj/app/main.cpp should add these lines:

/home/kpolok/worksapce/projects/using_iwyu_proj/app/main.cpp should remove these lines:
- #include <vector>  // lines 2-2

The full include-list for /home/kpolok/worksapce/projects/using_iwyu_proj/app/main.cpp:
#include <iostream>        // for operator<<, endl, basic_ostream, cout, ost...
#include "inc/Car.h"       // for Car
#include "inc/CarMotor.h"  // for CarMotor

That part went pretty smooth, but now i want to use the script fix_includes.py. So I once more followed the documentation and invoked this commands in my project root /home/kpolok/worksapce/projects/using_iwyu_proj:

make -k CXX=/usr/bin/iwyu/build/bin/include-what-you-use CXXFLAGS="-Xiwyu --error_always" 2> ./err.txt
python3 /usr/bin/iwyu/include-what-you-use/fix_includes.py < ./err.txt

But after first command I get:

rm -rf build
mkdir build
cmake -S . -B ./build/
-- The CXX compiler identification is unknown
-- Check for working CXX compiler: /usr/bin/iwyu/build/bin/include-what-you-use
-- Check for working CXX compiler: /usr/bin/iwyu/build/bin/include-what-you-use -- broken
CMake Error at /usr/share/cmake-3.16/Modules/CMakeTestCXXCompiler.cmake:53 (message):
  The C++ compiler

    "/usr/bin/iwyu/build/bin/include-what-you-use"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/kpolok/worksapce/projects/using_iwyu_proj/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make cmTC_d324c/fast && make[1]: Entering directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build/CMakeFiles/CMakeTmp'
    /usr/bin/make -f CMakeFiles/cmTC_d324c.dir/build.make CMakeFiles/cmTC_d324c.dir/build
    make[2]: Entering directory '/home/kpolok/worksapce/projects/using_iwyu_proj/build/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_d324c.dir/testCXXCompiler.cxx.o
...
 CMake will not be able to correctly generate this project.
 Call Stack (most recent call first):
 CMakeLists.txt:3 (project)

By the way, the Makefile in my project root looks like this:

all: prepare configure build_project

prepare:
    rm -rf build
    mkdir build

configure:
    cmake -S . -B ./build/

build_project:
    cmake --build ./build/

I'm quite new to cmake/clang stuff so I don't know what I'm doing wrong. I suppose that those commands should be invoked somewhere else?But exactly where? Maybe something else is off?

Btw IMO the instructions I found in IWYU documentations aren't really clear.

Answer 1

As @Friedrich mentioned, I was confusing make with cmake (in fact, I was blindly following the IWYU documentation, without trying to understand, what I was actually doing). The solution to the problem is easy. So, during the build process, CMake already generates the report required by the fix_includes.py script by utilizing the CMAKE_CXX_INCLUDE_WHAT_YOU_USE variable.

option(USE_IWYU "Enable include-what-you-use reports during build" ON)

if(USE_IWYU)
    find_program(IWYU_PATH NAMES include-what-you-use iwyu)
    if(NOT IWYU_PATH)
        message(FATAL_ERROR "Could not find the program include-what-you-use")
    endif()
    message("Found IWYU ${IWYU_PATH}")
    set(CMAKE_CXX_INCLUDE_WHAT_YOU_USE ${IWYU_PATH})
endif()

I just needed to separate IWYU report from the build logs part:

if(USE_IWYU)
    find_program(IWYU_PATH NAMES include-what-you-use iwyu)
    if(NOT IWYU_PATH)
        message(FATAL_ERROR "Could not find the program include-what-you-use")
    endif()
    message("Found IWYU ${IWYU_PATH}")
    set(CMAKE_CXX_INCLUDE_WHAT_YOU_USE ${IWYU_PATH} CXXFLAGS="-Xiwyu --error_always")
endif()

Thanks to --error_always, IWYU report parts are treated as errors. Now I need to collect this report e.g. by redirecting it to separate file:

build_project:
    cmake --build ./build/ 2> report.txt

This is the part of my Makefile that triggers cmake build process. All that's left is to run the fix_includes.py:

using_iwyu_proj$ python3 /usr/bin/iwyu/include-what-you-use/fix_includes.py < report.txt
>>> Fixing #includes in '/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/inc/Car.h'
>>> Fixing #includes in '/home/kpolok/worksapce/projects/using_iwyu_proj/src/Car/src/Car.cpp'
>>> Fixing #includes in '/home/kpolok/worksapce/projects/using_iwyu_proj/app/main.cpp'
IWYU edited 3 files on your behalf.

Answer 2

I can explain why it's not working but unfortunately I wasn't able to find how to run fix_includes.py from CMake.

There's something peculiar about your project: you're using Make to call CMake. CMake will in turn generate a makefile and call Make to build the project (cmake --build does this under the hood). Having a makefile to call CMake is uncommon but not inherently wrong.

When using CMake, projects usually decide to only have a CMakeLists.txt in their root dir and generate makefiles with cmake -S <project_root> -B <some_build_dir>.

The instructions you refer to are - I think - iwyu's readme. The guidelines for use with Make and CMake found there are mutually exclusive. Take a look in your build directory, there's another Makefile there and it looks nothing like yours. If anything, you would have to call make CXX=iwyu 2> err there but that will not work either. This is owed to how CMake generated makefiles look. There's just no variable CXX in there to override.

Replacing the compiler with iwyu before running CMake will not work either, as you found out. CMake will try to compile some test programs to gauge your compiler's capabilities and iwyu just doesn't compile anything. So CMake considers this "compiler" to be broken and refuses to go on.

The correct way to use iwyu from CMake is to set a special CMake variable CMAKE_CXX_INCLUDE_WHAT_YOU_USE either from command line when configuring with CMake:

cmake "-DCMAKE_CXX_INCLUDE_WHAT_YOU_USE=/usr/bin/iwyu" -S . -B build

or from CMakeLists.txt as you already do (twice, btw). You also observed this to work.

So you will either have to give up on using CMake and write a plain old makefile (containing a variable CXX) or find a way to massage CMake's output into something that can be fed into fix_includes.py.

As I initially wrote, I don't have a solution for you. Maybe you can figure something out on your own. I'm hoping a more proficient user will write a better answer.

To get you started, CMake is actually very well documented. For using static code checkers (iwyu and others), I find this blog post helpful. And there's also the question How can I use the tool 'Include What You Use' together with CMake to detect unused headers? None of them explain how to run fix_includes.py from CMake, unfortunately.