The Unresolved extern function 'cudaCGGetIntrinsicHandle' strikes back

Question

I'm building the examples for my cuda-api-wrappers repo. A while ago, I encountered a problem: Whenever I tried compiling a .cu file which code using the "cooperative groups" mechanism, I got an error message:

ptxas fatal   : Unresolved extern function 'cudaCGGetIntrinsicHandle'

which was strange, since I was only compiling. Well, I resolve that using the advice on this nVIDIA DevTalk forum thread:

I suggest ... specifying -rdc=true

I did, and that seemed to work. Yet, now (with CUDA 10.1 on Devuan 3 ~= Debian 10), I'm getting an error during linking:

/usr/bin/nvcc   -I/path/to/src/cuda-api-wrappers/src -I/path/to/src/cuda-api-wrappers  -gencode arch=compute_61,code=sm_61 -Xcompiler -Wall -O3 -DNDEBUG   -rdc true -std=c++11 -x cu -c /path/to/src/cuda-api-wrappers/examples/by_runtime_api_module/execution_control.cu -o CMakeFiles/execution_control.dir/examples/by_runtime_api_module/execution_control.cu.o
/usr/bin/nvcc   -gencode arch=compute_61,code=sm_61 -Xcompiler -Wall -O3 -DNDEBUG -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink CMakeFiles/execution_control.dir/examples/by_runtime_api_module/execution_control.cu.o -o CMakeFiles/execution_control.dir/cmake_device_link.o  -L/usr/lib/x86_64-linux-gnu/stubs  -L/usr/lib/gcc/x86_64-linux-gnu/8 lib/libcuda-api-wrappers.a /usr/lib/x86_64-linux-gnu/libcudart_static.a -lpthread -ldl  -lcudadevrt -lcudart_static -lrt 
nvlink error   : Undefined reference to 'cudaCGGetIntrinsicHandle' in 'CMakeFiles/execution_control.dir/examples/by_runtime_api_module/execution_control.cu.o'
nvlink error   : Undefined reference to 'cudaCGSynchronizeGrid' in 'CMakeFiles/execution_control.dir/examples/by_runtime_api_module/execution_control.cu.o'

Why am I still getting these errors, and what can I do about it?

Answer 1

This is a very tricky bug... I haven't managed to generate a proper MCVE for it yet. However, if I append:

-L /usr/lib/x86_64-linux-gnu

to the failing command-line, i.e. make it:

/usr/bin/nvcc   -gencode arch=compute_61,code=sm_61 -Xcompiler -Wall -O3 -DNDEBUG -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink CMakeFiles/execution_control.dir/examples/by_runtime_api_module/execution_control.cu.o -o CMakeFiles/execution_control.dir/cmake_device_link.o  -L/usr/lib/x86_64-linux-gnu/stubs  -L/usr/lib/gcc/x86_64-linux-gnu/8 lib/libcuda-api-wrappers.a /usr/lib/x86_64-linux-gnu/libcudart_static.a -lpthread -ldl  -lcudadevrt -lcudart_static -lrt -L /usr/lib/x86_64-linux-gnu

it works. I wonder how I can get CMake to want to do that.