I made further investigations and at least now I understand why I have this linker error. Here is a reduced reproducible example.
Regarding CMake, I still consider this behavior as a bug because these 2 additional -L
paths are the reason why why compilation fails. Is there a way to tell CMake to not add them?
I have the following toolchain file:
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
set(CMAKE_SYSROOT /home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/sysroot)
set(tools /home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12)
#set(tools /usr) # explained later
set(CMAKE_C_COMPILER ${tools}/bin/aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER ${tools}/bin/aarch64-linux-gnu-g++)
set(CMAKE_CUDA_COMPILER /home/nc/.benv/tools/cuda/Linux_10.2/bin/nvcc)
set(CMAKE_CUDA_HOST_COMPILER ${CMAKE_CXX_COMPILER})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
In my setup, the linker reports an error only when the project has both .cpp
and .cu
files. With only one type of file, everything works fine. The error is:
/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/bin/../lib/gcc/aarch64-linux-gnu/7.5.0/../../../../aarch64-linux-gnu/bin/ld: cannot find /lib/libpthread.so.0
/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/bin/../lib/gcc/aarch64-linux-gnu/7.5.0/../../../../aarch64-linux-gnu/bin/ld: cannot find /usr/lib/libpthread_nonshared.a
Here's the output of cmake --build . --verbose
(only the link part, the compilation part has no problems):
/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/bin/aarch64-linux-gnu-g++ --sysroot=/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/sysroot CMakeFiles/cuda_hello_world.dir/src/main.cpp.o CMakeFiles/cuda_hello_world.dir/src/main.cu.o CMakeFiles/cuda_hello_world.dir/cmake_device_link.o -o bin/cuda_hello_world -L/home/nc/.benv/tools/cuda/Linux_10.2/targets/aarch64-linux/lib/stubs -L/home/nc/.benv/tools/cuda/Linux_10.2/targets/aarch64-linux/lib -L/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/aarch64-linux-gnu/libc/lib -L/home/nc/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/aarch64-linux-gnu/libc/usr/lib -lcudadevrt -lcudart_static -lrt -lpthread -ldl
Please note the two additional -L
paths (last two). I don't know where they come from, I do not set them explicitly. But now the linker searches for the -pthread
in these folders.
The first path has the library:
~/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/aarch64-linux-gnu/libc/lib $
find . -name "libpthread*"
./libpthread-2.25.so
./libpthread.so.0
The second has the file too:
~/.benv/tools/linaro-l4t/Linux_7.5.0-2019.12/aarch64-linux-gnu/libc/usr/lib $
find . -name "libpthread*"
./libpthread.a
./libpthread_nonshared.a
./libpthread.so
but here libpthread.so
is a text file with the contents:
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf64-littleaarch64)
GROUP ( /lib/libpthread.so.0 /usr/lib/libpthread_nonshared.a )
To be honest, I don't know why the .so
file is not a binary but text, but it looks like the linker uses exactly these library names to link and fails.
Next, if I remove set(CMAKE_SYSROOT ...)
command, these 2 directories are not generated and everything works. So somehow the sysroot path affects the generated commands.
Let's return it back. The requested library exists in ${CMAKE_SYSROOT}/lib
, but if I manually add the path with target_link_directories(cuda_hello_world PRIVATE ${CMAKE_SYSROOT}/lib)
it's not added, I don't see it in the output, it's simply ignored. Any other path (just ${CMAKE_SYSROOT}
, for example) is added and I can see it.
Next, if I uncomment set(tools /usr)
where I have a crosscompiler (/usr/bin/aarch64-linux-gnu-g++
) these 2 directories are not generated and everything works.
Next, if I remove either .cu
or .cpp
these 2 directories are not generated and everything works.
The linaro cross-compiler toolchain and the sysroot are 100% correct - I got them from the official page and was able to compile official Cuda samples which use make. Also I tried to mount a filesystem of the device as sysroot and still got the same error.
I've read a lot, we had a discussion with my colleagues but we don't know why is this happening. At the moment this is a blocker for us and I'd like to find a workaround.
Sorry for the long text, but I feel I had to explain it carefully because this is very specific case, I believe.
Though the issue is closed I can't agree with the suggestion. The purpose of --compiler-options
is to pass options to the host compiler (which is a c++
compiler, gcc
in my case). In my example, when I call nvcc
directly, the options are passed to CUDA
initially (nvcc
is CUDA
) but propagated to c++
. nvcc
doesn't support -Wall
and -Werror
at all. With CMake
, these options are lost.
Your suggestion with passing options directly, i.e. target_compile_options(cuda_hello_world PUBLIC -Werror -Wall)
is not correct. By doing so I'm getting the following building error:
nvcc fatal : Value '-Wall' is not defined for option 'Werror'
Hello. Consider the following simple application:
// main.cu
#include "test.h"
int main() {
foo();
return 0;
}
// test.h
auto foo() -> void;
// test.cpp
#include "test.h"
auto foo() -> void {
int a{};
}
Please notice the unused variable a
in the test.cpp
. I want to turn compiler warnings and treat them as errors by providing -Werror
and -Wall
compiler options.
Manually executing the command
nvcc src/main.cu src/test.cpp -g --generate-code arch=compute_61,code=[compute_61,sm_61] -G --compiler-options -Werror,-Wall -std=c++14 --verbose
gives me the desired result - the compilation fails with the message
error: unused variable ‘a’
When I look at the verbose
output I see the following lines:
#$ gcc -std=c++14 -c -x c++ -D__NVCC__ -Werror -Wall "-I/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include" -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=2 -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=2 -m64 -g -gdwarf-2 "src/test.cpp" -o "/tmp/tmpxft_00005a49_00000000-12_test.o"
Please notice the options -Werror
and -Wall
.
Now I want to do the same with the CMake. I added the following CMakeLists.txt
:
cmake_minimum_required(VERSION 3.21 FATAL_ERROR)
project(cuda_hello_world LANGUAGES CXX CUDA)
add_executable(cuda_hello_world src/main.cu src/test.h src/test.cpp)
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
target_compile_options(cuda_hello_world PRIVATE $<$<COMPILE_LANGUAGE:CUDA>:-G>)
endif()
set_target_properties(cuda_hello_world PROPERTIES
CXX_STANDARD 14
CXX_STANDARD_REQUIRED YES
CXX_EXTENSIONS NO
CUDA_STANDARD 14
CUDA_STANDARD_REQUIRED YES
CUDA_EXTENSIONS NO
RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin
CUDA_ARCHITECTURES "61"
)
target_compile_options(cuda_hello_world PUBLIC $<$<COMPILE_LANGUAGE:CUDA>:--compiler-options -Werror,-Wall>)
Executing cmake --build . --verbose
gives me the following lines:
[ 33%] Building CUDA object CMakeFiles/cuda_hello_world.dir/src/main.cu.o
/usr/local/cuda-11.2/bin/nvcc -forward-unknown-to-host-compiler --generate-code=arch=compute_61,code=[compute_61,sm_61] --compiler-options -Werror,-Wall -std=c++14 -MD -MT CMakeFiles/cuda_hello_world.dir/src/main.cu.o -MF CMakeFiles/cuda_hello_world.dir/src/main.cu.o.d -x cu -c /home/nc/code/cuda_hello_world/src/main.cu -o CMakeFiles/cuda_hello_world.dir/src/main.cu.o
[ 66%] Building CXX object CMakeFiles/cuda_hello_world.dir/src/test.cpp.o
/usr/bin/c++ -std=c++14 -MD -MT CMakeFiles/cuda_hello_world.dir/src/test.cpp.o -MF CMakeFiles/cuda_hello_world.dir/src/test.cpp.o.d -o CMakeFiles/cuda_hello_world.dir/src/test.cpp.o -c /home/nc/code/cuda_hello_world/src/test.cpp
Please notice how the options -Werror
and -Wall
are passed to nvcc
to compile main.cu
but missing when compiling test.cpp
. Because of that the code compiles despite the presence of the unused variable.
Linux Ubuntu 18.04 cmake 3.21.1 gcc 7.5.0