CUDA/Makefiles: CMake 3.20.0 detects dependencies without -ccbin=
I'm a maintainer on a project called LightGBM (https://github.com/microsoft/LightGBM). That project is a C++ library with many wrappers / extensions, and CMake is our build tool of choice.
After the release of CMake
3.20.0, some builds for a version of LightGBM that includes compilation of CUDA kernels started failing with confusing errors. A build where clang
was identified failed with an error about gcc
.
gcc: error: unrecognized command line option '-fopenmp=libomp'
Short Description
To test compiling CUDA-enabled LightGBM on Linux, we install clang
and set the following environment variables.
export CXX=clang++
export CC=clang
The logs below show that CMake found clang
as the preferred C/C++ compiler.
-- The C compiler identification is Clang 3.8.0
-- The CXX compiler identification is Clang 3.8.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The CUDA compiler identification is NVIDIA 9.0.176
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found OpenMP_C: -fopenmp=libomp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp=libomp (found version "4.0")
-- Found OpenMP: TRUE (found version "4.0")
...
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "9.0")
-- CMAKE_CUDA_FLAGS: -Xcompiler=-fopenmp=libomp -Xcompiler=-fPIC -Xcompiler=-Wall -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70 -O3 -lineinfo
But then we see the following gcc error during compilation.
With clang
3.8.0 and CUDA 9.0:
[ 10%] Building CUDA object CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
gcc: error: unrecognized command line option '-fopenmp=libomp'
With clang
6.0.0 and CUDA 10.0:
[ 10%] Building CUDA object CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
gcc: error: unrecognized command line option '-fopenmp=libomp'; did you mean '-fopenmp-simd'?
Downgrading to cmake
3.19.5-0 solves the problem, for all versions of CUDA and clang
that we have tested.
Reproducible example
I tried to create a slimmer version of LightGBM with just enough code to reproduce this error. I'm not that experienced with C++ though, so I apologize if this is not clear.
You can run the examples below on any system with Docker to simulate the issue. The script test-cuda.sh
just installs some packages and then compiles the library with something like mkdir build && cmake .. && make -j4 _lightgbm
.
You can see the contents at https://github.com/jameslamb/LightGBM/blob/cuda-cmake-repro/test-cuda.sh.
git clone --recursive --branch cuda-cmake-repro git@github.com:jameslamb/LightGBM.git
cd LightGBM
# this will compile successfully
docker run \
-v $(pwd):/opt/test \
-w /opt/test \
--env DEBIAN_FRONTEND=noninteractive \
-t nvcr.io/nvidia/cuda:11.2.2-devel \
./test-cuda.sh
# this will fail with the error mentioned above
docker run \
-v $(pwd):/opt/test \
-w /opt/test \
--env DEBIAN_FRONTEND=noninteractive \
-t nvcr.io/nvidia/cuda:10.0-devel \
./test-cuda.sh
# this will fail with the error mentioned above
docker run \
-v $(pwd):/opt/test \
-w /opt/test \
--env DEBIAN_FRONTEND=noninteractive \
-t nvcr.io/nvidia/cuda:9.0-devel \
./test-cuda.sh
full logs from a failed run (click me)
Preparing to unpack .../cmake-data_3.20.0-0kitware1_all.deb ...
Unpacking cmake-data (3.20.0-0kitware1) ...
Selecting previously unselected package cmake.
Preparing to unpack .../cmake_3.20.0-0kitware1_amd64.deb ...
Unpacking cmake (3.20.0-0kitware1) ...
Setting up cmake-data (3.20.0-0kitware1) ...
Setting up cmake (3.20.0-0kitware1) ...
cmake version 3.20.0
CMake suite maintained and supported by Kitware (kitware.com/cmake).
-- The C compiler identification is Clang 3.8.0
-- The CXX compiler identification is Clang 3.8.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The CUDA compiler identification is NVIDIA 9.0.176
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found OpenMP_C: -fopenmp=libomp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp=libomp (found version "4.0")
-- Found OpenMP: TRUE (found version "4.0")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable version "9.0", minimum required is "9.0")
-- CMAKE_CUDA_FLAGS: -Xcompiler=-fopenmp=libomp -Xcompiler=-fPIC -Xcompiler=-Wall -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_70,code=compute_70 -O3 -lineinfo
-- ALLFEATS_DEFINES: -DPOWER_FEATURE_WORKGROUPS=12;-DUSE_CONSTANT_BUF=0;-DENABLE_ALL_FEATURES
-- FULLDATA_DEFINES: -DPOWER_FEATURE_WORKGROUPS=12;-DUSE_CONSTANT_BUF=0;-DENABLE_ALL_FEATURES;-DIGNORE_INDICES
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256_sp".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256-allfeats_sp".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256-allfeats_sp_const".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256-fulldata_sp_const".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256_sp_const".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "histo_16_64_256-fulldata_sp".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "_lightgbm".
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) in CMakeLists.txt:
Policy CMP0104 is not set: CMAKE_CUDA_ARCHITECTURES now detected for NVCC,
empty CUDA_ARCHITECTURES not allowed. Run "cmake --help-policy CMP0104"
for policy details. Use the cmake_policy command to set the policy and
suppress this warning.
CUDA_ARCHITECTURES is empty for target "_lightgbm".
This warning is for project developers. Use -Wno-dev to suppress it.
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:
USE_CUDA
-- Build files have been written to: /opt/test/build
[ 2%] Building CUDA object CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
[ 7%] Building CUDA object CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
[ 7%] Building CUDA object CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
[ 10%] Building CUDA object CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o
gcc: error: unrecognized command line option '-fopenmp=libomp'
CMakeFiles/histo_16_64_256_sp.dir/build.make:75: recipe for target 'CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o' failed
make[3]: *** [CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o] Error 1
make[3]: *** Deleting file 'CMakeFiles/histo_16_64_256_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o'
CMakeFiles/Makefile2:94: recipe for target 'CMakeFiles/histo_16_64_256_sp.dir/all' failed
make[2]: *** [CMakeFiles/histo_16_64_256_sp.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
gcc: error: unrecognized command line option '-fopenmp=libomp'
CMakeFiles/histo_16_64_256_sp_const.dir/build.make:75: recipe for target 'CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o' failed
make[3]: *** [CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o] Error 1
make[3]: *** Deleting file 'CMakeFiles/histo_16_64_256_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o'
CMakeFiles/Makefile2:198: recipe for target 'CMakeFiles/histo_16_64_256_sp_const.dir/all' failed
make[2]: *** [CMakeFiles/histo_16_64_256_sp_const.dir/all] Error 2
gcc: error: unrecognized command line option '-fopenmp=libomp'
CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/build.make:75: recipe for target 'CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o' failed
make[3]: *** [CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o] Error 1
make[3]: *** Deleting file 'CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/src/treelearner/kernels/histogram_16_64_256.cu.o'
CMakeFiles/Makefile2:172: recipe for target 'CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/all' failed
make[2]: *** [CMakeFiles/histo_16_64_256-fulldata_sp_const.dir/all] Error 2
gcc: error: unrecognized command line option '-fopenmp=libomp'
CMakeFiles/histo_16_64_256-fulldata_sp.dir/build.make:75: recipe for target 'CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o' failed
make[3]: *** [CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o] Error 1
make[3]: *** Deleting file 'CMakeFiles/histo_16_64_256-fulldata_sp.dir/src/treelearner/kernels/histogram_16_64_256.cu.o'
CMakeFiles/Makefile2:224: recipe for target 'CMakeFiles/histo_16_64_256-fulldata_sp.dir/all' failed
make[2]: *** [CMakeFiles/histo_16_64_256-fulldata_sp.dir/all] Error 2
CMakeFiles/Makefile2:260: recipe for target 'CMakeFiles/_lightgbm.dir/rule' failed
make[1]: *** [CMakeFiles/_lightgbm.dir/rule] Error 2
Makefile:247: recipe for target '_lightgbm' failed
make: *** [_lightgbm] Error 2
I've captured the versions of relevant software in each of these images below. Please let me know if there are additional versions you'd like me to report.
cuda-9.0 | cuda-10.0 | cuda-11.2.2 | |
---|---|---|---|
clang | 3.8.0-2 | 6.0.0 | 10.0.0 |
gcc | 5.4.0 | 7.5.0 | 9.3.0 |
ld | 2.26.1 | 2.30 | 2.34 |
nvcc | 9.0.176 | 10.0.130 | 11.2.152 |
result | FAILURE | FAILURE | SUCCESS |
Thank you very much for your time and consideration.