CUDA not working with IntelLLVM host compiler (2023.01.13.)
I'm running into a very puzzling issue with my builds. I am trying to use CUDA together with Intel's oneAPI compiler in a single project, and I'm just not able to make it work.
Let's take the following trivial project configuration:
# Set up a C++ project.
cmake_minimum_required( VERSION 3.10 )
project( oneAPICUDATest VERSION 0.0.1 LANGUAGES CXX )
# Enable the CUDA language in addition.
enable_language( CUDA )When I try to use it together with a hand compiled version of intel/llvm, things work fine.
[bash][celeborn]:build > source ~/software/intel/clang-2022-09/x86_64-ubuntu2204-gcc12-opt/setup.sh
Configured CUDA from: /home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204
Configured Clang from: /home/krasznaa/software/intel/clang-2022-09/x86_64-ubuntu2204-gcc12-opt
[bash][celeborn]:build > echo $CXX
/home/krasznaa/software/intel/clang-2022-09/x86_64-ubuntu2204-gcc12-opt/bin/clang++
[bash][celeborn]:build > echo $CUDACXX
/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/nvcc
[bash][celeborn]:build > echo $CUDAHOSTCXX
/home/krasznaa/software/intel/clang-2022-09/x86_64-ubuntu2204-gcc12-opt/bin/clang++
[bash][celeborn]:build > echo $CUDAFLAGS
-allow-unsupported-compiler
[bash][celeborn]:build > cmake ../source/
-- The CXX compiler identification is Clang 16.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/krasznaa/software/intel/clang-2022-09/x86_64-ubuntu2204-gcc12-opt/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The CUDA compiler identification is NVIDIA 12.0.76
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/krasznaa/development/cmake/oneapi-cuda/build
[bash][celeborn]:build >As you can see, in this case the compiler is identified as "vanilla clang", and with the help of -allow-unsupported-compiler CUDA accepts it as a valid host compiler. (I am able to build "real" projects like this, this setup does work.)
However when I try to do the same with the official oneAPI compiler, I get the following:
[bash][celeborn]:build > source ~/software/intel/oneapi-2023.0.0/setvars.sh
:: initializing oneAPI environment ...
bash: BASH_VERSION = 5.1.16(1)-release
args: Using "$@" for setvars.sh arguments:
:: advisor -- latest
:: ccl -- latest
:: compiler -- latest
:: debugger -- latest
:: dev-utilities -- latest
:: dpcpp-ct -- latest
:: dpl -- latest
:: ipp -- latest
:: mkl -- latest
:: mpi -- latest
:: tbb -- latest
:: vtune -- latest
:: oneAPI environment initialized ::
[bash][celeborn]:build > source ~/software/cuda/12.0.0/x86_64-ubuntu2204/setup.sh
Configured CUDA from: /home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204
[bash][celeborn]:build > export CXX=`which icpx`
[bash][celeborn]:build > export CUDAHOSTCXX=$CXX
[bash][celeborn]:build > export CUDAFLAGS="-allow-unsupported-compiler"
[bash][celeborn]:build > cmake ../source/
-- The CXX compiler identification is IntelLLVM 2023.0.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/bin/icpx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:726 (message):
Compiling the CUDA compiler identification source file
"CMakeCUDACompilerId.cu" failed.
Compiler: /home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/nvcc
Build flags:
Id flags:
--keep;--keep-dir;tmp;-ccbin=/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/bin/icpx
-v
The output was:
1
#$ _NVVM_BRANCH_=nvvm
#$ _SPACE_=
#$ _CUDART_=cudart
#$ _HERE_=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin
#$ _THERE_=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_DIR_=targets/x86_64-linux
#$ TOP=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/..
#$
NVVMIR_LIBRARY_DIR=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../nvvm/libdevice
#$
LD_LIBRARY_PATH=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../lib:/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/lib64:/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/extras/CUPTI/lib64:/home/krasznaa/software/intel/oneapi-2023.0.0/tbb/2021.8.0/env/../lib/intel64/gcc4.8:/home/krasznaa/software/intel/oneapi-2023.0.0/mpi/2021.8.0//libfabric/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/mpi/2021.8.0//lib/release:/home/krasznaa/software/intel/oneapi-2023.0.0/mpi/2021.8.0//lib:/home/krasznaa/software/intel/oneapi-2023.0.0/mkl/2023.0.0/lib/intel64:/home/krasznaa/software/intel/oneapi-2023.0.0/ipp/2021.7.0/lib/intel64:/home/krasznaa/software/intel/oneapi-2023.0.0/debugger/2023.0.0/gdb/intel64/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/debugger/2023.0.0/libipt/intel64/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/debugger/2023.0.0/dep/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/lib/x64:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/lib/oclfpga/host/linux64/lib:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/compiler/lib/intel64_lin:/home/krasznaa/software/intel/oneapi-2023.0.0/ccl/2021.8.0/lib/cpu_gpu_dpcpp
#$
PATH=/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../nvvm/bin:/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin:/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/vtune/2023.0.0/bin64:/home/krasznaa/software/intel/oneapi-2023.0.0/mpi/2021.8.0//libfabric/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/mpi/2021.8.0//bin:/home/krasznaa/software/intel/oneapi-2023.0.0/mkl/2023.0.0/bin/intel64:/home/krasznaa/software/intel/oneapi-2023.0.0/dpcpp-ct/2023.0.0/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/dev-utilities/2021.8.0/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/debugger/2023.0.0/gdb/intel64/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/lib/oclfpga/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/bin/intel64:/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/bin:/home/krasznaa/software/intel/oneapi-2023.0.0/advisor/2023.0.0/bin64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
#$
INCLUDES="-I/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/include"
#$ LIBRARIES=
"-L/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/lib/stubs"
"-L/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/lib"
#$ CUDAFE_FLAGS=
#$ PTXAS_FLAGS=
#$ rm tmp/a_dlink.reg.c
#$
"/home/krasznaa/software/intel/oneapi-2023.0.0/compiler/2023.0.0/linux/bin"/icpx
-D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__ -D__NVCC__
"-I/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/include"
-D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=0
-D__CUDACC_VER_BUILD__=76 -D__CUDA_API_VER_MAJOR__=12
-D__CUDA_API_VER_MINOR__=0 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
"cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
"tmp/CMakeCUDACompilerId.cpp4.ii"
In file included from <built-in>:1:
In file included from
/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/include/cuda_runtime.h:83:
/home/krasznaa/software/cuda/12.0.0/x86_64-ubuntu2204/bin/../targets/x86_64-linux/include/crt/host_config.h:140:2:
error: -- unsupported clang version! clang version must be less than 15 and
greater than 3.2 . The nvcc flag '-allow-unsupported-compiler' can be used
to override this version check; however, using an unsupported host compiler
may cause compilation failure or incorrect run time execution. Use at your
own risk.
#error -- unsupported clang version! clang version must be less than 15 and
greater than 3.2 . The nvcc flag '-allow-unsupported-compiler' can be used
to override this version check; however, using an unsupported host compiler
may cause compilation failure or incorrect run time execution. Use at your
own risk.
^
1 error generated.
# --error 0x1 --
Call Stack (most recent call first):
/usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
/usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
/usr/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
CMakeLists.txt:7 (enable_language)
-- Configuring incomplete, errors occurred!
See also "/home/krasznaa/development/cmake/oneapi-cuda/build/CMakeFiles/CMakeOutput.log".
See also "/home/krasznaa/development/cmake/oneapi-cuda/build/CMakeFiles/CMakeError.log".
[bash][celeborn]:build >As you can see, in this case the compiler is identified as a "proper Intel compiler". (Which I think plays a role in my problems.) And then, for some reason that I can't figure out, the value of $CUDAFLAGS just does not make it into CMAKE_CUDA_FLAGS.
It does not seem to be a problem with my compiler/environment setup. As in the previous environment I am able to build test .cu files by hand. (As long as I provide -allow-unsupported-compiler by hand to it.)
Do you have any ideas how/why IntelLLVM.cmake (and/or related files) would so destructively interfere with the CUDA setup?
Cheers, Attila