CrayPrgEnv.cmake mishandles CMAKE_<LANG>_IMPLICIT_INCLUDE_DIRECTORIES
GCC 6 and newer changed their include files so that if you compile a C++ program with "-isystem /usr/include" it fails with "stdlib.h: No such file or directory"... see:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129 and #16291 (closed)
One of our cmake-based projects is impacted by this on the Cray, so I stated looking at the CrayPrgEnv.cmake to see exactly what it is doing and I found several problems in the cmake code:
-
The function __cray_extract_implicit's first arg is ${src}.
{src} is the name of a source file to do a test compile with (e.g. "cc -v" or "c++ -v"). For CXX the CrayPrgEnv-CXX.cmake sets this to "
{CMAKE_ROOT}/Modules/CMakeCXXCompilerABI.cpp"... Unfortunately, the __cray_extract_implicit function doesn't actually use the ${src} variable anywhere in the body of the function! This results in the output of execute_process() being incomplete (e.g. it does not contain any INCLUDE information). Thus, when you run "cmake" on a Cray with CrayPrgEnv the{CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES} is incorrectly empty. The fix for this is to add "
{src}" after "-o ${BIN}" in the COMMAND part of the call to execute_process(). -
if you manually fix the handling of ${src} in __cray_extract_implicit() as suggested above, you will get a partial (but incomplete!) list of includes in ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES}. This is because __cray_extract_implicit()'s handling is incorrect. The code uses a regular expression in ${compiler_command} to look for a line like "/opt/gcc/./cc1plus" (or "/opt/gcc/./cc1" for C) to extract -I's from the line to generate the implicit include list. Unfortunately, this line does not include important directories like /usr/include in the list. But that information is included later on in the output between the lines "#include <...> search starts here:" and "End of search list." ... the output parsing in CrayPrgEnv.cmake needs to be adjusted to handle this. Here is some sample output from the GNU C compiler on a Cray:
/opt/gcc/7.1.0/snos/libexec/gcc/x86_64-suse-linux/7.1.0/cc1 -quiet -v -I /opt/cray/pe/libsci/17.09.1/GNU/6.1/x86_64/include -I /opt/cray/pe/mpt/7.6.2/gni/mpich-gnu/5.1/include -I /opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/include -I /opt/cray/alps/6.4.1-6.0.4.0_7.2__g86d0f3d.ari/include -I /opt/cray/xpmem/2.2.2-6.0.4.0_3.1__g43b0535.ari/include -I /opt/cray/gni-headers/5.0.11-6.0.4.0_7.2__g7136988.ari/include -I /opt/cray/pe/pmi/5.0.12/include -I /opt/cray/ugni/6.0.14-6.0.4.0_14.1__ge7db4a2.ari/include -I /opt/cray/udreg/2.3.2-6.0.4.0_12.2__g2f9c3ee.ari/include -I /opt/cray/wlm_detect/1.2.1-6.0.4.0_22.1__gd26a3dc.ari/include -I /opt/cray/krca/2.2.2-6.0.4.0_5.2__g4614cf3.ari/include -I /opt/cray-hss-devel/8.0.0/include -D __CRAYXC -D __CRAY_HASWELL -D __CRAYXT_COMPUTE_LINUX_TARGET -D __TARGET_LINUX__ /users/ccranor/tmp/cmake/share/cmake-3.10/Modules/CMakeCCompilerABI.c -quiet -dumpbase CMakeCCompilerABI.c -march=core-avx2 -auxbase CMakeCCompilerABI -version -o /tmp/ccsd57Wh.s
GNU C11 (GCC) version 7.1.0 20170502 (Cray Inc.) (x86_64-suse-linux)
compiled by GNU C version 7.1.0 20170502 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/opt/gcc/7.1.0/snos/lib/gcc/x86_64-suse-linux/7.1.0/../../../../x86_64-suse-linux/include"
#include "..." search starts here:
#include <...> search starts here:
/opt/cray/pe/libsci/17.09.1/GNU/6.1/x86_64/include
/opt/cray/pe/mpt/7.6.2/gni/mpich-gnu/5.1/include
/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/include
/opt/cray/alps/6.4.1-6.0.4.0_7.2__g86d0f3d.ari/include
/opt/cray/xpmem/2.2.2-6.0.4.0_3.1__g43b0535.ari/include
/opt/cray/gni-headers/5.0.11-6.0.4.0_7.2__g7136988.ari/include
/opt/cray/pe/pmi/5.0.12/include
/opt/cray/ugni/6.0.14-6.0.4.0_14.1__ge7db4a2.ari/include
/opt/cray/udreg/2.3.2-6.0.4.0_12.2__g2f9c3ee.ari/include
/opt/cray/wlm_detect/1.2.1-6.0.4.0_22.1__gd26a3dc.ari/include
/opt/cray/krca/2.2.2-6.0.4.0_5.2__g4614cf3.ari/include
/opt/cray-hss-devel/8.0.0/include
/opt/gcc/7.1.0/snos/lib/gcc/x86_64-suse-linux/7.1.0/include
/usr/local/include
/opt/gcc/7.1.0/snos/include
/opt/gcc/7.1.0/snos/lib/gcc/x86_64-suse-linux/7.1.0/include-fixed
/usr/include
End of search list.
GNU C11 (GCC) version 7.1.0 20170502 (Cray Inc.) (x86_64-suse-linux)
compiled by GNU C version 7.1.0 20170502 (Cray Inc.), GMP version 6.0.0, MPFR version 3.1.3, MPC version 1.0.3, isl version isl-0.15-GMP
Note that the last 5 includes (from "/opt/gcc/7.1.0/snos/lib/gcc/x86_64-suse-linux/7.1.0/include" to "/usr/include" are not in the "cc1" output line.
- the CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES variable is not preserved between runs. It isn't a cache variable and it isn't saved in the output directory file: "CMakeFiles/*/CmakeCXXCompiler.cmake" either. On CrayPrgEnv.cmake, this results in the include processing being done the first time you run cmake, but then not on later times (because the CRAY__EXTRACTED_IMPLICIT cache variable is set to 1). Because UnixPaths.cmake init's the value to "/usr/include" if you rerun cmake it gets set to "/usr/include" instead of the correct value. I believe that files like "Modules/CMakeCXXCompiler.cmake.in" should be modified to include CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES in the list of data that they preserve between runs. I tried this, and it works (although it has an extra copy of /usr/include in the path... I think one is from the CrayPrgEnv and the other is the one that UnixPaths appends).
Here is a demo of the problem using a simple CMakeLists.txt on a cray:
cmake_minimum_required (VERSION 3.5)
project (cray-test C CXX)
message ("STATUS: incdir: ${CMAKE_CXX_IMPLICIT_INCLUDE_DIRECTORIES}")
message ("STATUS: lnkdir: ${CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES}")
message ("STATUS: lnklib: ${CMAKE_CXX_IMPLICIT_LINK_LIBRARIES}")
Here's the output from 3.6.2 on a Cray:
% cmake --version
cmake version 3.6.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
%
%
% env CC=cc CXX=CC cmake ..
-- The C compiler identification is Intel 17.0.4.20170411
-- The CXX compiler identification is Intel 17.0.4.20170411
-- Cray Programming Environment 2.5.12 C
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.5.12 CXX
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
STATUS: incdir:
STATUS: lnkdir: /opt/cray/pe/libsci/17.09.1/INTEL/16.0/x86_64/lib;/opt/cray/dmapp/default/lib64;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/lib;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/lib64;/opt/cray/pe/atp/2.1.1/libApp;/opt/intel/2017.4.196/compilers_and_libraries_2017/linux/mkl/lib/intel64;/opt/intel/2017.4.196/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin;/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0;/opt/gcc/6.3.0/snos/lib64;/lib64;/usr/lib64;/opt/gcc/6.3.0/snos/lib;/lib;/usr/lib
STATUS: lnklib: intlc;AtpSigHandler;AtpSigHCommData;rca;sci_intel_mpi;sci_intel;mpich_intel;mpichcxx_intel;stdc++;imf;m;ifcore;ifport;pthread;imf;svml;irng;stdc++;m;ipgo;decimal;cilkrts;stdc++;irc;svml;c;irc_s;dl;c
-- Configuring done
-- Generating done
-- Build files have been written to: /users/ccranor/tmp/cmake4/b
%
notice that "incdir:" is empty the first time you run cmake. Now if you rerun cmake, "incdir:" becomes "/usr/include!
% env CC=cc CXX=CC cmake ..
-- Cray Programming Environment 2.5.12 C
-- Cray Programming Environment 2.5.12 CXX
STATUS: incdir: /usr/include
STATUS: lnkdir: /opt/cray/pe/libsci/17.09.1/INTEL/16.0/x86_64/lib;/opt/cray/dmapp/default/lib64;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/lib;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/lib64;/opt/cray/pe/atp/2.1.1/libApp;/opt/intel/2017.4.196/compilers_and_libraries_2017/linux/mkl/lib/intel64;/opt/intel/2017.4.196/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin;/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0;/opt/gcc/6.3.0/snos/lib64;/lib64;/usr/lib64;/opt/gcc/6.3.0/snos/lib;/lib;/usr/lib
STATUS: lnklib: intlc;AtpSigHandler;AtpSigHCommData;rca;sci_intel_mpi;sci_intel;mpich_intel;mpichcxx_intel;stdc++;imf;m;ifcore;ifport;pthread;imf;svml;irng;stdc++;m;ipgo;decimal;cilkrts;stdc++;irc;svml;c;irc_s;dl;c
-- Configuring done
-- Generating done
-- Build files have been written to: /users/ccranor/tmp/cmake4/b
%
I compiled a more recent version of cmake and fixed the ${src} bug in CrayPrgEnv. Here is output from that:
% rm -rf *
% ~/tmp/cmake/bin/cmake --version
cmake version 3.10.0-rc3
CMake suite maintained and supported by Kitware (kitware.com/cmake).
%
%
% env CC=cc CXX=CC ~/tmp/cmake/bin/cmake ..
-- The C compiler identification is Intel 17.0.4.20170411
-- The CXX compiler identification is Intel 17.0.4.20170411
-- Cray Programming Environment 2.5.12 C
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc
-- Check for working C compiler: /opt/cray/pe/craype/2.5.12/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.5.12 CXX
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC
-- Check for working CXX compiler: /opt/cray/pe/craype/2.5.12/bin/CC -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
STATUS: incdir: /opt/cray/pe/libsci/17.09.1/INTEL/16.0/x86_64/include;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/include;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/include;/opt/cray/alps/6.4.1-6.0.4.0_7.2__g86d0f3d.ari/include;/opt/cray/xpmem/2.2.2-6.0.4.0_3.1__g43b0535.ari/include;/opt/cray/gni-headers/5.0.11-6.0.4.0_7.2__g7136988.ari/include;/opt/cray/pe/pmi/5.0.12/include;/opt/cray/ugni/6.0.14-6.0.4.0_14.1__ge7db4a2.ari/include;/opt/cray/udreg/2.3.2-6.0.4.0_12.2__g2f9c3ee.ari/include;/opt/cray/wlm_detect/1.2.1-6.0.4.0_22.1__gd26a3dc.ari/include;/opt/cray/krca/2.2.2-6.0.4.0_5.2__g4614cf3.ari/include;/opt/cray-hss-devel/8.0.0/include;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/include;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/include;/opt/cray/alps/6.4.1-6.0.4.0_7.2__g86d0f3d.ari/include;/opt/cray/xpmem/2.2.2-6.0.4.0_3.1__g43b0535.ari/include;/opt/cray/gni-headers/5.0.11-6.0.4.0_7.2__g7136988.ari/include;/opt/cray/pe/pmi/5.0.12/include;/opt/cray/ugni/6.0.14-6.0.4.0_14.1__ge7db4a2.ari/include;/opt/cray/udreg/2.3.2-6.0.4.0_12.2__g2f9c3ee.ari/include;/opt/cray/wlm_detect/1.2.1-6.0.4.0_22.1__gd26a3dc.ari/include;/opt/cray/krca/2.2.2-6.0.4.0_5.2__g4614cf3.ari/include;/opt/cray-hss-devel/8.0.0/include
STATUS: lnkdir: /opt/cray/pe/libsci/17.09.1/INTEL/16.0/x86_64/lib;/opt/cray/dmapp/default/lib64;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/lib;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/lib64;/opt/cray/pe/atp/2.1.1/libApp;/opt/intel/2017.4.196/compilers_and_libraries_2017/linux/mkl/lib/intel64;/opt/intel/2017.4.196/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin;/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0;/opt/gcc/6.3.0/snos/lib64;/lib64;/usr/lib64;/opt/gcc/6.3.0/snos/lib;/lib;/usr/lib
STATUS: lnklib: intlc;AtpSigHandler;AtpSigHCommData;rca;sci_intel_mpi;sci_intel;mpich_intel;mpichcxx_intel;stdc++;imf;m;ifcore;ifport;pthread;imf;svml;irng;stdc++;m;ipgo;decimal;cilkrts;stdc++;gcc;gcc_s;irc;svml;c;gcc;gcc_s;irc_s;dl;c
-- Configuring done
-- Generating done
-- Build files have been written to: /users/ccranor/tmp/cmake4/b
%
Now incdir: is partially correct (but doesn't include /usr/include, as I noted above). But if I rerun "cmake" it gets reduced to "/usr/include" because the output isn't saved anywhere like CMakeCXXCompiler.cmake...
% env CC=cc CXX=CC ~/tmp/cmake/bin/cmake ..
-- Cray Programming Environment 2.5.12 C
-- Cray Programming Environment 2.5.12 CXX
STATUS: incdir: /usr/include
STATUS: lnkdir: /opt/cray/pe/libsci/17.09.1/INTEL/16.0/x86_64/lib;/opt/cray/dmapp/default/lib64;/opt/cray/pe/mpt/7.6.2/gni/mpich-intel/16.0/lib;/opt/cray/rca/2.2.11-6.0.4.0_13.2__g84de67a.ari/lib64;/opt/cray/pe/atp/2.1.1/libApp;/opt/intel/2017.4.196/compilers_and_libraries_2017/linux/mkl/lib/intel64;/opt/intel/2017.4.196/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin;/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0;/opt/gcc/6.3.0/snos/lib64;/lib64;/usr/lib64;/opt/gcc/6.3.0/snos/lib;/lib;/usr/lib
STATUS: lnklib: intlc;AtpSigHandler;AtpSigHCommData;rca;sci_intel_mpi;sci_intel;mpich_intel;mpichcxx_intel;stdc++;imf;m;ifcore;ifport;pthread;imf;svml;irng;stdc++;m;ipgo;decimal;cilkrts;stdc++;gcc;gcc_s;irc;svml;c;gcc;gcc_s;irc_s;dl;c
-- Configuring done
-- Generating done
-- Build files have been written to: /users/ccranor/tmp/cmake4/b
%