FindMPI: Interrogate Cray compiler
Cray traditionally does not ship mpicc/mpic++ compiler wrappers. Due to that, the traditional CMake logic to find MPI is to only support MPI on Cray systems if either
- Cray Wrappers are used for compiling
- an auxiliary MPI executable is found (usually only present when deviating from the system-recommended MPI implementation)
On many new Exascale prototypes, using the Cray wrappers for develop work is sub-optimal. Vendors such as AMD and integrator seem not to go in lock-step in provided modules and programming envs [1], making offloading+MPI too challenging. Also, frequent vendor releases mean that we need to move fast with new compilers from AMD as we approach Exascale deliverables.
For example, AMD simply recommends to use their clang++
/clang
with CMake [2].
They work great. But matching them to the Cray compiler wrappers is out of
control for the user, so there is no generic CMake way to get HIP and MPI compiled
into our applications on a Cray machine [1].
This PR extends the fallback MPI search logic by interrogating also
the CC
/cc
/ftn
binaries if found for their MPI flags. This does not
change existing logic, i.e. using the Cray wrappers directly is still possible
and preferred once they become stable, but
provides an additional way to get MPI programs compiled on Cray.
[1] https://docs.olcf.ornl.gov/systems/spock_quick_start_guide.html#compilers
[2] https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake
Notes
Please feel free to ping HPE/Cray or OLCF/FCOE people on this PR :-) Improvements & generalizations welcome.
I am just a scientific application developer and Exascale machines in DOE are HPE/Cray machines. With that background, I might lack deep insights on HPE/Cray flags and system details. The logic I am proposing here is how we (ECP AMReX/WarpX) also compile with manually written Makefiles.
Tested on OLCF Spock.
Current work-arounds
[2] Typical work-arounds look like this on an HPE/Cray machine for AMD HIP compilation:
export CC=$(which clang) # from AMD/ROCm
export CXX=$(which clang++) # from AMD/ROCm
export LDFLAGS="-L${CRAYLIBS_X86_64} $(CC --cray-print-opts=libs) -lmpi"
# GPU aware MPI: ${PE_MPICH_GTL_DIR_gfx908} -lmpi_gtl_hsa
cmake -S . -B build -DMPI_CXX_COMPILER=$(which CC) -DMPI_C_COMPILER=$(which cc) -DMPI_COMPILER_FLAGS="--cray-print-opts=all"
This PR removes the need for these side-injected options & flags.
Fixes: #22368 (closed)
Topic-rename: FindMPI-cray-interrogate