Skip to content

FindMPI: Interrogate Cray compiler

Axel Huebl requested to merge ax3l/cmake:cray-findMPI-interrogate into master

Cray traditionally does not ship mpicc/mpic++ compiler wrappers. Due to that, the traditional CMake logic to find MPI is to only support MPI on Cray systems if either

  1. Cray Wrappers are used for compiling
  2. an auxiliary MPI executable is found (usually only present when deviating from the system-recommended MPI implementation)

On many new Exascale prototypes, using the Cray wrappers for develop work is sub-optimal. Vendors such as AMD and integrator seem not to go in lock-step in provided modules and programming envs [1], making offloading+MPI too challenging. Also, frequent vendor releases mean that we need to move fast with new compilers from AMD as we approach Exascale deliverables.

For example, AMD simply recommends to use their clang++/clang with CMake [2]. They work great. But matching them to the Cray compiler wrappers is out of control for the user, so there is no generic CMake way to get HIP and MPI compiled into our applications on a Cray machine [1].

This PR extends the fallback MPI search logic by interrogating also the CC/cc/ftn binaries if found for their MPI flags. This does not change existing logic, i.e. using the Cray wrappers directly is still possible and preferred once they become stable, but provides an additional way to get MPI programs compiled on Cray.

[1] https://docs.olcf.ornl.gov/systems/spock_quick_start_guide.html#compilers
[2] https://rocmdocs.amd.com/en/latest/Installation_Guide/Using-CMake-with-AMD-ROCm.html#using-hip-in-cmake

Notes

Please feel free to ping HPE/Cray or OLCF/FCOE people on this PR :-) Improvements & generalizations welcome.

I am just a scientific application developer and Exascale machines in DOE are HPE/Cray machines. With that background, I might lack deep insights on HPE/Cray flags and system details. The logic I am proposing here is how we (ECP AMReX/WarpX) also compile with manually written Makefiles.

Tested on OLCF Spock.

Current work-arounds

[2] Typical work-arounds look like this on an HPE/Cray machine for AMD HIP compilation:

export CC=$(which clang)     # from AMD/ROCm
export CXX=$(which clang++)  # from AMD/ROCm
export LDFLAGS="-L${CRAYLIBS_X86_64} $(CC --cray-print-opts=libs) -lmpi"
# GPU aware MPI: ${PE_MPICH_GTL_DIR_gfx908} -lmpi_gtl_hsa

cmake -S . -B build -DMPI_CXX_COMPILER=$(which CC) -DMPI_C_COMPILER=$(which cc) -DMPI_COMPILER_FLAGS="--cray-print-opts=all"

This PR removes the need for these side-injected options & flags.

Fixes: #22368 (closed)
Topic-rename: FindMPI-cray-interrogate

Edited by Brad King

Merge request reports