MPIEXEC_MAX_NUMPROCS being set to 1 on ORNL Summit
Hello,
I am running into an issue building a package that does some conditional compilation based on the value of MPIEXEC_MAX_NUMPROCS on the ORNL Summit machine. MPIEXEC_MAX_NUMPROCS is always detected to be 1, despite the login nodes having many cores.
I believe the reason for this is that /proc/cpuinfo only contains logical core entries on the login node, and not physical core/socket information. I see the 'processor' attribute for each logical core, but the 'physical id' or 'cpu cores' attribute are not present in /proc/cpuinfo. Looking at SystemInformation.cxx, it looks like the number of physical cores being accurate depends on 'physical id' and 'cpu cores' being available:
#ifdef __linux
// Count sockets.
std::set<int> PhysicalIDs;
std::string idc = this->ExtractValueFromCpuInfoFile(buffer, "physical id");
while (this->CurrentPositionInFile != std::string::npos) {
int id = atoi(idc.c_str());
PhysicalIDs.insert(id);
idc = this->ExtractValueFromCpuInfoFile(buffer, "physical id",
this->CurrentPositionInFile + 1);
}
uint64_t NumberOfSockets = PhysicalIDs.size();
NumberOfSockets = std::max(NumberOfSockets, (uint64_t)1);
// Physical ids returned by Linux don't distinguish cores.
// We want to record the total number of cores in this->NumberOfPhysicalCPU
// (checking only the first proc)
std::string Cores = this->ExtractValueFromCpuInfoFile(buffer, "cpu cores");
unsigned int NumberOfCoresPerSocket = (unsigned int)atoi(Cores.c_str());
NumberOfCoresPerSocket = std::max(NumberOfCoresPerSocket, 1u);
this->NumberOfPhysicalCPU =
NumberOfCoresPerSocket * (unsigned int)NumberOfSockets;
Further, looking at FindMPI.cmake, it looks like MPIEXEC_MAX_NUMPROCS is set to the number of physical cores:
# Set the number of processes to the physical processor count
cmake_host_system_information(RESULT _MPIEXEC_NUMPROCS QUERY NUMBER_OF_PHYSICAL_CORES)
set(MPIEXEC_MAX_NUMPROCS "${_MPIEXEC_NUMPROCS}" CACHE STRING "Maximum number of processors available to run MPI applications.")
unset(_MPIEXEC_NUMPROCS)
mark_as_advanced(MPIEXEC_EXECUTABLE MPIEXEC_NUMPROC_FLAG MPIEXEC_PREFLAGS MPIEXEC_POSTFLAGS MPIEXEC_MAX_NUMPROCS)
It looks like MPIEXEC_MAX_NUMPROCS used to be based on logical core count, but was changed in !1405 (merged). I'm not sure why the socket info is not in /proc/cpuinfo. I suspect it's a PowerPC thing, but I'm not at all sure.