FindPython: Inconsistent Python_SOABI when cross-compiling
Brief description
Depending on what COMPONENTS
are passed to the find_package(Python3 ...)
call, Python3_SOABI
is either set to the empty string, or set to the correct value for the target.
Would it be possible to make this behavior more consistent so it is always set for the target's SOABI?
Locating Python when cross-compiling
In some projects, it is necessary to have both a Python interpreter and the Python development targets, even when cross-compiling.
For example, the Python interpreter could be necessary to perform a code generation step during the build, and later the development targets could be needed to build an extension module for the target.
Although tricky, this is possible, and I usually achieve it by explicitly setting the Python3_ROOT_DIR
and Python3_EXECUTABLE
hints/artifacts. The executable is the one on the host, so that it can be used during the build, and the root directory points to a staging area containing a cross-compiled Python installation (with a python3-config
script that is correctly picked up by CMake).
Python3_SOABI
The problem with When invoking find_package(Python3 COMPONENTS Development)
, the python3-config
script is used to set the Python3_SOABI
variable appropriately (for the target).
However, when also searching for the Python interpreter in the same call, using find_package(Python3 COMPONENTS Interpreter Development)
, the Python3_SOABI
variable is always set to the empty string when cross-compiling.
This can be demonstrated by the experiment at the bottom of this post.
Causes and possible fixes
The root cause is the combination of three things:
-
https://gitlab.kitware.com/cmake/cmake/-/blob/eb9195e9ef6fd299b25d44b966129c587ec46e2d/Modules/FindPython/Support.cmake#L606-609
As a special case forSOABI
,SOSABI
andABIFLAGS
, the__python_get_config_var
function always sets the result variable, even if determining the SOABI failed (i.e. when the local_values
variable is unset). -
https://gitlab.kitware.com/cmake/cmake/-/blob/eb9195e9ef6fd299b25d44b966129c587ec46e2d/Modules/FindPython/Support.cmake#L2216
This means that if__python_get_config_var
was unable to determine the SOABI (e.g. because we are cross-compiling), the value ofPython3_SOABI
is set to the empty string. -
https://gitlab.kitware.com/cmake/cmake/-/blob/eb9195e9ef6fd299b25d44b966129c587ec46e2d/Modules/FindPython/Support.cmake#L3680-3682
When looking for theLIBRARY
component, the assignment ofPython3_SOABI
is guarded by a checkif (NOT DEFINED ${_PYTHON_PREFIX}_SOABI)
.
However, we know that${_PYTHON_PREFIX}_SOABI
is defined unconditionally at line 2216 (2.), so this code is never executed if the Python interpreter is also requested in the same call.
In the case of two separate calls to find_package
, Python3_SOABI
is set correctly, because ${_PYTHON_PREFIX}_SOABI)
is always unset()
first
(https://gitlab.kitware.com/cmake/cmake/-/blob/eb9195e9ef6fd299b25d44b966129c587ec46e2d/Modules/FindPython/Support.cmake#L1419), so the branch on line 3681 is taken.
There are at least two possible solutions here:
- Update
__python_get_config_var
to not set the SOABI result variable in the case of failure. - Remove the
if (NOT DEFINED ${_PYTHON_PREFIX}_SOABI)
guard when looking for the library.
History
The special case for SOABI
that sets the result variable to the empty string in the case of failure dates back to this commit: 951640f1
However, the same commit did not remove the if (NOT DEFINED ${_PYTHON_PREFIX}_SOABI)
check further down: 951640f1
Was this intentional? Am I missing something? Was/is the check needed to get the desired behavior, or is it just for performance reasons, to avoid calculating the SOABI twice?
Background
I maintain the py-build-cmake build backend, which I use to cross-compile Python extension modules that are built using CMake.
My goal is to make the interactions between py-build-cmake and CMake itself more robust.
Thanks!
Pieter
Experiment
Click to expand
toolchain.cmake
# System information
set(CMAKE_SYSTEM_NAME "Linux")
set(CMAKE_SYSTEM_PROCESSOR "aarch64")
set(CROSS_GNU_TRIPLE "aarch64-rpi3-linux-gnu" CACHE STRING "The GNU triple of the toolchain to use")
set(CMAKE_LIBRARY_ARCHITECTURE aarch64-linux-gnu CACHE STRING "")
# Search path configuration
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
# Toolchain and sysroot
set(TOOLCHAIN_DIR "${CMAKE_CURRENT_LIST_DIR}/x-tools/${CROSS_GNU_TRIPLE}")
set(CMAKE_SYSROOT "${TOOLCHAIN_DIR}/${CROSS_GNU_TRIPLE}/sysroot")
set(CMAKE_C_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-gcc" CACHE FILEPATH "C compiler")
set(CMAKE_CXX_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-g++" CACHE FILEPATH "C++ compiler")
set(CMAKE_Fortran_COMPILER "${TOOLCHAIN_DIR}/bin/${CROSS_GNU_TRIPLE}-gfortran" CACHE FILEPATH "Fortran compiler")
# Locating Python
cmake_policy(SET CMP0094 NEW)
if (NOT DEFINED Python3_EXECUTABLE)
message(STATUS "Looking for a Python executable")
find_program(Python3_EXECUTABLE "python3.11")
endif()
set(CROSS_PYTHON_ROOT "${CMAKE_CURRENT_LIST_DIR}/${CROSS_GNU_TRIPLE}/python3.11")
list(APPEND CMAKE_FIND_ROOT_PATH "${CROSS_PYTHON_ROOT}")
set(Python3_ROOT_DIR "${CROSS_PYTHON_ROOT}/usr/local")
CMakeLists.txt
cmake_minimum_required(VERSION 3.27.1)
project(cmake-python-cross)
set(EXPERIMENT 1 CACHE STRING "")
set(CMAKE_FIND_DEBUG_MODE Off)
if (EXPERIMENT EQUAL 0)
message(STATUS "Only Python3::Interpreter")
find_package(Python3 COMPONENTS Interpreter)
elseif (EXPERIMENT EQUAL 1)
message(STATUS "Only Python3::Development")
find_package(Python3 COMPONENTS Development)
elseif (EXPERIMENT EQUAL 2)
message(STATUS "Python3::{Interpreter,Development}")
find_package(Python3 COMPONENTS Interpreter Development)
elseif (EXPERIMENT EQUAL 3)
message(STATUS "First Python3::Interpreter, then Python3::Development")
find_package(Python3 COMPONENTS Interpreter)
find_package(Python3 COMPONENTS Development)
elseif (EXPERIMENT EQUAL 4)
message(STATUS "First Python3::Development, then Python3::Interpreter")
find_package(Python3 COMPONENTS Development)
find_package(Python3 COMPONENTS Interpreter)
endif()
set(CMAKE_FIND_DEBUG_MODE Off)
message(STATUS "(CMakeLists) ROOT_DIR: '${Python3_ROOT_DIR}'")
message(STATUS "(CMakeLists) EXECUTABLE: '${Python3_EXECUTABLE}'")
message(STATUS "(CMakeLists) CONFIG: '${_Python3_CONFIG}'")
message(STATUS "(CMakeLists) INCLUDE_DIRS: '${Python3_INCLUDE_DIRS}'")
message(STATUS "(CMakeLists) LIBRARIES: '${Python3_LIBRARIES}'")
message(STATUS "(CMakeLists) SOABI: '${Python3_SOABI}'")
if (DEFINED Python3_SOABI)
message(STATUS "(CMakeLists) SOABI is defined")
else()
message(STATUS "(CMakeLists) SOABI is NOT defined")
endif()
Command line
rm build/CMakeCache.txt && cmake -Bbuild -S. --toolchain=toolchain.cmake -DEXPERIMENT=2
Results
- Only
Interpreter
:(CMakeLists) ROOT_DIR: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local' (CMakeLists) EXECUTABLE: '~/.local/bin/python3.11' (CMakeLists) CONFIG: '' (CMakeLists) INCLUDE_DIRS: '' (CMakeLists) LIBRARIES: '' (CMakeLists) SOABI: '' (CMakeLists) SOABI is defined
- Only
Development
:(CMakeLists) ROOT_DIR: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local' (CMakeLists) EXECUTABLE: '~/.local/bin/python3.11' (CMakeLists) CONFIG: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/bin/python3.11-config' (CMakeLists) INCLUDE_DIRS: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/include/python3.11' (CMakeLists) LIBRARIES: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/lib/libpython3.11.so' (CMakeLists) SOABI: 'cpython-311-aarch64-linux-gnu' (CMakeLists) SOABI is defined
-
Interpreter
andDevelopment
in a single call:(CMakeLists) ROOT_DIR: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local' (CMakeLists) EXECUTABLE: '~/.local/bin/python3.11' (CMakeLists) CONFIG: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/bin/python3.11-config' (CMakeLists) INCLUDE_DIRS: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/include/python3.11' (CMakeLists) LIBRARIES: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/lib/libpython3.11.so' (CMakeLists) SOABI: '' (CMakeLists) SOABI is defined
-
Interpreter
andDevelopment
in two separate calls:(CMakeLists) ROOT_DIR: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local' (CMakeLists) EXECUTABLE: '~/.local/bin/python3.11' (CMakeLists) CONFIG: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/bin/python3.11-config' (CMakeLists) INCLUDE_DIRS: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/include/python3.11' (CMakeLists) LIBRARIES: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/lib/libpython3.11.so' (CMakeLists) SOABI: 'cpython-311-aarch64-linux-gnu' (CMakeLists) SOABI is defined
-
Development
andInterpreter
in two separate calls:(CMakeLists) ROOT_DIR: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local' (CMakeLists) EXECUTABLE: '~/.local/bin/python3.11' (CMakeLists) CONFIG: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/bin/python3.11-config' (CMakeLists) INCLUDE_DIRS: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/include/python3.11' (CMakeLists) LIBRARIES: '~/check-python-order/aarch64-rpi3-linux-gnu/python3.11/usr/local/lib/libpython3.11.so' (CMakeLists) SOABI: 'cpython-311-aarch64-linux-gnu' (CMakeLists) SOABI is defined