FindPython PyPy
The SOABI in PyPy is wrong, and it was fixed three weeks ago. However, packaging
relies on the wrong value of SOABI, so fixing this breaks packaging
, and therefore pip
, so the change had to be reverted a couple of days ago.
The correct fix IMO would be for CMake to follow CPython and PyPy's developers recommendation and compute SOABI from EXT_SUFFIX
(just remove the suffix) for PyPy and CPython 3.8+. The value of SOABI from PyPy is broken and won't be able to be fixed for a while without breaking compatibly with existing Pip's, due to it incorrectly using it in computing wheel tags (but not for CPython, which it special cases). I don't think an extra variable needs to be added (#23651 (closed)), and the Python_add_library( WITH_SOABI
usage needs to be fixed. The value of SOABI is sometimes empty on Windows as well, apparently. Getting SOABI
from EXT_SUFFIX
will stabilize this value while users work happens on PyPy and packaging to correct this issue. By PEP 3149, SOABI
must be the extensionless part of EXT_SUFFIX
; if they differ, that's a bug. So, IMO, changing this will only affect buggy implementations (PyPy). :) It could be protected by a policy, as long as I have some way to activate the non-buggy behavior from scikit-build-core via a define in the vendored "backport" copy used if CMake is 3.15-3.25.
For future reference going forward,
sysconfig.get_config_var('EXT_SUFFIX')
is the recommended way to spell this for CPython3.8+ and all current versions of PyPy. See CPython issues https://bugs.python.org/issue39825 (for windows) and https://bugs.python.org/issue42604 (for FreeBSD and AIX).
I don't think this would even be a compatibly problem, because as it stands, you cannot make a module that is loadable from PyPy with FindPython. You have to manually change the extension (which is what pybind11 has been doing since adding support for FindPython in 2020). This is magnified by the fact that PyPy can't load an extension without a suffix, to protect itself from loading CPython extensions. Historically, getting the SOABI from EXT_SUFFIX is safest, since that is what is used elsewhere. SO was broken for a little while in PyPy IIRC (could be mis-remembering), and SOABI is still broken. This is affecting scikit-build-core
, because it is supposed to use official FindPython instead of custom scripts, and this bug breaks PyPy support.
See https://github.com/pybind/pybind11/pull/2462, https://github.com/scikit-build/scikit-build-core/pull/33#issuecomment-1295734678, https://foss.heptapod.net/pypy/pypy/-/issues/3816, and https://github.com/pypa/packaging/issues/606. Follow up to #21070 (closed) (which I forgot about).
This is a simple reproducer:
cmake_minimum_required(VERSION 3.24)
project(simplest LANGUAGES C)
find_package(Python COMPONENTS COMPONENTS Interpreter Development.Module)
Python_add_library(_module MODULE src/module.c WITH_SOABI)
install(TARGETS _module DESTINATION .)
#define PY_SSIZE_T_CLEAN
#include <Python.h>
float square(float x) { return x * x; }
static PyObject *square_wrapper(PyObject *self, PyObject *args) {
float input, result;
if (!PyArg_ParseTuple(args, "f", &input)) {
return NULL;
}
result = square(input);
return PyFloat_FromDouble(result);
}
static PyMethodDef pysimple_methods[] = {
{"square", square_wrapper, METH_VARARGS, "Square function"},
{NULL, NULL, 0, NULL}};
static struct PyModuleDef pysimple_module = {PyModuleDef_HEAD_INIT, "pysimple",
NULL, -1, pysimple_methods};
PyMODINIT_FUNC PyInit__module(void) {
return PyModule_Create(&pysimple_module);
}
The extension can't be loaded in PyPy.
This would have been fixed by !7395 (closed), though that adds a duplicate piece of information that can't be removed later (once bugs are fixed and PEP 3149 is followed). Having Python_EXT_SUFFIX
could still be useful, I supposed, though WITH_SOABI
and WITH_EXT_SUFFIX
would be truly identical once bugs are fixed.
Any library that wants to support PyPy has to do something like this (not mine): https://github.com/maxbachmann/RapidFuzz/blob/8fd63c2be1d930ef6bedd8c8bc27afabd9094d30/CMakeLists.txt#L32-L49