Problems with Python library target (new FindPython module) when the library is statically linked into the interpreter
Have been testing the new FindPython
modules, and the Python::Python
target (or Python3::Python
) for the Python Library (from the current master branch, as these are still unreleased).
In a scenario where one wants to build Python modules compiled from C++ code, it is typical for the compiled module to link agains the Python Library. Currently can think of two ways:
-
Python3_add_library
, defined inFindPython3.cmake
, createsMODULE
targets that are linked against the Python library target -
swig_add_library
, defined inUseSWIG.cmake
(I'm using the current one inmaster
branch), will typically require linking against the Python library if the target language is Python. This would have to be done explicitly viatarget_link_libraries
.
However, loading these modules from Python is likely to crash if the Python interpreter is linked statically against the Python library. When the modules are loaded by the interpreter, the shared python library will also be loaded, causing it to crash (as the symbols are already provided by the executable itself at that time).
The solution in this case is to drop the python library from the linking statement, and tell the linker to ignore missing symbols:
-undefined dynamic_lookup
A more in-depth explanation can be found here: https://groups.google.com/a/continuum.io/forum/#!topic/anaconda/057P4uNWyCU
With the current Python3::Python
target, one would have to go from this:
target_link_libraries(MyModule PRIVATE Python3::Python)
to something like this:
target_include_directories(MyModule PRIVATE ${Python3_INCLUDE_DIRS})
set_target_properties(MyModule PROPERTIES LINK_FLAGS "-undefined dynamic_lookup")
^ in order to get a module to load correctly in an interpreter that statically links the library/
Now my knowledge of Python module creation is limited, but it appears that one can query the interpreter to find out which compile or link flags one must use when creating modules (see discussion above): e.g.
$ python3 -c 'import sysconfig; print(sysconfig.get_config_var("LDSHARED"))'
I wonder whether the correct approach would be for the FindPython
scripts to query that value and add the correct flags to the import target, however I see a few potential problems with this approach:
- Some of the flags returned may conflict with other CMake ones (e.g., some Python distributions will add optimisation flags to that variable).
- For a python module, I believe that dropping the library from the linker statement and adding the
-undefined dynamic_lookup
flag should technically be fine both when the interpreter is linked statically against the library and when it's not (because at runtime the library would be loaded by the interpreter anyway). - However it would be confusing for the
Python3::Python
target not to actually fill out theIMPORTED_LOCATION*
target property, as there may be other use cases where one might want to link against it regardless of what the interpreter does?
Hope some python expert can shed more light into this :)