README.rst

Wrapper Generator
==================

This repository contains the first set of Python code which could be used to
automatically generate pybind11 code from C++ files with proper annotation.

**CAUTION**: The content, structure, and API of this repository are still under
development.  No guarantees of backwards compatibility are given for the
functionality.

License
+++++++

This repository is distributed under the OSI-approved BSD 3-clause License.
See Copyright.txt_ for details.

Program execution
+++++++++++++++++

Prerequisites
---------------

All instructions below assume that a Python3 environment is available.

CMake
#####

The CMake-based portion of the tool requires CMake 3.12 or greater.
Download or build CMake: https://cmake.org/

CastXML
#######

A version of the CastXML tool is required for this program to run.  It, likely, can
be downloaded from your Linux package management system. For Ubuntu, the command looks
like this::

  sudo apt install castxml

If it is not available, a selection of binary distributions can be found on the
`data.kitware.com`_ website.

Python Libraries
################

The repository contains a ``setup.py`` file which can be provided to the
PIP program to install all required Python Libraries.  Execute::

    pip install -e .

to install the this library and other required programs.
It is **strongly** recommended to use this through a virtual enviroment - see
below.

Running Tests
#############

Here's a small recipe for setting up local dependencies and building and
running the tests:

.. code-block:: shell

    # In project root
    python3 -m virtualenv -p python3 ./venv/
    source ./venv/bin/activate
    pip install -e .
    mkdir build && cd build
    cmake ..
    make -j
    ctest -V -R

Preparing C++ code
------------------

The setup for the C++ code to be wrapped is simple. Each C++ directory with content
to be wrapped should contain a file named ``wrapper_input.yml``.
This content should describe the classes, "free" functions
(functions that are not a part of a class), and non-typed enumerations.

This information follows a certain structure.  This structure has three reserved key
tags: ``classes``, ``functions``, and ``enums``.  Any tag not found within one of these
structures, is considered a namespace for the code to be found under::

  <namespace>:
    classes:
      <class_name>:
        <class_data>,
      <class_2_name>:
        <class_2_data>
    functions:
      <function_name>:
        <function_data>
    <namespace_2>:
      enums:

Each type of object within the reserved tags requires different pieces of information:

    +-----------------------------+---------------------------------+
    |           C++ Object        |      Required YAML pieces       |
    +=============================+=================================+
    |           Class             | file:  <str>                    |
    |                             |   Path to file with class       |
    |                             +---------------------------------+
    |                             | inst: []                        |
    |                             |   List of instantiation types   |
    |                             |   for class.                    |
    |                             |   Leave blank for non-templated |
    +-----------------------------+---------------------------------+
    |        Function             | file:  <str>                    |
    |                             |   Path to file with function    |
    |                             +---------------------------------+
    |                             | is_template: <bool>             |
    |                             |  "true" if function is template |
    |                             |  "false" otherwise              |
    |                             +---------------------------------+
    |                             | inst: []                        |
    |                             |   List of instantiation types   |
    |                             |   for function, if is_template  |
    |                             |   is true                       |
    +-----------------------------+---------------------------------+
    |        Enumeration          | file:  <str>                    |
    |                             |   Path to file with enumeration |
    +-----------------------------+---------------------------------+

For an example of a correctly written object, see `wrapper_input.yml`_ in the ``example`` directory.

Execution
---------

via CMake
##########

The recommended path for the usage of this tool is via CMake.

Setup
%%%%%

By utilizing the ``find_package`` utility, the created macros and functions can be
introduced into the environment.  This repository contains the
``AutoPyBind11Config.cmake`` file which is the targe of the following command::

  find_package(AutoPyBind11)

After this command is found, one additional command will be used to acquire pybind11 via the
``FetchContent`` module::

    autopybind11_fetch_build_pybind11()

Add Library
%%%%%%%%%%%

To add a library for the written pybind11 code, we use the ``autopybind11_add_module``
function.  This function does the work of setting custom commands to generate
the pybind11 code and adds a library to CMake which contains the pybind11 code., and then
This command should be written once for each ``wrapper_input.yml`` file in the repository.
The command has the following structure::

    autopybind11_add_module(<name> YAML_INPUT <path_to>/wrapper_input.yml
                        DESTINATION <path_to_output>
                        LINK_LIBRARIES <library_1> <library_2>
                        [CONFIG_INPUT <path_to>/config.yml])

Only the ``CONFIG_INPUT`` flag is optional.  The file given to that argument can be used
to customize the templates used to create the pybind11 CPP code.  The default templates
can be found in `text_blocks.py`_
An example of the function invocation is here::

    autopybind11_add_module("example" YAML_INPUT ${CMAKE_CURRENT_SOURCE_DIR}/wrapper_input.yml
                            DESTINATION ${CMAKE_CURRENT_BINARY_DIR}
                            LINK_LIBRARIES wrapper_example)

via Python
##########
This script assumes that Python 3.* is used.
The Python script has help for the arguments::

    $ python3 autopybind11.py -h
    usage: autopybind11.py [-h] [-o OUTPUT_DIR] -y YAML_PATH --module_name
                           MODULE_NAME [-g CASTXML_PATH] [-cg CONFIG_DIR]
                           [--no-generation] [-rs RSP_PATH] [-pm]
                           [--common_cpp_body_fmt COMMON_CPP_BODY_FMT]
                           [--class_info_body_fmt CLASS_INFO_BODY_FMT]
                           [--init_fun_signature_fmt INIT_FUN_SIGNATURE_FMT]
                           [--init_fun_forward_fmt INIT_FUN_FORWARD_FMT]
                           [--cppbody_fmt CPPBODY_FMT]
                           [--module_cpp_fmt MODULE_CPP_FMT]
                           [--member_func_fmt MEMBER_FUNC_FMT]
                           [--constructor_fmt CONSTRUCTOR_FMT]
                           [--member_func_arg_fmt MEMBER_FUNC_ARG_FMT]
                           [--public_member_var_fmt PUBLIC_MEMBER_VAR_FMT]
                           [--private_member_var_fmt PRIVATE_MEMBER_VAR_FMT]
                           [--member_reference_fmt MEMBER_REFERENCE_FMT]
                           [--overload_template_fmt OVERLOAD_TEMPLATE_FMT]
                           [--wrap_header_fmt WRAP_HEADER_FMT]
                           [--operator_fmt OPERATOR_FMT]
                           [--call_operator_fmt CALL_OPERATOR_FMT]
                           [--enum_header_fmt ENUM_HEADER_FMT]
                           [--enum_val_fmt ENUM_VAL_FMT]

    Args that start with '--' (eg. -o) can also be set in a config file (specified
    via -cg). The config file uses YAML syntax and must represent a YAML 'mapping'
    (for details, see http://learn.getgrav.org/advanced/yaml). If an arg is
    specified in more than one place, then commandline values override config file
    values which override defaults.

    optional arguments:
      -h, --help            show this help message and exit
      -o OUTPUT_DIR, --output OUTPUT_DIR
      -y YAML_PATH, --input_yaml YAML_PATH
                            Path to input YAML file of objects to process
      --module_name MODULE_NAME
                            Desired name of the output pybind11 module
      -g CASTXML_PATH, --castxml-path CASTXML_PATH
                            Path to castxml
      -cg CONFIG_DIR, --config-path CONFIG_DIR
                            config file path
      --no-generation, -n   Only print name of files to be generated
      -rs RSP_PATH, --input_response RSP_PATH
      -pm, --private_members
      --common_cpp_body_fmt COMMON_CPP_BODY_FMT
      --class_info_body_fmt CLASS_INFO_BODY_FMT
      --init_fun_signature_fmt INIT_FUN_SIGNATURE_FMT
      --init_fun_forward_fmt INIT_FUN_FORWARD_FMT
      --cppbody_fmt CPPBODY_FMT
      --module_cpp_fmt MODULE_CPP_FMT
      --member_func_fmt MEMBER_FUNC_FMT
      --constructor_fmt CONSTRUCTOR_FMT
      --member_func_arg_fmt MEMBER_FUNC_ARG_FMT
      --public_member_var_fmt PUBLIC_MEMBER_VAR_FMT
      --private_member_var_fmt PRIVATE_MEMBER_VAR_FMT
      --member_reference_fmt MEMBER_REFERENCE_FMT
      --overload_template_fmt OVERLOAD_TEMPLATE_FMT
      --wrap_header_fmt WRAP_HEADER_FMT
      --operator_fmt OPERATOR_FMT
      --call_operator_fmt CALL_OPERATOR_FMT
      --enum_header_fmt ENUM_HEADER_FMT
      --enum_val_fmt ENUM_VAL_FMT

This is useful for some small testing. But since a Python call would be needed to wrap each input file, we recommend
using the CMake system above.

Style Checking / Linting
-------------------------

This repository uses the pycodestyle_ tool for linting and style checking which
can be installed via the ``pip`` program.
The program should be run at the top level so that the ``setup.cfg`` in the
repository is available. An example run is as follows::

  $ pycodestyle autopybind11/

or setup a CMake build system and execute the ``lint_lib`` test::

  $ ctest -R lint_lib -VV

Alternatives Comparison
-----------------------

*Note*: Some mentions here are purely observational, and do not yet have
explicit comparisons against ``autopybind11``.

CLIF
####

https://github.com/google/clif

Philosophically in some ways similar to hooking up CastXML (they use clang /
llvm as their C++ parser), a code generator (like this project), and a runtime
library for interfacing (like pybind11). This means mean most of the
glue that would need to be written is already there. It was originally written
as a general IDL (Interface Definition Language), which is great for
multi-language support (e.g. Python, MATLAB, etc), but will suffer when there
are language-specific foreign functional interfaces (FFIs).

However, it is an incomplete and currently dead project (from a public point of
view). Also, it has no Numpy and Eigen support.  It had a high profile
developer in Google and there were once some mentions of CLIF in the Bazel
codebase.

This project may possibly resume at some point
(`drake#7889 <https://github.com/RobotLocomotion/drake/issues/7889#issuecomment-634260874>`_).

Shibboken2
##########

* https://doc.qt.io/qtforpython/shiboken2/shibokengenerator.html
* https://doc.qt.io/qtforpython/shiboken2/considerations.html#general

This provides both the CPython interface and C++ API scanning (similar to
PyBindGen, CLIF, etc.), and is used in PySide, but is sufficiently general to
use in other projects.

The documentation is extremely comprehensive, and provides detailed
explanations of each step in the process, especially in sections where
customizations are necessary.

One drawback is that debugging will involve both generated CPython code as well
as the binding analysis itself, similar as with CLIF. (It's similar to
debugging ``pybind11``, but in ``pybind11`` the main hoop to jump through is
C++ template metaprogramming).

pybind11
########

.. note: This should be moved to the top.

https://pybind11.readthedocs.io/en/stable/

This is the "spiritual successor" to Boost.Python, and is a very popular Python
binding library.

Binding definitions are written in pure C++ code, which allows more familiarity
with developers, and slightly easier debugging (vs. interfaces like SWIG, where
you must debug both the binding generation *and* the runtime). pybind11 not
only provides a mechanism for exposing C++ to Python, but also provides its
Python C++ interface for exposing Python to C++.

pybind11 (but not Boost.Python) includes extensive builtin Numpy support with
C++ Eigen interface, ranging from interfacing with ``Eigen::Map<>`` to handling
sparse and dense arrays.

Some features are missing and have been added via patches with varying success
(in the author's opinion, the codebase may be difficult to understand).

pybind11 upstream is generally tentative about taking on major changes that
only benefit a few downstream projects, and at times can have a slow response
rate.

At present, there is an increasing trend of responses on the maintainer's part.

Several open source projects are already working on generating pybind11
bindings.

.. note::

  As mentioned above, this project emits pybind11 C++ code, and is meant to
  help minimize the minutae of writing high-quality bindings with pybind11.

SWIG
####

http://www.swig.org/

Multi-language support would be an advantage.  Projects in the open source
community already have code that generates SWIG bindings,
but the authors have stated that if they were starting now, they would use
pybind11.

The SWIG intermediate language can be seen as a boon when the interfaces are
simple, but a bottleneck when more complexity is necessary -- a standard issue
with IDLs aiming towards providing FFIs.

The developers of `Drake`_ previously used SWIG, but then transitioned to
pybind11 due to its extensive NumPy and Eigen support, and the ability to
(relatively easily) fork the code to provide support for dynamic scalar types.

.. _`Drake`: https://drake.mit.edu/

CPPWG
#####

https://github.com/jmsgrogan/cppwg/

CPPWG was the first tool used in the attempt to automate Python bindings in
Drake (`drake#7889 <https://github.com/RobotLocomotion/drake/issues/7889>`_).

This tool was the inspiration for the structure of the AutoPyBind11 system. It
uses
CastXML with pygccxml and nested configuration files to denote what classes and
free functions needed to be wrapped by the tool.  It also had a configurable set
of pybind11 output, but with much less customization opportunity as the
customization was limited to a few specific places in the code as opposed to
the structure of the pybind11 code which AutoPyBind11 allows.

The tool’s major shortcoming was the method’s handling of templated classes.
It used string matching while parsing the C++ source file to determine if
any of the lines found in the object required templating.  Additionally,
a test case was created which found an issue with the usage of templates
of wrapped code using a single instantiation type.

Binder
######

https://github.com/RosettaCommons/binder and https://cppbinder.readthedocs.io/en/latest/

Binder is another tool that performs the automatic wrapping of C++ code into
Python via pybind11. It uses C++ as the language of the tool, as opposed to
AutoPyBind11 which uses Python.  The Binder tool also uses a configuration file.
The system uses a “+/-” for inclusion and omission of the three basic objects:
namespaces, classes, and functions.  Binder’s configuration file can also alter
the functions used to bind the code.  The user can specify a function which is
run in place of the default binding code.

genpybind
#########

https://github.com/kljohann/genpybind

This uses the both ``clang`` C++ AST API and the ``clang.cindex`` Python AST
API to generate bindings ``pybind11``, similar to ``binder`` mentioned above.
However, the primary philosophy here is to annotate any additional information
(e.g. return value policies, renames, etc.) using explicit compiler attributes
using macros.

More information on why both the C++ and Python APIs are used are detailed
here: https://github.com/kljohann/genpybind#implementation

Interesting, the author would prefer to make a single C++ tool rather than a
single Python tool ;)

AutoWIG
#######

https://github.com/StatisKit/AutoWIG |
`Paper <https://arxiv.org/abs/1705.11000>`_ |
`Docs <https://autowig.readthedocs.io/en/latest/index.html>`_ |
`Example Code <https://github.com/StatisKit/FP17>`_

Provides a means to parse C++ code and emit either Boost.Python or pybind11
code, using ``clang.cindex`` in Python (Python bindings of C ``libclang`` API).
It can also translate docstrings from Doxygen to Sphinx, including symbol
references.

It also provides a comprehensive class structure with different passes, which
look great for generalization, but may also cause developers some pain with
indirection via abstraction.

It's an impressive project, and the paper (cited above) has some interesting
comparisons (including a comparison of methods for VTK and ITK). However,
the current development is a bit unclear. The documentation seems most
comprehensive in the arXiv publication, but appears lacking in the ReadTheDocs
website.

PyBindGen
#########

https://pybindgen.readthedocs.io/en/latest/ and https://github.com/gjcarneiro/pybindgen

PyBindGen is a Python-only module that allows the specification of C++ modules
to wrap into custom Python runtime (not pybind11 code!).

It doesn’t rely on a configuration file to determine what parts of the C++ code
are wrapped but instead uses a Python script to
create the modules and add all objects.  The compilation of the resultant code
is done via a “python setup.py” command and is what creates the C++ code that
calls into the CPython API.

It does have some pygccxml integration.  One mode of execution uses pygccxml to
parse C++ code and write the pybind11 code into a separate file.  The other
mode parses the header files but writes out the information into a PyBindGen
script.

Most notably, because this code generates its own bindings, any runtime
debugging must done jointly with the code generation, which would incur similar
overhead like debugging errors with SWIG.

cppyy
#####

* https://cppyy.readthedocs.io/en/latest/index.html
* https://bitbucket.org/wlav/cppyy/src/master/
* https://bitbucket.org/wlav/cppyy-backend/src/master/clingwrapper/
* https://bitbucket.org/wlav/cppyy-backend/src/master/cling/

  - https://bitbucket.org/wlav/cppyy-backend/src/master/cling/python/cppyy_backend/

Generates Python bindings of C++ code on the fly using Cling (which leverages
the clang frontend and LLVM backend).

The PyHPC paper in the above mentioned links cite `pygccxml` and `Reflex` as
motivating examples. The goal is to provide (interactive) bindings for C++
projects with APIs the size of ROOT. (Additionally, the paper has normalize
benchmarking, which is an excellent goal in any project!)

For an overview of the package structure, see
`the docs <https://cppyy.readthedocs.io/en/latest/packages.html#package-structure>`_.
Note that the ``cppyy_backend``
Python module actually accesses ``clang.cindex`` to provide the
``cppyy-generator`` tool used in its workflow:
`cppyy: Utilities <https://cppyy.readthedocs.io/en/latest/utilities.html#bindings-collection>`_.

Due to usage of Cling and the philosophy of lazy binding, user experience may
be heavily impacted at load time and when heavy new instantiations are
necessary (similar to complaintgs about Julia startup time).

However, these should generally only apply to projects with template-heavy
public APIs (e.g. those that use Eigen). Performance could possible be tuned.
For more details, see this GitHub discussion:
`drake#7889 (comment)
<https://github.com/RobotLocomotion/drake/issues/7889#issuecomment-663721506>`_.

Other Mentions
##############

* https://github.com/robotpy/robotpy-build

  - Only for RobotPy
  - Emits pybind11 code
  - Uses `header2whatever <https://pypi.org/project/header2whatever/>`_

.. _pycodestyle: https://pypi.org/project/pycodestyle/
.. _Copyright.txt: Copyright.txt
.. _`data.kitware.com`: https://data.kitware.com/#folder/57b5de948d777f10f2696370
.. _`text_blocks.py`: text_blocks.py
.. _`wrapper_input.yml`: example/wrapper_input.yml