README.rst 20 KB
Newer Older
Joe Snyder's avatar
Joe Snyder committed
1
2
3
4
Wrapper Generator
==================

This repository contains the first set of Python code which could be used to
5
automatically generate pybind11 code from C++ files with proper annotation.
Joe Snyder's avatar
Joe Snyder committed
6

Joe Snyder's avatar
Joe Snyder committed
7
8
9
10
**CAUTION**: The content, structure, and API of this repository are still under
development.  No guarantees of backwards compatibility are given for the
functionality.

Joe Snyder's avatar
Joe Snyder committed
11
12
13
14
15
16
License
+++++++

This repository is distributed under the OSI-approved BSD 3-clause License.
See Copyright.txt_ for details.

17
18
19
Program execution
+++++++++++++++++

Joe Snyder's avatar
Joe Snyder committed
20
Prerequisites
Joe Snyder's avatar
Joe Snyder committed
21
22
---------------

Joe Snyder's avatar
Joe Snyder committed
23
24
25
26
27
28
29
30
31
32
33
34
35
36
All instructions below assume that a Python3 environment is available.

CMake
#####

The CMake-based portion of the tool requires CMake 3.12 or greater.
Download or build CMake: https://cmake.org/

CastXML
#######

A version of the CastXML tool is required for this program to run.  It, likely, can
be downloaded from your Linux package management system. For Ubuntu, the command looks
like this::
Joe Snyder's avatar
Joe Snyder committed
37

38
39
  sudo apt install castxml

Joe Snyder's avatar
Joe Snyder committed
40
41
42
43
44
45
If it is not available, a selection of binary distributions can be found on the
`data.kitware.com`_ website.

Python Libraries
################

46
The repository contains a ``setup.py`` file which can be provided to the
Joe Snyder's avatar
Joe Snyder committed
47
48
PIP program to install all required Python Libraries.  Execute::

49
    pip install -e .
Joe Snyder's avatar
Joe Snyder committed
50

51
52
53
to install the this library and other required programs.
It is **strongly** recommended to use this through a virtual enviroment - see
below.
54
55
56
57
58
59
60
61
62
63
64
65

Running Tests
#############

Here's a small recipe for setting up local dependencies and building and
running the tests:

.. code-block:: shell

    # In project root
    python3 -m virtualenv -p python3 ./venv/
    source ./venv/bin/activate
66
    pip install -e .
67
68
69
70
    mkdir build && cd build
    cmake ..
    make -j
    ctest -V -R
Joe Snyder's avatar
Joe Snyder committed
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128

Preparing C++ code
------------------

The setup for the C++ code to be wrapped is simple. Each C++ directory with content
to be wrapped should contain a file named ``wrapper_input.yml``.
This content should describe the classes, "free" functions
(functions that are not a part of a class), and non-typed enumerations.

This information follows a certain structure.  This structure has three reserved key
tags: ``classes``, ``functions``, and ``enums``.  Any tag not found within one of these
structures, is considered a namespace for the code to be found under::

  <namespace>:
    classes:
      <class_name>:
        <class_data>,
      <class_2_name>:
        <class_2_data>
    functions:
      <function_name>:
        <function_data>
    <namespace_2>:
      enums:

Each type of object within the reserved tags requires different pieces of information:

    +-----------------------------+---------------------------------+
    |           C++ Object        |      Required YAML pieces       |
    +=============================+=================================+
    |           Class             | file:  <str>                    |
    |                             |   Path to file with class       |
    |                             +---------------------------------+
    |                             | inst: []                        |
    |                             |   List of instantiation types   |
    |                             |   for class.                    |
    |                             |   Leave blank for non-templated |
    +-----------------------------+---------------------------------+
    |        Function             | file:  <str>                    |
    |                             |   Path to file with function    |
    |                             +---------------------------------+
    |                             | is_template: <bool>             |
    |                             |  "true" if function is template |
    |                             |  "false" otherwise              |
    |                             +---------------------------------+
    |                             | inst: []                        |
    |                             |   List of instantiation types   |
    |                             |   for function, if is_template  |
    |                             |   is true                       |
    +-----------------------------+---------------------------------+
    |        Enumeration          | file:  <str>                    |
    |                             |   Path to file with enumeration |
    +-----------------------------+---------------------------------+

For an example of a correctly written object, see `wrapper_input.yml`_ in the ``example`` directory.

Execution
---------
129

Joe Snyder's avatar
Joe Snyder committed
130
131
via CMake
##########
132

Joe Snyder's avatar
Joe Snyder committed
133
The recommended path for the usage of this tool is via CMake.
134

Joe Snyder's avatar
Joe Snyder committed
135
136
Setup
%%%%%
Joe Snyder's avatar
Joe Snyder committed
137

Joe Snyder's avatar
Joe Snyder committed
138
139
140
By utilizing the ``find_package`` utility, the created macros and functions can be
introduced into the environment.  This repository contains the
``AutoPyBind11Config.cmake`` file which is the targe of the following command::
Joe Snyder's avatar
Joe Snyder committed
141

Joe Snyder's avatar
Joe Snyder committed
142
  find_package(AutoPyBind11)
Joe Snyder's avatar
Joe Snyder committed
143

144
After this command is found, one additional command will be used to acquire pybind11 via the
Joe Snyder's avatar
Joe Snyder committed
145
146
147
``FetchContent`` module::

    autopybind11_fetch_build_pybind11()
Joe Snyder's avatar
Joe Snyder committed
148

Joe Snyder's avatar
Joe Snyder committed
149
150
Add Library
%%%%%%%%%%%
Joe Snyder's avatar
Joe Snyder committed
151

152
To add a library for the written pybind11 code, we use the ``autopybind11_add_module``
Joe Snyder's avatar
Joe Snyder committed
153
function.  This function does the work of setting custom commands to generate
154
the pybind11 code and adds a library to CMake which contains the pybind11 code., and then
Joe Snyder's avatar
Joe Snyder committed
155
156
This command should be written once for each ``wrapper_input.yml`` file in the repository.
The command has the following structure::
Joe Snyder's avatar
Joe Snyder committed
157

Joe Snyder's avatar
Joe Snyder committed
158
159
160
161
162
163
    autopybind11_add_module(<name> YAML_INPUT <path_to>/wrapper_input.yml
                        DESTINATION <path_to_output>
                        LINK_LIBRARIES <library_1> <library_2>
                        [CONFIG_INPUT <path_to>/config.yml])

Only the ``CONFIG_INPUT`` flag is optional.  The file given to that argument can be used
164
to customize the templates used to create the pybind11 CPP code.  The default templates
Joe Snyder's avatar
Joe Snyder committed
165
166
167
168
169
170
171
172
173
can be found in `text_blocks.py`_
An example of the function invocation is here::

    autopybind11_add_module("example" YAML_INPUT ${CMAKE_CURRENT_SOURCE_DIR}/wrapper_input.yml
                            DESTINATION ${CMAKE_CURRENT_BINARY_DIR}
                            LINK_LIBRARIES wrapper_example)

via Python
##########
Joe Snyder's avatar
Joe Snyder committed
174
175
176
This script assumes that Python 3.* is used.
The Python script has help for the arguments::

Joe Snyder's avatar
Joe Snyder committed
177
    $ python3 autopybind11.py -h
Joe Snyder's avatar
Joe Snyder committed
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
    usage: autopybind11.py [-h] [-o OUTPUT_DIR] -y YAML_PATH --module_name
                           MODULE_NAME [-g CASTXML_PATH] [-cg CONFIG_DIR]
                           [--no-generation] [-rs RSP_PATH] [-pm]
                           [--common_cpp_body_fmt COMMON_CPP_BODY_FMT]
                           [--class_info_body_fmt CLASS_INFO_BODY_FMT]
                           [--init_fun_signature_fmt INIT_FUN_SIGNATURE_FMT]
                           [--init_fun_forward_fmt INIT_FUN_FORWARD_FMT]
                           [--cppbody_fmt CPPBODY_FMT]
                           [--module_cpp_fmt MODULE_CPP_FMT]
                           [--member_func_fmt MEMBER_FUNC_FMT]
                           [--constructor_fmt CONSTRUCTOR_FMT]
                           [--member_func_arg_fmt MEMBER_FUNC_ARG_FMT]
                           [--public_member_var_fmt PUBLIC_MEMBER_VAR_FMT]
                           [--private_member_var_fmt PRIVATE_MEMBER_VAR_FMT]
                           [--member_reference_fmt MEMBER_REFERENCE_FMT]
                           [--overload_template_fmt OVERLOAD_TEMPLATE_FMT]
                           [--wrap_header_fmt WRAP_HEADER_FMT]
                           [--operator_fmt OPERATOR_FMT]
                           [--call_operator_fmt CALL_OPERATOR_FMT]
                           [--enum_header_fmt ENUM_HEADER_FMT]
                           [--enum_val_fmt ENUM_VAL_FMT]

    Args that start with '--' (eg. -o) can also be set in a config file (specified
    via -cg). The config file uses YAML syntax and must represent a YAML 'mapping'
    (for details, see http://learn.getgrav.org/advanced/yaml). If an arg is
    specified in more than one place, then commandline values override config file
    values which override defaults.
Joe Snyder's avatar
Joe Snyder committed
205
206
207

    optional arguments:
      -h, --help            show this help message and exit
Joe Snyder's avatar
Joe Snyder committed
208
      -o OUTPUT_DIR, --output OUTPUT_DIR
tao558's avatar
tao558 committed
209
      -y YAML_PATH, --input_yaml YAML_PATH
Joe Snyder's avatar
Joe Snyder committed
210
211
                            Path to input YAML file of objects to process
      --module_name MODULE_NAME
212
                            Desired name of the output pybind11 module
Joe Snyder's avatar
Joe Snyder committed
213
214
      -g CASTXML_PATH, --castxml-path CASTXML_PATH
                            Path to castxml
Tom Osika's avatar
Tom Osika committed
215
      -cg CONFIG_DIR, --config-path CONFIG_DIR
Tom Osika's avatar
Tom Osika committed
216
                            config file path
Joe Snyder's avatar
Joe Snyder committed
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
      --no-generation, -n   Only print name of files to be generated
      -rs RSP_PATH, --input_response RSP_PATH
      -pm, --private_members
      --common_cpp_body_fmt COMMON_CPP_BODY_FMT
      --class_info_body_fmt CLASS_INFO_BODY_FMT
      --init_fun_signature_fmt INIT_FUN_SIGNATURE_FMT
      --init_fun_forward_fmt INIT_FUN_FORWARD_FMT
      --cppbody_fmt CPPBODY_FMT
      --module_cpp_fmt MODULE_CPP_FMT
      --member_func_fmt MEMBER_FUNC_FMT
      --constructor_fmt CONSTRUCTOR_FMT
      --member_func_arg_fmt MEMBER_FUNC_ARG_FMT
      --public_member_var_fmt PUBLIC_MEMBER_VAR_FMT
      --private_member_var_fmt PRIVATE_MEMBER_VAR_FMT
      --member_reference_fmt MEMBER_REFERENCE_FMT
      --overload_template_fmt OVERLOAD_TEMPLATE_FMT
      --wrap_header_fmt WRAP_HEADER_FMT
      --operator_fmt OPERATOR_FMT
      --call_operator_fmt CALL_OPERATOR_FMT
      --enum_header_fmt ENUM_HEADER_FMT
      --enum_val_fmt ENUM_VAL_FMT

This is useful for some small testing. But since a Python call would be needed to wrap each input file, we recommend
using the CMake system above.
Joe Snyder's avatar
Joe Snyder committed
241

242
243
244
245
246
247
248
249
Style Checking / Linting
-------------------------

This repository uses the pycodestyle_ tool for linting and style checking which
can be installed via the ``pip`` program.
The program should be run at the top level so that the ``setup.cfg`` in the
repository is available. An example run is as follows::

250
  $ pycodestyle autopybind11/
251

252
or setup a CMake build system and execute the ``lint_lib`` test::
Joe Snyder's avatar
Joe Snyder committed
253

254
  $ ctest -R lint_lib -VV
Joe Snyder's avatar
Joe Snyder committed
255

256
Alternatives Comparison
257
258
-----------------------

259
260
*Note*: Some mentions here are purely observational, and do not yet have
explicit comparisons against ``autopybind11``.
261
262
263
264
265
266

CLIF
####

https://github.com/google/clif

267
268
269
270
271
272
273
274
275
276
277
278
Philosophically in some ways similar to hooking up CastXML (they use clang /
llvm as their C++ parser), a code generator (like this project), and a runtime
library for interfacing (like pybind11). This means mean most of the
glue that would need to be written is already there. It was originally written
as a general IDL (Interface Definition Language), which is great for
multi-language support (e.g. Python, MATLAB, etc), but will suffer when there
are language-specific foreign functional interfaces (FFIs).

However, it is an incomplete and currently dead project (from a public point of
view). Also, it has no Numpy and Eigen support.  It had a high profile
developer in Google and there were once some mentions of CLIF in the Bazel
codebase.
279

280
281
282
This project may possibly resume at some point
(`drake#7889 <https://github.com/RobotLocomotion/drake/issues/7889#issuecomment-634260874>`_).

283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
Shibboken2
##########

* https://doc.qt.io/qtforpython/shiboken2/shibokengenerator.html
* https://doc.qt.io/qtforpython/shiboken2/considerations.html#general

This provides both the CPython interface and C++ API scanning (similar to
PyBindGen, CLIF, etc.), and is used in PySide, but is sufficiently general to
use in other projects.

The documentation is extremely comprehensive, and provides detailed
explanations of each step in the process, especially in sections where
customizations are necessary.

One drawback is that debugging will involve both generated CPython code as well
as the binding analysis itself, similar as with CLIF. (It's similar to
debugging ``pybind11``, but in ``pybind11`` the main hoop to jump through is
C++ template metaprogramming).

302
pybind11
303
304
########

305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
.. note: This should be moved to the top.

https://pybind11.readthedocs.io/en/stable/

This is the "spiritual successor" to Boost.Python, and is a very popular Python
binding library.

Binding definitions are written in pure C++ code, which allows more familiarity
with developers, and slightly easier debugging (vs. interfaces like SWIG, where
you must debug both the binding generation *and* the runtime). pybind11 not
only provides a mechanism for exposing C++ to Python, but also provides its
Python C++ interface for exposing Python to C++.

pybind11 (but not Boost.Python) includes extensive builtin Numpy support with
C++ Eigen interface, ranging from interfacing with ``Eigen::Map<>`` to handling
sparse and dense arrays.

Some features are missing and have been added via patches with varying success
(in the author's opinion, the codebase may be difficult to understand).

pybind11 upstream is generally tentative about taking on major changes that
only benefit a few downstream projects, and at times can have a slow response
rate.

At present, there is an increasing trend of responses on the maintainer's part.

Several open source projects are already working on generating pybind11
332
333
bindings.

334
335
336
337
338
.. note::

  As mentioned above, this project emits pybind11 C++ code, and is meant to
  help minimize the minutae of writing high-quality bindings with pybind11.

339
340
341
342
343
344
345
346
SWIG
####

http://www.swig.org/

Multi-language support would be an advantage.  Projects in the open source
community already have code that generates SWIG bindings,
but the authors have stated that if they were starting now, they would use
347
348
349
350
351
352
353
354
355
356
357
pybind11.

The SWIG intermediate language can be seen as a boon when the interfaces are
simple, but a bottleneck when more complexity is necessary -- a standard issue
with IDLs aiming towards providing FFIs.

The developers of `Drake`_ previously used SWIG, but then transitioned to
pybind11 due to its extensive NumPy and Eigen support, and the ability to
(relatively easily) fork the code to provide support for dynamic scalar types.

.. _`Drake`: https://drake.mit.edu/
358
359
360
361
362
363

CPPWG
#####

https://github.com/jmsgrogan/cppwg/

364
365
366
367
368
CPPWG was the first tool used in the attempt to automate Python bindings in
Drake (`drake#7889 <https://github.com/RobotLocomotion/drake/issues/7889>`_).

This tool was the inspiration for the structure of the AutoPyBind11 system. It
uses
369
370
CastXML with pygccxml and nested configuration files to denote what classes and
free functions needed to be wrapped by the tool.  It also had a configurable set
371
of pybind11 output, but with much less customization opportunity as the
372
customization was limited to a few specific places in the code as opposed to
373
374
the structure of the pybind11 code which AutoPyBind11 allows.

375
376
377
378
379
380
381
382
383
384
385
386
The tool’s major shortcoming was the method’s handling of templated classes.
It used string matching while parsing the C++ source file to determine if
any of the lines found in the object required templating.  Additionally,
a test case was created which found an issue with the usage of templates
of wrapped code using a single instantiation type.

Binder
######

https://github.com/RosettaCommons/binder and https://cppbinder.readthedocs.io/en/latest/

Binder is another tool that performs the automatic wrapping of C++ code into
387
Python via pybind11. It uses C++ as the language of the tool, as opposed to
388
389
390
391
392
393
AutoPyBind11 which uses Python.  The Binder tool also uses a configuration file.
The system uses a “+/-” for inclusion and omission of the three basic objects:
namespaces, classes, and functions.  Binder’s configuration file can also alter
the functions used to bind the code.  The user can specify a function which is
run in place of the default binding code.

394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
genpybind
#########

https://github.com/kljohann/genpybind

This uses the both ``clang`` C++ AST API and the ``clang.cindex`` Python AST
API to generate bindings ``pybind11``, similar to ``binder`` mentioned above.
However, the primary philosophy here is to annotate any additional information
(e.g. return value policies, renames, etc.) using explicit compiler attributes
using macros.

More information on why both the C++ and Python APIs are used are detailed
here: https://github.com/kljohann/genpybind#implementation

Interesting, the author would prefer to make a single C++ tool rather than a
single Python tool ;)

411
412
413
414
415
416
417
418
419
AutoWIG
#######

https://github.com/StatisKit/AutoWIG |
`Paper <https://arxiv.org/abs/1705.11000>`_ |
`Docs <https://autowig.readthedocs.io/en/latest/index.html>`_ |
`Example Code <https://github.com/StatisKit/FP17>`_

Provides a means to parse C++ code and emit either Boost.Python or pybind11
420
421
422
code, using ``clang.cindex`` in Python (Python bindings of C ``libclang`` API).
It can also translate docstrings from Doxygen to Sphinx, including symbol
references.
423
424
425
426
427
428
429
430
431
432
433

It also provides a comprehensive class structure with different passes, which
look great for generalization, but may also cause developers some pain with
indirection via abstraction.

It's an impressive project, and the paper (cited above) has some interesting
comparisons (including a comparison of methods for VTK and ITK). However,
the current development is a bit unclear. The documentation seems most
comprehensive in the arXiv publication, but appears lacking in the ReadTheDocs
website.

434
435
436
437
438
439
PyBindGen
#########

https://pybindgen.readthedocs.io/en/latest/ and https://github.com/gjcarneiro/pybindgen

PyBindGen is a Python-only module that allows the specification of C++ modules
440
441
442
443
to wrap into custom Python runtime (not pybind11 code!).

It doesn’t rely on a configuration file to determine what parts of the C++ code
are wrapped but instead uses a Python script to
444
create the modules and add all objects.  The compilation of the resultant code
445
446
447
is done via a “python setup.py” command and is what creates the C++ code that
calls into the CPython API.

448
It does have some pygccxml integration.  One mode of execution uses pygccxml to
449
parse C++ code and write the pybind11 code into a separate file.  The other
450
451
452
mode parses the header files but writes out the information into a PyBindGen
script.

453
454
455
456
Most notably, because this code generates its own bindings, any runtime
debugging must done jointly with the code generation, which would incur similar
overhead like debugging errors with SWIG.

457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
cppyy
#####

* https://cppyy.readthedocs.io/en/latest/index.html
* https://bitbucket.org/wlav/cppyy/src/master/
* https://bitbucket.org/wlav/cppyy-backend/src/master/clingwrapper/
* https://bitbucket.org/wlav/cppyy-backend/src/master/cling/

  - https://bitbucket.org/wlav/cppyy-backend/src/master/cling/python/cppyy_backend/

Generates Python bindings of C++ code on the fly using Cling (which leverages
the clang frontend and LLVM backend).

The PyHPC paper in the above mentioned links cite `pygccxml` and `Reflex` as
motivating examples. The goal is to provide (interactive) bindings for C++
projects with APIs the size of ROOT. (Additionally, the paper has normalize
benchmarking, which is an excellent goal in any project!)

For an overview of the package structure, see
`the docs <https://cppyy.readthedocs.io/en/latest/packages.html#package-structure>`_.
Note that the ``cppyy_backend``
Python module actually accesses ``clang.cindex`` to provide the
``cppyy-generator`` tool used in its workflow:
`cppyy: Utilities <https://cppyy.readthedocs.io/en/latest/utilities.html#bindings-collection>`_.

Due to usage of Cling and the philosophy of lazy binding, user experience may
be heavily impacted at load time and when heavy new instantiations are
necessary (similar to complaintgs about Julia startup time).

However, these should generally only apply to projects with template-heavy
public APIs (e.g. those that use Eigen). Performance could possible be tuned.
For more details, see this GitHub discussion:
`drake#7889 (comment)
<https://github.com/RobotLocomotion/drake/issues/7889#issuecomment-663721506>`_.

Other Mentions
##############
494
495
496
497
498
499
500

* https://github.com/robotpy/robotpy-build

  - Only for RobotPy
  - Emits pybind11 code
  - Uses `header2whatever <https://pypi.org/project/header2whatever/>`_

Joe Snyder's avatar
Joe Snyder committed
501
.. _pycodestyle: https://pypi.org/project/pycodestyle/
Joe Snyder's avatar
Joe Snyder committed
502
.. _Copyright.txt: Copyright.txt
Joe Snyder's avatar
Joe Snyder committed
503
504
.. _`data.kitware.com`: https://data.kitware.com/#folder/57b5de948d777f10f2696370
.. _`text_blocks.py`: text_blocks.py
505
.. _`wrapper_input.yml`: example/wrapper_input.yml