Chaobin Tang (唐超斌)

Inter-operate with C/C++ in Python

This article outlines several existing options, providing a high level comparison.

It’s in Python’s early days that its ability to inter-operate with lower level languages made it promising. Later, many solutions emerged to improve this ability in some way.

Out of many, we have these that are more relevant these days:

Python C APIs

Exposing your APIs to Python by wrapping them. The wrapping is a routine of calling several reoccuring Python C APIs that does this in sequence:

After wrapping all the desired APIs, you need to write a boiler-plate module for Python to be able to pick up all the bindings.

Pros

Cons

boost.python

Boost.Python is made specifically for inter-operating with C++ code in Python. It is a C++ template library that provides a declarative syntax to wrap up your code, then cleverly exposing these to Python using Python C API. Its cleverness includes type conversion, exception translation, and more. Using it is straightforward. You write a reasonably thin layer of ad-hoc code in your C++ code to define the APIs that you want to export as Python APIs.

An example:


class_<World>("World")
    .def("greet", &World::greet)
    .def("set", &World::set)

Here has a number of demos that can give you a better look.

Pros

Cons

CFFI

CFFI is one of the modern solutions that combines several nice things. It tremendously reduces what one needs to do to give the C/C++ library a Python interface, it is fast, and it frees you from having to know about Python C APIs and its internals.

Its cffi.cdef(), cffi.dlopen(), cffi.verify() are almost all that you need to be working with, and it is declarative.

ABI compatibility with cffi.dlload


>>> from cffi import FFI
>>> ffi = FFI()
>>> ffi.cdef("""
...     int printf(const char *format, ...); // copy-pasted from the man page
... """)
>>> C = ffi.dlopen(None)               # loads the entire C namespace
>>> arg = ffi.new("char[]", "world")   # equivalent to C code: char arg[] = "world";
>>> C.printf("hi there, %s!\n", arg)   # call printf
hi there, world!

The cffi.dlopen(SHARED_LIB) can be used to load shared library (.so) and from there, generate the Python interface that you can access using the dynamic library returned by cffi.dlopen(). CFFI doesn’t recommend users to use this approach as this requires ABI compatibilities which are less persistent than API compatibilities.

API compatibility with cffi.verify


from cffi import FFI
ffi = FFI()
ffi.cdef("""     // some declarations from the man page
    struct passwd {
        char *pw_name;
        ...;
    };
    struct passwd *getpwuid(int uid);
""")
C = ffi.verify("""   // passed to the real C compiler
#include <sys/types.h>
#include <pwd.h>
""", libraries=[])   # or a list of libraries to link with
p = C.getpwuid(0)
assert ffi.string(p.pw_name) == 'root'    # on Python 3: b'root'

The cffi.verify() is the heart of CFFI. CFFI calls the C compiler just-in-time to compile the source and return a dynamic library. The cffi.verify() takes everything that revolves around using GCC to build a library, -I, -L, and other flags.

Starting from 0.9, cffi.verify() supports C++ code. Because the current latest version is 0.9.2, C++ support hasn’t gone through a time long enough to receive many real world usage though.

Underlying, CFFI relies on the same libffi that Python’s ctypes does. But it aproaches differently to be more reliable and performant. Here is an article that explains nicely how CFFI works.

Here has number of demos that should quickly give you a better look.

Pros

Cons

Cython

Cython is a standalone language. It is in fact a super set of Python, extending it with:

Compared to the alternatives listed above, Cython is unique in that it is a compiler in fact. It first generates the Cython code into a C file, then compiles that C file into a library. This approach takes a price of being very complex, but with the advantage of being the most performant solution. In fact, Cython is most popular in the scientific area of Python, where every library that does number crunching or matrix/vector manipulation hungers for speed.

Cython is also unique in that it allows one to surgically optimize Python code by redefining any piece of Python code in Cython, then gain a C-like speed, and the redefinition is often not much different than the Python version, except you add type annotations to the function, parameters, and variables, and that is often trivial to do.

Since 0.13, it supports C++. The latest version is 0.22.

Pros

Cons

SWIG

Short for Simplified Wrapper and Interface Generator. Swig has been there for quite some time and certainly helped get Python popular in its earlier days. I came across many libraries using swig for wrapper generation but I never used it myself in any project.