Wrapping a C Library

The four questions to answer first

Before writing any code, answer these about the library you are wrapping. Each one determines a structural decision, and getting one wrong means a rewrite.

1. What is the resource, and who frees it? A handle from lib_open that needs lib_close? A struct the library allocated and you must free with lib_free? A pointer into memory the library owns and you must not free?
2. Is the library thread-safe? Fully reentrant, thread-safe per handle, or "one call at a time, globally"? This decides whether you release the interpreter and what you lock.
3. Does it call back into your code? If so, on which thread, and can the callback re-enter the library?
4. How does it report errors? Return codes, an errno-like global, a per-handle error string, or a callback?

The handle type

The core of almost every wrapper is a heap type (Lesson 03) holding an opaque pointer:

typedef struct {
    PyObject_HEAD
    lib_handle *h;        /* NULL once closed */
    PyObject   *callback; /* a Python callable, or NULL */
    PyObject   *weakreflist;  /* optional, if you want weakref support */
} ConnObject;

Three design decisions follow immediately.

1. Release deterministically, not just at deallocation

Relying on tp_dealloc means the resource is released when the refcount happens to hit zero — which on a cycle is not until a collection runs, and on another implementation may be much later. Every wrapper of a scarce resource should expose an explicit close() and the context-manager protocol:

static PyObject *
Conn_close(PyObject *op, PyObject *Py_UNUSED(ignored))
{
    ConnObject *self = (ConnObject *)op;
    if (self->h != NULL) {
        lib_handle *h = self->h;
        self->h = NULL;              /* NULL first: close() must be idempotent
                                        and re-entrant */
        Py_BEGIN_ALLOW_THREADS
        lib_close(h);
        Py_END_ALLOW_THREADS
    }
    Py_RETURN_NONE;
}

static PyObject *Conn_enter(PyObject *op, PyObject *Py_UNUSED(i)) {
    return Py_NewRef(op);
}
static PyObject *Conn_exit(PyObject *op, PyObject *const *a, Py_ssize_t n) {
    return Conn_close(op, NULL);     /* ignore the exception triple */
}

Registered as "close", "__enter__" and "__exit__" in tp_methods. Then tp_dealloc (or better, tp_finalize) calls the same close path as a backstop, optionally emitting a ResourceWarning so users learn to close explicitly — which is precisely what socket and io do.

2. Check the handle in every method

Because close() exists, every other method must cope with a closed object. One helper, used everywhere:

static lib_handle *
get_handle(ConnObject *self)
{
    if (self->h == NULL) {
        PyErr_SetString(PyExc_ValueError, "operation on a closed connection");
        return NULL;
    }
    return self->h;
}

Skipping this is how wrappers turn a Python-level mistake into a segfault. The same check protects against double-close and against use-after-close from another thread.

3. Never expose the raw pointer as an integer

Returning a pointer as a Python int means any user can fabricate one and hand it back to you. Use a capsule if the pointer must travel through Python, and otherwise keep it inside your type.

Capsules

A PyCapsule is a Python object wrapping a void * together with a name and an optional destructor. The name is the type check: retrieval fails unless the name matches exactly.

static void
capsule_free(PyObject *cap)
{
    lib_handle *h = PyCapsule_GetPointer(cap, "mymod.handle");
    if (h != NULL) lib_close(h);
}

PyObject *cap = PyCapsule_New(h, "mymod.handle", capsule_free);
/* ... later, possibly in a different extension ... */
lib_handle *h = PyCapsule_GetPointer(cap, "mymod.handle");
if (h == NULL) return NULL;      /* ValueError already set on name mismatch */

Two real uses:

• Passing an opaque pointer through Python code that has no business interpreting it — a context handed to a callback, a resource routed between two extensions.
• Exporting a C API from one extension to another. This is the standard mechanism: module A builds a struct of function pointers, publishes it as a capsule named "A._C_API", and module B retrieves it with PyCapsule_Import("A._C_API", 0). NumPy's PyArray_API works exactly this way. It lets extensions call each other's C functions without link-time coupling.

Callbacks into Python

Storing a Python callable in your struct creates four obligations at once, and they draw on four earlier lessons.

/* Storing it: strong reference, and it must be traversed (Lesson 04),
   because the callback may be a closure that refers back to self. */
static int
Conn_set_callback(PyObject *op, PyObject *value, void *closure)
{
    ConnObject *self = (ConnObject *)op;
    if (value != NULL && value != Py_None && !PyCallable_Check(value)) {
        PyErr_SetString(PyExc_TypeError, "callback must be callable or None");
        return -1;
    }
    Py_XSETREF(self->callback, Py_XNewRef(value));
    return 0;
}

/* Invoking it from a thread the library owns (Lesson 10). */
static void
trampoline(void *userdata, int code)
{
    ConnObject *self = (ConnObject *)userdata;

    if (Py_IsFinalizing()) return;                 /* shutting down: do nothing */

    PyGILState_STATE gstate = PyGILState_Ensure();

    PyObject *cb = Py_XNewRef(self->callback);     /* strong ref before use */
    if (cb != NULL) {
        PyObject *arg = PyLong_FromLong(code);
        PyObject *res = arg ? PyObject_CallOneArg(cb, arg) : NULL;
        if (res == NULL) {
            /* No caller to propagate to (Lesson 05). */
            PyErr_FormatUnraisable("Exception ignored in %R callback", (PyObject *)self);
        }
        Py_XDECREF(res);
        Py_XDECREF(arg);
        Py_DECREF(cb);
    }

    PyGILState_Release(gstate);
}

Every line here exists because of a specific failure mode.

The obligations, named:

1. Own a strong reference to the callable, and take a second local one before calling, so that a concurrent conn.callback = None cannot free it mid-call.
2. Traverse it in tp_traverse and clear it in tp_clear. A callback that closes over the connection is a cycle.
3. Attach if the library may call you on its own thread, and check Py_IsFinalizing first.
4. Handle exceptions. A C callback signature usually has nowhere to report one, so route it to the unraisable hook rather than swallowing it. If the library's signature does allow an error return, translate the exception into that and let the library unwind.

Strings, bytes and paths

PyObject *path = NULL;   /* PyUnicode_FSConverter yields a bytes object */
if (!PyArg_ParseTuple(args, "O&:open", PyUnicode_FSConverter, &path)) return NULL;
const char *cpath = PyBytes_AS_STRING(path);
... use cpath ...
Py_DECREF(path);

Returning library-allocated memory

A library function hands you a malloc'd buffer that you must free. Two options:

• Copy it into a bytes with PyBytes_FromStringAndSize and free it immediately. Simple, always correct, costs one copy. Do this unless the data is large.
• Wrap it in a small heap type that owns the pointer, frees it in tp_dealloc, and exports it via the buffer protocol (Lesson 12). Zero-copy, and the Python side sees something memoryview-compatible. Costs you a second type and its lifetime rules.

Do not return a pointer to memory the library owns and may reuse — a static error-message buffer, a pointer into an internal structure — without copying. That is the classic wrapper bug that appears only under concurrency.

Exporting the library's constants and enums

if (PyModule_AddIntMacro(module, LIB_MODE_FAST) < 0) return -1;
if (PyModule_AddIntMacro(module, LIB_MODE_SAFE) < 0) return -1;
if (PyModule_AddStringConstant(module, "LIB_VERSION", lib_version()) < 0) return -1;

PyModule_AddIntMacro uses the macro's own name, so the Python name matches the C name automatically. For a large enum, consider building a Python enum.IntEnum in a thin Python wrapper module instead — you get nicer repr and grouping for free, and the C side stays a flat list of ints.

A sanity checklist for any wrapper

• Every library call that can block or run long is wrapped in Py_BEGIN_ALLOW_THREADS.
• No pointer derived from a Python object survives into a detached region unless it came from a Py_buffer.
• Every error return is translated at one place, and every call site checks it.
• Every method validates that the handle is still open.
• close() is idempotent; __enter__/__exit__ exist; tp_finalize is the backstop.
• Every stored PyObject * is traversed and cleared.
• Any callback registration that stores self in the library holds a reference for the duration.
• If the library is not thread-safe, a PyMutex guards it, and that is documented.
• The module declares its subinterpreter and free-threading stance honestly (Lessons 08 and 11).

import sqlite3, warnings

con = sqlite3.connect(":memory:")
con.close()
try:
    con.execute("select 1")
except sqlite3.ProgrammingError as e:
    print("closed-handle check:", e)

# ResourceWarning as the backstop for a forgotten close:
warnings.simplefilter("always", ResourceWarning)
def leak():
    import socket; s = socket.socket()
leak()
import gc; gc.collect()

Further reading

• C API — Capsules, and Providing a C API for an Extension Module.
• C API — Unicode Objects, especially the UTF-8 and filesystem-encoding functions.
• CPython Modules/_sqlite/. A modern, heavily reviewed wrapper: heap types, module state, error translation, callbacks, thread handling.
• CPython Modules/zlibmodule.c. The canonical example of releasing the interpreter around library work.
• Python Extension Patterns. Long-form patterns for many of the problems above.