From 896be6fa2e29f956256963ac9e6e4c3cee8585d2 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:25:26 -0500 Subject: [PATCH 01/15] Write the motivation and rationale. --- peps/pep-0852.rst | 210 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 peps/pep-0852.rst diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst new file mode 100644 index 00000000000..313d7138491 --- /dev/null +++ b/peps/pep-0852.rst @@ -0,0 +1,210 @@ +PEP: 852 +Title: Asynchronous functions in the C API +Author: Peter Bierma +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 02-Nov-2026 +Python-Version: 3.16 +Post-History: `14-Jan-2023 `__, + `09-Jan-2024 `__, + `12-Apr-2024 `__ + + +Abstract +======== + +This PEP describes a new suite of functions in the C API to interact with +:term:`coroutines `. For example: + + +Motivation +========== + + +The C API has no easy way to invoke asynchronous code +----------------------------------------------------- + +Currently, Python's C API provides no simple interface for defining :term:`coroutine +functions `. This lack of API support has not gone unnoticed by +users. For example, ``mypyc`` still `does not support all asynchronous features +`__ as a result of the lack of support +in CPython. + +For extension developers, the current solution to this problem is to do one +of three things: + +1. Don't write the asynchronous code in C, and instead define a pure-Python + function that handles the :keyword:`await`\ s, and then call an + event-loop-specific API from C. Given that a common use case for building + extension modules is to improve performance, this can be a step backwards + for developers. +2. Implement a type that implements the :term:`coroutine` protocol on its own, + complete with all the necessary logic for a state machine. This is possible, + but non-trivial and bad for extension maintenance; C API users should not + have to learn about the inner workings of generators in order to use a + built-in syntax construct. +3. Use a binding generator that has done one of the above. To `quote + `__ + Petr Viktorin: "CPython should have a usable and stable C API for all + stakeholders, and a list of a few blessed users goes against that goal." + In other words, CPython should aim to make its C API usable (and ergonomic) + where possible -- not encourage users to switch to binding generators. + +All of these options are less than ideal, and as a result, there is support +from both Python users and core developers for adding such a C API to Python. +To `quote `__ Marc-André Lemburg: + + I very much do think that we need a core maintained async C API in CPython. + + We currently don’t have a way to write Python C extensions which work in + async mode without using a shim written in Python which takes care of + the interactions with the asyncio (and other) event loops. + + +A third-party library is bad for both maintainers and users +----------------------------------------------------------- + +The API described in this PEP is heavily influenced by the third-party +`PyAwaitable `__ library -- but why does that +need to come into CPython at all? + +First and foremost, adding a dependency to a C extension typically has a very +high bar. In most cases, it's deemed safer to pick one of the solutions described +above, which each come with their own set of problems. + +Second, distributing native dependencies is extremely difficult in Python's extension +ecosystem. As of recently, extension modules are increasingly being written +using C API binding libraries and not the C API itself. Unfortunately, these +binding libraries cannot wrap APIs that don't come directly from CPython, so +native libraries are effectively unusable to projects using other C API bindings. +In fact, PyAwaitable uses a vendoring technique that would make it functionally +impossible to use it from something such as PyO3. + +There are have been `some `__ `discussions +`__ about making this easier (both of whic +arised *because* PyAwaitable was so difficult to distribute), but in general, fixing +this problem seems like a much more massive proposal than what is currently being +proposed. An asynchronous C API in Python's core would be useful, and adding +one to CPython is much simpler and easier for both maintainers and users than +trying to solve a more generic problem in the packaging ecosystem; in other words, +to quote :pep:`20`, "practicality beats purity". + + +Why was a third-party library created in the first place? +********************************************************** + +When PyAwaitable was first created in early 2024, the author of the implementation +(who is also the author of this proposal) was new to CPython contribution and +thus not in a great place to author a PEP or oversee an implementation of PyAwaitable +in CPython. A relatively simple implementation was possible without changes to +the CPython core, and as such, PyAwaitable was better fit as a library for the +time being -- though always with the intent of eventually becoming part of the +public C API. + +The author has since become a core developer, and as such, it is now much more +feasible for them to maintain PyAwaitable in CPython directly. + + +Specification +============= + + +Rationale +========= + + +An API for coroutine functions, not futures +------------------------------------------- + +This proposal aims to add a C API for generic ``async``/``await`` constructs, *not* +an API for describing all capabilities of :mod:`asyncio`. While this PEP may +be reason for adding an ``asyncio`` C API in the future, its only goal is to +make :keyword:`async def` functions possible to implement in the C API. + + +Deviations from PyAwaitable +--------------------------- + + +``PyAsync`` instead of ``PyAwaitable`` +************************************** + +The most obvious deviation from the original library is the use of the +``PyAsync`` prefix over ``PyAwaitable``. This is because the original +"PyAwaitable" name is a historical artifact; it was chosen as +a parallel to the ``PyCoro`` prefix, with the idea that a ``PyCoro`` was +analogous to a native ``coroutine`` object, while a ``PyAwaitable`` would +be analogous to a native ``awaitable`` object. Given that this API (and new +versions of PyAwaitable) support all of the :class:`~collections.abc.Coroutine` +protocol (in contrast to the :class:`~collections.abc.Awaitable` protocol), +this name is now misleading, and thus the rename to ``PyAsync`` felt necessary. + + +Simpler storage for callback values +*********************************** + +This API uses a single ``PyObject *`` reference and ``void *`` pointer for value +storage between callbacks instead of variadic argument lists that were stored as +arrays. PyAwaitable used the latter because it was optimized for ergonomics when +writing pure C code, but since this API is also designed to be accessible from +bindings generators (and FFIs) as well, a single pointer for each felt like the +better call. PyAwaitable's array system also turned out to be a source of +complexity and bugs, and avoiding extra complexity for CPython where possible +is ideal. + + +Backwards Compatibility +======================= + +This PEP has no known backwards compatibility concerns. + + +Security Implications +===================== + +This PEP has no known security concerns. + + +How to Teach This +================= + +[How to teach users, new and experienced, how to apply the PEP to their work.] + + +Reference Implementation +======================== + +[Link to any existing implementation and details about its state, e.g. proof-of-concept.] + + +Rejected Ideas +============== + +TBD + + +Open Issues +=========== + +TBD + + +Acknowledgements +================ + +Thanks to Petr Viktorin for his help with the original PyAwaitable design +discussion, and to all users of PyAwaitable over the past few years. + + +Change History +============== + +TBD. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. From 73b66430a3f83722d904d94d98097557a9cca38c Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:41:15 -0500 Subject: [PATCH 02/15] Add the specification based on PyAwaitable's docs. --- peps/pep-0852.rst | 207 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 207 insertions(+) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 313d7138491..0808980a942 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -109,6 +109,203 @@ feasible for them to maintain PyAwaitable in CPython directly. Specification ============= +The following APIs are available through ``Python.h`` and require an +:term:`attached thread state`. Unless otherwise specified, an *awaitable* +argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed +references to their arguments. + + +Async objects +------------- + +.. c:type:: PyAsync + + An opaque Python object that implements the :class:`~collections.abc.Coroutine` + protocol. A :c:type:`PyAsync` object stores a sequence of asynchronous + operations and callbacks to execute when it is awaited. + + +.. c:var:: PyTypeObject PyAsync_Type + + The type object for :c:type:`PyAsync` objects. + + +.. c:function:: int PyAsync_CheckExact(PyObject *obj) + + Return nonzero if *obj* is a :c:type:`PyAsync` object, and ``0`` otherwise. + This function cannot fail. + + +.. c:function:: PyObject *PyAsync_New(void) + + Create a new, empty :c:type:`PyAsync` object. Return a new + :term:`strong reference` on success, or ``NULL`` with an exception set + on failure. + + By default, awaiting the object returns ``None``. The result can be changed + with :c:func:`PyAsync_SetResult`. An object can only be awaited once; + attempting to reuse a completed object raises :exc:`RuntimeError`. + + Destroying an object that was never awaited or cancelled with + :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. + + +.. c:function:: int PyAsync_SetResult(PyObject *awaitable, PyObject *result) + + Set the object returned by awaiting *awaitable* to *result*. + *result* must not be ``NULL``. + + This function stores a strong reference to *result*, releasing any + previously stored result. It cannot fail and returns ``0``. + + +Awaiting and callbacks +---------------------- + +.. c:type:: int (*PyAsync_Callback)(PyObject *awaitable, PyObject *result) + + The type of a result callback. *awaitable* is the :c:type:`PyAsync` + object executing the callback, and *result* is the value returned by + the awaited operation. + + The callback must return one of the following: + + * ``1`` on success. + * ``0`` with an exception set to invoke the associated error callback, + or propagate the exception if no error callback was provided. + * ``-1`` with an exception set to propagate the exception directly, + skipping the associated error callback. + + +.. c:type:: int (*PyAsync_Error)(PyObject *awaitable, PyObject *exception) + + The type of an error callback. *exception* is the exception raised by + an awaited operation or its result callback. The error callback is + invoked with no exception set. + + The callback must return one of the following: + + * ``1`` to handle the exception and continue executing queued operations. + * ``0`` to restore and propagate the original exception. + * ``-1`` with a new exception set to propagate that exception instead. + + +.. c:type:: int (*PyAsync_Defer)(PyObject *awaitable) + + The type of a deferred callback, registered with + :c:func:`PyAsync_DeferAwait`. The callback must return ``1`` on success, + or ``0`` with an exception set to stop execution and propagate the + exception to the caller awaiting *awaitable*. + + +.. c:function:: int PyAsync_AddAwait(PyObject *awaitable, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append *coroutine* to the operations to await when *awaitable* executes. + Operations execute in the order they were added. This function stores a + strong reference to *coroutine*, which must be a non-``NULL`` awaitable + object distinct from *awaitable*. + + On successful completion, *result_callback* is invoked with *awaitable* + and the operation's return value. If *result_callback* is ``NULL``, the + return value is discarded. + + If the operation raises an exception, or *result_callback* returns ``0``, + *error_callback* is invoked. If *error_callback* is ``NULL``, the exception + propagates to the caller awaiting *awaitable*. + + Callbacks may append further operations to *awaitable*. Adding operations + to a completed object raises :exc:`RuntimeError`. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: int PyAsync_AddExpr(PyObject *awaitable, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on + both success and failure. If *expr* is ``NULL``, return ``-1`` without + changing the current exception. + + Otherwise, return ``0`` on success, or ``-1`` with an exception set + on failure. + + +.. c:function:: int PyAsync_DeferAwait(PyObject *awaitable, PyAsync_Defer callback) + + Append a synchronous callback to the operations executed when *awaitable* + is awaited. *callback* must not be ``NULL``. It is invoked with *awaitable* + after previously queued operations complete, and may queue further + operations. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +.. c:function:: void PyAsync_Cancel(PyObject *awaitable) + + Clear the queued operations and callbacks, releasing their references + and discarding the current await iterator. This also suppresses the + :exc:`ResourceWarning` for an object that was never awaited. + + This function may be called from a callback. Further operations may be + added to *awaitable* until it completes. This function cannot fail. + + +.. c:function:: int PyAsync_AsyncWith(PyObject *awaitable, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) + + Append an :keyword:`async with` operation using *context_manager*. + Await its ``__aenter__`` method, invoke *result_callback*, and await its + ``__aexit__`` method. *result_callback* receives a :c:type:`PyAsync` + object representing the context manager operation and the value + returned by ``__aenter__``. It may be ``NULL``. + + If *result_callback* fails, its exception is passed to ``__aexit__``, + which may suppress it by returning a true value. An exception from + ``__aenter__`` propagates without invoking ``__aexit__``. + + *error_callback* handles exceptions from the operation, including any + exception left unsuppressed by ``__aexit__``. It receives *awaitable* + and the exception. If it is ``NULL``, the exception propagates to the + caller awaiting *awaitable*. + + Return ``0`` on success, or ``-1`` with an exception set on failure. + + +Value storage +------------- + +Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer +slot for storing state between callbacks. Both slots are initially ``NULL``. + +.. c:function:: int PyAsync_SetValue(PyObject *awaitable, PyObject *value) + + Store a strong reference to *value* in the object slot, releasing the + previously stored reference. If *value* is ``NULL``, clear the slot. + + This function cannot fail and returns ``0``. + + +.. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) + + Return a :term:`borrowed reference` to the object stored in the object + slot, or ``NULL`` if the slot is empty. This function cannot fail and + does not set an exception when the slot is empty. + + +.. c:function:: int PyAsync_SetArbValue(PyObject *awaitable, void *value) + + Store *value* in the arbitrary pointer slot, replacing the previous + pointer. *value* may be ``NULL``. The caller is responsible for the + lifetime of any memory referenced by the pointer. + + This function cannot fail and returns ``0``. + + +.. c:function:: void *PyAsync_GetArbValue(PyObject *awaitable) + + Return the pointer stored in the arbitrary pointer slot, or ``NULL`` + if the slot is empty. This function cannot fail and does not set an + exception when the slot is empty. + + Rationale ========= @@ -154,6 +351,16 @@ complexity and bugs, and avoiding extra complexity for CPython where possible is ideal. +Different callback return values +******************************** + +In PyAwaitable, callbacks may return ``0``, ``-1``, and ``-2``, which correspond +to ``1``, ``0``, and ``-1`` in this proposal, respectively. This is also a historical +artifact; PyAwaitable did not originally have the ``-2`` return value, and changing +the meaning of ``0`` and ``-1`` could not be done for the sake of backwards +compatibility. + + Backwards Compatibility ======================= From c445a84ded9ccdd588e94d06ecf5f89a952c58ed Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:48:59 -0500 Subject: [PATCH 03/15] Use strong references instead of borrowed references. --- peps/pep-0852.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 0808980a942..a8135087f15 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -285,7 +285,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. .. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) - Return a :term:`borrowed reference` to the object stored in the object + Return a new :term:`strong reference` to the object stored in the object slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an exception when the slot is empty. @@ -350,6 +350,11 @@ better call. PyAwaitable's array system also turned out to be a source of complexity and bugs, and avoiding extra complexity for CPython where possible is ideal. +This API also returns a :term:`strong reference` from ``GetValue`` instead of a +:term:`borrowed reference`. PyAwaitable used the latter because it predated the +acceptance of :pep:`703`, and thus any concurrency implications were not +considered. + Different callback return values ******************************** From 3009471eefab4f381ff0ba0165bc61eb9c10b16a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 14:52:06 -0500 Subject: [PATCH 04/15] Avoid the outdated argument names. --- peps/pep-0852.rst | 54 +++++++++++++++++++++++------------------------ 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index a8135087f15..c3317fa72a1 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -110,7 +110,7 @@ Specification ============= The following APIs are available through ``Python.h`` and require an -:term:`attached thread state`. Unless otherwise specified, an *awaitable* +:term:`attached thread state`. Unless otherwise specified, an *asyncobj* argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed references to their arguments. @@ -150,9 +150,9 @@ Async objects :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. -.. c:function:: int PyAsync_SetResult(PyObject *awaitable, PyObject *result) +.. c:function:: int PyAsync_SetResult(PyObject *asyncobj, PyObject *result) - Set the object returned by awaiting *awaitable* to *result*. + Set the object returned by awaiting *asyncobj* to *result*. *result* must not be ``NULL``. This function stores a strong reference to *result*, releasing any @@ -162,9 +162,9 @@ Async objects Awaiting and callbacks ---------------------- -.. c:type:: int (*PyAsync_Callback)(PyObject *awaitable, PyObject *result) +.. c:type:: int (*PyAsync_Callback)(PyObject *asyncobj, PyObject *result) - The type of a result callback. *awaitable* is the :c:type:`PyAsync` + The type of a result callback. *asyncobj* is the :c:type:`PyAsync` object executing the callback, and *result* is the value returned by the awaited operation. @@ -177,7 +177,7 @@ Awaiting and callbacks skipping the associated error callback. -.. c:type:: int (*PyAsync_Error)(PyObject *awaitable, PyObject *exception) +.. c:type:: int (*PyAsync_Error)(PyObject *asyncobj, PyObject *exception) The type of an error callback. *exception* is the exception raised by an awaited operation or its result callback. The error callback is @@ -190,36 +190,36 @@ Awaiting and callbacks * ``-1`` with a new exception set to propagate that exception instead. -.. c:type:: int (*PyAsync_Defer)(PyObject *awaitable) +.. c:type:: int (*PyAsync_Defer)(PyObject *asyncobj) The type of a deferred callback, registered with :c:func:`PyAsync_DeferAwait`. The callback must return ``1`` on success, or ``0`` with an exception set to stop execution and propagate the - exception to the caller awaiting *awaitable*. + exception to the caller awaiting *asyncobj*. -.. c:function:: int PyAsync_AddAwait(PyObject *awaitable, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AddAwait(PyObject *asyncobj, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) - Append *coroutine* to the operations to await when *awaitable* executes. + Append *coroutine* to the operations to await when *asyncobj* executes. Operations execute in the order they were added. This function stores a strong reference to *coroutine*, which must be a non-``NULL`` awaitable - object distinct from *awaitable*. + object distinct from *asyncobj*. - On successful completion, *result_callback* is invoked with *awaitable* + On successful completion, *result_callback* is invoked with *asyncobj* and the operation's return value. If *result_callback* is ``NULL``, the return value is discarded. If the operation raises an exception, or *result_callback* returns ``0``, *error_callback* is invoked. If *error_callback* is ``NULL``, the exception - propagates to the caller awaiting *awaitable*. + propagates to the caller awaiting *asyncobj*. - Callbacks may append further operations to *awaitable*. Adding operations + Callbacks may append further operations to *asyncobj*. Adding operations to a completed object raises :exc:`RuntimeError`. Return ``0`` on success, or ``-1`` with an exception set on failure. -.. c:function:: int PyAsync_AddExpr(PyObject *awaitable, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AddExpr(PyObject *asyncobj, PyObject *expr, PyAsync_Callback result_callback, PyAsync_Error error_callback) Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on both success and failure. If *expr* is ``NULL``, return ``-1`` without @@ -229,27 +229,27 @@ Awaiting and callbacks on failure. -.. c:function:: int PyAsync_DeferAwait(PyObject *awaitable, PyAsync_Defer callback) +.. c:function:: int PyAsync_DeferAwait(PyObject *asyncobj, PyAsync_Defer callback) - Append a synchronous callback to the operations executed when *awaitable* - is awaited. *callback* must not be ``NULL``. It is invoked with *awaitable* + Append a synchronous callback to the operations executed when *asyncobj* + is awaited. *callback* must not be ``NULL``. It is invoked with *asyncobj* after previously queued operations complete, and may queue further operations. Return ``0`` on success, or ``-1`` with an exception set on failure. -.. c:function:: void PyAsync_Cancel(PyObject *awaitable) +.. c:function:: void PyAsync_Cancel(PyObject *asyncobj) Clear the queued operations and callbacks, releasing their references and discarding the current await iterator. This also suppresses the :exc:`ResourceWarning` for an object that was never awaited. This function may be called from a callback. Further operations may be - added to *awaitable* until it completes. This function cannot fail. + added to *asyncobj* until it completes. This function cannot fail. -.. c:function:: int PyAsync_AsyncWith(PyObject *awaitable, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AsyncWith(PyObject *asyncobj, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) Append an :keyword:`async with` operation using *context_manager*. Await its ``__aenter__`` method, invoke *result_callback*, and await its @@ -262,9 +262,9 @@ Awaiting and callbacks ``__aenter__`` propagates without invoking ``__aexit__``. *error_callback* handles exceptions from the operation, including any - exception left unsuppressed by ``__aexit__``. It receives *awaitable* + exception left unsuppressed by ``__aexit__``. It receives *asyncobj* and the exception. If it is ``NULL``, the exception propagates to the - caller awaiting *awaitable*. + caller awaiting *asyncobj*. Return ``0`` on success, or ``-1`` with an exception set on failure. @@ -275,7 +275,7 @@ Value storage Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer slot for storing state between callbacks. Both slots are initially ``NULL``. -.. c:function:: int PyAsync_SetValue(PyObject *awaitable, PyObject *value) +.. c:function:: int PyAsync_SetValue(PyObject *asyncobj, PyObject *value) Store a strong reference to *value* in the object slot, releasing the previously stored reference. If *value* is ``NULL``, clear the slot. @@ -283,14 +283,14 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. This function cannot fail and returns ``0``. -.. c:function:: PyObject *PyAsync_GetValue(PyObject *awaitable) +.. c:function:: PyObject *PyAsync_GetValue(PyObject *asyncobj) Return a new :term:`strong reference` to the object stored in the object slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an exception when the slot is empty. -.. c:function:: int PyAsync_SetArbValue(PyObject *awaitable, void *value) +.. c:function:: int PyAsync_SetArbValue(PyObject *asyncobj, void *value) Store *value* in the arbitrary pointer slot, replacing the previous pointer. *value* may be ``NULL``. The caller is responsible for the @@ -299,7 +299,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. This function cannot fail and returns ``0``. -.. c:function:: void *PyAsync_GetArbValue(PyObject *awaitable) +.. c:function:: void *PyAsync_GetArbValue(PyObject *asyncobj) Return the pointer stored in the arbitrary pointer slot, or ``NULL`` if the slot is empty. This function cannot fail and does not set an From cec7a1493ba5a4415fa99e856bcc86a68803cccf Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:12:36 -0500 Subject: [PATCH 05/15] Add an examples section. --- peps/pep-0852.rst | 182 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 181 insertions(+), 1 deletion(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index c3317fa72a1..206e297df2b 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -381,7 +381,187 @@ This PEP has no known security concerns. How to Teach This ================= -[How to teach users, new and experienced, how to apply the PEP to their work.] +The new APIs will be documented in the C API documentation. +A tutorial can also be added based on the tutorial present in PyAwaitable's +documentation. + +Examples +-------- + +The first three examples adapt patterns from PyAwaitable's documentation. +Callbacks return ``1`` on success, while functions such as +:c:func:`PyAsync_AddExpr` return ``0`` on success. + + +Example: Forwarding an asynchronous result +****************************************** + +An extension method can call an asynchronous function and forward the +result of awaiting it. :c:type:`PyAsync` handles suspension and resumption +for the extension. + +.. code-block:: c + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + // Equivalent to def call_async(func): return await func() + static PyObject * + call_async(PyObject *self, PyObject *func) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddExpr(asyncobj, PyObject_CallNoArgs(func), + return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + + +Example: Keeping state across an await +************************************** + +To implement ``return (await get_number()) * factor``, store *factor* with +:c:func:`PyAsync_SetValue` before queuing ``get_number()`` with this callback. +The :c:type:`PyAsync` object keeps the factor alive while execution is +suspended. The callback releases the strong reference returned by +:c:func:`PyAsync_GetValue` after using it. + +.. code-block:: c + + static int + multiply_result(PyObject *asyncobj, PyObject *value) + { + PyObject *factor = PyAsync_GetValue(asyncobj); + PyObject *result = PyNumber_Multiply(value, factor); + Py_DECREF(factor); + if (result == NULL) { + return 0; + } + PyAsync_SetResult(asyncobj, result); + Py_DECREF(result); + return 1; + } + + +Example: Handling a timeout +*************************** + +In ``call_async`` above, replace the final ``NULL`` argument to +:c:func:`PyAsync_AddExpr` with ``timeout_to_none``. Awaiting the result +then returns ``None`` on :exc:`TimeoutError`. Returning ``0`` from the +callback lets other exceptions propagate. + +.. code-block:: c + + static int + timeout_to_none(PyObject *asyncobj, PyObject *exception) + { + if (!PyErr_GivenExceptionMatches(exception, PyExc_TimeoutError)) { + return 0; + } + PyAsync_SetResult(asyncobj, Py_None); + return 1; + } + +For example, :func:`asyncio.timeout` can impose a deadline on an operation: + +.. code-block:: python + + import asyncio + + async def slow_operation(): + async with asyncio.timeout(0.01): + return await asyncio.sleep(1, result=42) + + assert asyncio.run(call_async(slow_operation)) is None + + +Example: Scheduling a dependent read +************************************ + +Suppose a protocol reader exposes asynchronous ``read_size()`` and +``readexactly(size)`` methods. Store the reader with +:c:func:`PyAsync_SetValue`, then queue ``reader.read_size()`` with this +callback. The first read's result determines the next asynchronous call, +which reuses ``return_result`` to return the completed packet. + +.. code-block:: c + + static int + read_body(PyObject *asyncobj, PyObject *size) + { + PyObject *reader = PyAsync_GetValue(asyncobj); + PyObject *coro = PyObject_CallMethod(reader, "readexactly", "O", size); + Py_DECREF(reader); + return PyAsync_AddExpr(asyncobj, coro, return_result, NULL) == 0; + } + + +Example: Acquiring an async lock for native work +************************************************ + +An asynchronous context manager can acquire a lock and return the protected +dictionary to the result callback: + +.. code-block:: python + + import asyncio + from contextlib import asynccontextmanager + + @asynccontextmanager + async def locked_cache(lock, cache): + async with lock: + yield cache + +The C callback updates that dictionary while the lock is held: + +.. code-block:: c + + static int + mark_ready(PyObject *asyncobj, PyObject *cache) + { + return PyDict_SetItemString(cache, "ready", Py_True) == 0; + } + + static PyObject * + update_locked(PyObject *self, PyObject *guard) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AsyncWith(asyncobj, guard, mark_ready, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + +Python code passes the lock and dictionary through the context manager: + +.. code-block:: python + + async def main(): + lock, cache = asyncio.Lock(), {} + await update_locked(locked_cache(lock, cache)) + assert cache["ready"] is True + assert not lock.locked() + + asyncio.run(main()) + +:c:func:`PyAsync_AsyncWith` invokes the result callback after acquiring the +lock, and releases the lock even if the dictionary update raises an exception. Reference Implementation From 3a37ca43e0377f5b3672f09505cf998b8db64d50 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:28:21 -0500 Subject: [PATCH 06/15] Some cleanup. --- peps/pep-0852.rst | 38 +++++++++++++++++++++++++++++++------- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 206e297df2b..590cf0b60e9 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -17,6 +17,33 @@ Abstract This PEP describes a new suite of functions in the C API to interact with :term:`coroutines `. For example: +.. code-block:: c + + #include + + static int + return_result(PyObject *asyncobj, PyObject *result) + { + PyAsync_SetResult(asyncobj, result); + return 1; + } + + /* Equivalent to: async def await_from_c(coro): return await coro */ + static PyObject * + await_from_c(PyObject *self, PyObject *coro) /* METH_O */ + { + PyObject *asyncobj = PyAsync_New(); + if (asyncobj == NULL) { + return NULL; + } + if (PyAsync_AddAwait(asyncobj, coro, return_result, NULL) < 0) { + PyAsync_Cancel(asyncobj); + Py_DECREF(asyncobj); + return NULL; + } + return asyncobj; + } + Motivation ========== @@ -81,9 +108,9 @@ native libraries are effectively unusable to projects using other C API bindings In fact, PyAwaitable uses a vendoring technique that would make it functionally impossible to use it from something such as PyO3. -There are have been `some `__ `discussions -`__ about making this easier (both of whic -arised *because* PyAwaitable was so difficult to distribute), but in general, fixing +There have been `some `__ `discussions +`__ about making this easier (both of which +arose *because* PyAwaitable was so difficult to distribute), but in general, fixing this problem seems like a much more massive proposal than what is currently being proposed. An asynchronous C API in Python's core would be useful, and adding one to CPython is much simpler and easier for both maintainers and users than @@ -385,13 +412,10 @@ The new APIs will be documented in the C API documentation. A tutorial can also be added based on the tutorial present in PyAwaitable's documentation. + Examples -------- -The first three examples adapt patterns from PyAwaitable's documentation. -Callbacks return ``1`` on success, while functions such as -:c:func:`PyAsync_AddExpr` return ``0`` on success. - Example: Forwarding an asynchronous result ****************************************** From 204c7fd0148da58df6f7e52dbd15c22f641585aa Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 15:41:08 -0500 Subject: [PATCH 07/15] Add reference implementation. --- peps/pep-0852.rst | 32 ++++++++++++++++++++++++-------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 590cf0b60e9..502e95b073a 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -55,8 +55,8 @@ The C API has no easy way to invoke asynchronous code Currently, Python's C API provides no simple interface for defining :term:`coroutine functions `. This lack of API support has not gone unnoticed by users. For example, ``mypyc`` still `does not support all asynchronous features -`__ as a result of the lack of support -in CPython. +`__, because they are forced to +build coroutine types from scratch, which is difficult to do correctly. For extension developers, the current solution to this problem is to do one of three things: @@ -73,10 +73,11 @@ of three things: built-in syntax construct. 3. Use a binding generator that has done one of the above. To `quote `__ - Petr Viktorin: "CPython should have a usable and stable C API for all - stakeholders, and a list of a few blessed users goes against that goal." - In other words, CPython should aim to make its C API usable (and ergonomic) - where possible -- not encourage users to switch to binding generators. + Petr Viktorin (on a separate proposal): "CPython should have a usable and + stable C API for all stakeholders, and a list of a few blessed users goes + against that goal." In other words, CPython should aim to make its C API + usable (and ergonomic) where possible -- not encourage users to switch to + binding generators. All of these options are less than ideal, and as a result, there is support from both Python users and core developers for adding such a C API to Python. @@ -132,6 +133,11 @@ public C API. The author has since become a core developer, and as such, it is now much more feasible for them to maintain PyAwaitable in CPython directly. +It is also worth noting that the reference implementation of this proposal does +require the use of some internal helper functions (that PyAwaitable, as a +third-party, had to copy and work around), which is more evidence that PyAwaitable +is a good fit for CPython. + Specification ============= @@ -333,6 +339,11 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. exception when the slot is empty. +Limited API +----------- + +All APIs in this PEP will be made part of the :ref:`Stable ABI` in Python 3.16. + Rationale ========= @@ -346,6 +357,11 @@ an API for describing all capabilities of :mod:`asyncio`. While this PEP may be reason for adding an ``asyncio`` C API in the future, its only goal is to make :keyword:`async def` functions possible to implement in the C API. +As such, this proposal doesn't come with any generic mechanism to implement +suspension of a C function, though it is still possible to implement non-blocking +native operations using other parts of Python's C API (such as by manually creating +:class:`asyncio.Future` instances through one of the call APIs) + Deviations from PyAwaitable --------------------------- @@ -591,8 +607,8 @@ lock, and releases the lock even if the dictionary update raises an exception. Reference Implementation ======================== -[Link to any existing implementation and details about its state, e.g. proof-of-concept.] - +A reference implementation of this PEP can be found +`here `__. Rejected Ideas ============== From 35ebb40c5ffdce2af2442f2295989778420b1e8b Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:54:22 -0500 Subject: [PATCH 08/15] Touch ups. --- peps/pep-0852.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 502e95b073a..7496adaa716 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -30,7 +30,7 @@ This PEP describes a new suite of functions in the C API to interact with /* Equivalent to: async def await_from_c(coro): return await coro */ static PyObject * - await_from_c(PyObject *self, PyObject *coro) /* METH_O */ + await_from_c(PyObject *self, PyObject *coro) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -70,7 +70,7 @@ of three things: complete with all the necessary logic for a state machine. This is possible, but non-trivial and bad for extension maintenance; C API users should not have to learn about the inner workings of generators in order to use a - built-in syntax construct. + built-in syntax construct. This is the root of ``mypyc``'s problem, for example. 3. Use a binding generator that has done one of the above. To `quote `__ Petr Viktorin (on a separate proposal): "CPython should have a usable and @@ -451,7 +451,7 @@ for the extension. // Equivalent to def call_async(func): return await func() static PyObject * - call_async(PyObject *self, PyObject *func) /* METH_O */ + call_async(PyObject *self, PyObject *func) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -574,7 +574,7 @@ The C callback updates that dictionary while the lock is held: } static PyObject * - update_locked(PyObject *self, PyObject *guard) /* METH_O */ + update_locked(PyObject *self, PyObject *guard) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { @@ -610,16 +610,17 @@ Reference Implementation A reference implementation of this PEP can be found `here `__. + Rejected Ideas ============== -TBD +TBD. Open Issues =========== -TBD +TBD. Acknowledgements From 0e1ec9b93f72a83467d9420c86cd4237fe550e96 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:57:19 -0500 Subject: [PATCH 09/15] Add codeowner. --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 944a91ed755..c30f6960125 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -730,6 +730,7 @@ peps/pep-0849.rst @JelleZijlstra peps/pep-0850/ @willingc @rgommers peps/pep-0850.rst @willingc @rgommers peps/pep-0851.rst @StanFromIreland +peps/pep-0852.rst @ZeroIntensity # ... peps/pep-2026.rst @hugovk # ... From f08e5a35d3d44fcb17dcb2337735ce112af203fc Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 18:59:26 -0500 Subject: [PATCH 10/15] Remove misleading paragraph. --- peps/pep-0852.rst | 5 ----- 1 file changed, 5 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 7496adaa716..c2fa8b8606d 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -357,11 +357,6 @@ an API for describing all capabilities of :mod:`asyncio`. While this PEP may be reason for adding an ``asyncio`` C API in the future, its only goal is to make :keyword:`async def` functions possible to implement in the C API. -As such, this proposal doesn't come with any generic mechanism to implement -suspension of a C function, though it is still possible to implement non-blocking -native operations using other parts of Python's C API (such as by manually creating -:class:`asyncio.Future` instances through one of the call APIs) - Deviations from PyAwaitable --------------------------- From e107cb300ba772491d6ed896fadc6379744407db Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 19:08:26 -0500 Subject: [PATCH 11/15] Strengthen motivation a little bit. --- peps/pep-0852.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index c2fa8b8606d..7a9780526b4 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -128,15 +128,15 @@ thus not in a great place to author a PEP or oversee an implementation of PyAwai in CPython. A relatively simple implementation was possible without changes to the CPython core, and as such, PyAwaitable was better fit as a library for the time being -- though always with the intent of eventually becoming part of the -public C API. - -The author has since become a core developer, and as such, it is now much more -feasible for them to maintain PyAwaitable in CPython directly. +public C API. The author has since become a core developer, and as such, it is +now much more feasible for them to maintain a variant of PyAwaitable in CPython +directly. It is also worth noting that the reference implementation of this proposal does require the use of some internal helper functions (that PyAwaitable, as a third-party, had to copy and work around), which is more evidence that PyAwaitable -is a good fit for CPython. +is a good fit for CPython, as one of the original arguments against adding such +an API was that it was implementable using only the public C API. Specification From d03f89576a5217c95fe5bfe766e628630bf1aa6e Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 2 Nov 2026 20:36:11 -0500 Subject: [PATCH 12/15] Fix Sphinx errors. --- peps/pep-0852.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 7a9780526b4..150fa525db9 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -342,7 +342,7 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. Limited API ----------- -All APIs in this PEP will be made part of the :ref:`Stable ABI` in Python 3.16. +All APIs in this PEP will be made part of the :ref:`Stable ABI ` in Python 3.16. Rationale @@ -569,13 +569,13 @@ The C callback updates that dictionary while the lock is held: } static PyObject * - update_locked(PyObject *self, PyObject *guard) /* METH_O */ + update_locked(PyObject *self, PyObject *lock) /* METH_O */ { PyObject *asyncobj = PyAsync_New(); if (asyncobj == NULL) { return NULL; } - if (PyAsync_AsyncWith(asyncobj, guard, mark_ready, NULL) < 0) { + if (PyAsync_AsyncWith(asyncobj, lock, mark_ready, NULL) < 0) { PyAsync_Cancel(asyncobj); Py_DECREF(asyncobj); return NULL; From 5f128aed16a4b0af3721a5eae8556543050f06c3 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 8 Nov 2026 17:46:01 -0500 Subject: [PATCH 13/15] Some clarifications. --- peps/pep-0852.rst | 145 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 113 insertions(+), 32 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 150fa525db9..72a72cd38fe 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -144,8 +144,9 @@ Specification The following APIs are available through ``Python.h`` and require an :term:`attached thread state`. Unless otherwise specified, an *asyncobj* -argument must be a :c:type:`PyAsync` object. Callbacks receive borrowed -references to their arguments. +argument must be a :c:type:`PyAsync` object; passing any other object is +undefined behavior. Callbacks receive :term:`borrowed references ` to their arguments. Async objects @@ -153,9 +154,21 @@ Async objects .. c:type:: PyAsync - An opaque Python object that implements the :class:`~collections.abc.Coroutine` - protocol. A :c:type:`PyAsync` object stores a sequence of asynchronous - operations and callbacks to execute when it is awaited. + An opaque Python object that implements the + :class:`~collections.abc.Coroutine` protocol. A :c:type:`PyAsync` + object stores a queue of asynchronous operations and callbacks to + execute when it is awaited. + + Awaiting a :c:type:`PyAsync` object runs its queued operations in + order. The object suspends whenever an operation suspends, and + completes once the queue is empty, returning the result set with + :c:func:`PyAsync_SetResult` (``None`` by default). A :c:type:`PyAsync` + object is recognized by :func:`asyncio.iscoroutine` and + :func:`inspect.isawaitable`, and can be passed to + :func:`asyncio.create_task`. + + The type cannot be subclassed or instantiated from Python, and is not + exposed in any Python module. .. c:var:: PyTypeObject PyAsync_Type @@ -179,8 +192,9 @@ Async objects with :c:func:`PyAsync_SetResult`. An object can only be awaited once; attempting to reuse a completed object raises :exc:`RuntimeError`. - Destroying an object that was never awaited or cancelled with - :c:func:`PyAsync_Cancel` emits a :exc:`ResourceWarning`. + Destroying an object that was never awaited, cancelled with + :c:func:`PyAsync_Cancel`, or closed emits a :exc:`RuntimeWarning`, as + destroying a native coroutine that was never awaited does. .. c:function:: int PyAsync_SetResult(PyObject *asyncobj, PyObject *result) @@ -192,9 +206,45 @@ Async objects previously stored result. It cannot fail and returns ``0``. +Coroutine protocol +------------------ + +A :c:type:`PyAsync` object behaves as a native coroutine object does: + +* ``send(value)`` resumes the object. *value* must be ``None`` if the + object has not started. The object raises :exc:`StopIteration` carrying + its result when it completes. +* ``throw(exc)`` raises *exc* inside the currently awaited operation, as + :keyword:`await` would. If the operation does not handle it, the + operation's error callback is invoked with the exception, and so on as + described under :c:func:`PyAsync_AddAwait`. If the object has not started, + or is between operations, the exception propagates and the object + completes. This is how :mod:`asyncio` cancellation reaches an awaiting + :c:type:`PyAsync` object: a C error callback can observe + :exc:`asyncio.CancelledError` exactly as an ``except`` clause in a + coroutine would. +* ``close()`` throws :exc:`GeneratorExit` into the object. The currently + awaited operation is closed, pending error callbacks and + :keyword:`async with` exits observe :exc:`GeneratorExit`, and the object + completes. As with native coroutines, if the object suspends again while + handling :exc:`GeneratorExit`, ``close()`` raises :exc:`RuntimeError`. + Closing an object that has not started or has completed does nothing. + +A :c:type:`PyAsync` object may not be resumed while it is already executing; +doing so raises :exc:`ValueError`. Queue and storage mutations are +serialized per object, so on the :term:`free-threaded ` +build it is safe to add operations or set values from another thread, but +an object can only be driven by one thread at a time. + + Awaiting and callbacks ---------------------- +Callbacks report their outcome through their return value. Every callback +must set an exception if and only if it reports failure; a callback that +returns success with an exception set, or failure without one, causes the +:c:type:`PyAsync` object to raise :exc:`SystemError`. + .. c:type:: int (*PyAsync_Callback)(PyObject *asyncobj, PyObject *result) The type of a result callback. *asyncobj* is the :c:type:`PyAsync` @@ -231,12 +281,14 @@ Awaiting and callbacks exception to the caller awaiting *asyncobj*. -.. c:function:: int PyAsync_AddAwait(PyObject *asyncobj, PyObject *coroutine, PyAsync_Callback result_callback, PyAsync_Error error_callback) +.. c:function:: int PyAsync_AddAwait(PyObject *asyncobj, PyObject *awaitable, PyAsync_Callback result_callback, PyAsync_Error error_callback) - Append *coroutine* to the operations to await when *asyncobj* executes. + Append *awaitable* to the operations to await when *asyncobj* executes. Operations execute in the order they were added. This function stores a - strong reference to *coroutine*, which must be a non-``NULL`` awaitable - object distinct from *asyncobj*. + strong reference to *awaitable*, which must be a non-``NULL`` + :term:`awaitable` object distinct from *asyncobj*; anything else is + rejected with :exc:`TypeError` when this function is called, not when + *asyncobj* is awaited. On successful completion, *result_callback* is invoked with *asyncobj* and the operation's return value. If *result_callback* is ``NULL``, the @@ -246,8 +298,9 @@ Awaiting and callbacks *error_callback* is invoked. If *error_callback* is ``NULL``, the exception propagates to the caller awaiting *asyncobj*. - Callbacks may append further operations to *asyncobj*. Adding operations - to a completed object raises :exc:`RuntimeError`. + Callbacks may append further operations to *asyncobj*; they are added + to the end of the queue and run after all previously queued operations. + Adding operations to a completed object raises :exc:`RuntimeError`. Return ``0`` on success, or ``-1`` with an exception set on failure. @@ -256,7 +309,8 @@ Awaiting and callbacks Like :c:func:`PyAsync_AddAwait`, but steals a reference to *expr* on both success and failure. If *expr* is ``NULL``, return ``-1`` without - changing the current exception. + changing the current exception, which is assumed to have been set by + whatever produced *expr*. Otherwise, return ``0`` on success, or ``-1`` with an exception set on failure. @@ -276,28 +330,51 @@ Awaiting and callbacks Clear the queued operations and callbacks, releasing their references and discarding the current await iterator. This also suppresses the - :exc:`ResourceWarning` for an object that was never awaited. + :exc:`RuntimeWarning` for an object that was never awaited. - This function may be called from a callback. Further operations may be - added to *asyncobj* until it completes. This function cannot fail. + This function may be called from a callback, in which case the object + completes once the callback returns, unless the callback adds new + operations. Further operations may be added to *asyncobj* until it + completes. A pending :keyword:`async with` exit (see + :c:func:`PyAsync_AsyncWith`) is not cleared and still runs. + The currently awaited operation, if any, is dropped without being + closed. This function is unrelated to :mod:`asyncio` task cancellation, + which reaches the object through ``throw()``. It cannot fail. -.. c:function:: int PyAsync_AsyncWith(PyObject *asyncobj, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) - Append an :keyword:`async with` operation using *context_manager*. - Await its ``__aenter__`` method, invoke *result_callback*, and await its - ``__aexit__`` method. *result_callback* receives a :c:type:`PyAsync` - object representing the context manager operation and the value - returned by ``__aenter__``. It may be ``NULL``. +.. c:function:: int PyAsync_AsyncWith(PyObject *asyncobj, PyObject *context_manager, PyAsync_Callback result_callback, PyAsync_Error error_callback) - If *result_callback* fails, its exception is passed to ``__aexit__``, - which may suppress it by returning a true value. An exception from - ``__aenter__`` propagates without invoking ``__aexit__``. + Append an :keyword:`async with` block using *context_manager*. When the + operation runs, the ``__aenter__`` method of *context_manager* is + awaited, *result_callback* is invoked, any operations it queued are + awaited, and finally the ``__aexit__`` method is awaited. + + *result_callback*, which may be ``NULL``, is the body of the block. It + receives a :c:type:`PyAsync` object representing the block and the + value returned by ``__aenter__``. The block object shares its result + and value storage with *asyncobj* (:c:func:`PyAsync_SetResult`, + :c:func:`PyAsync_SetValue` and the other storage functions act on + *asyncobj*), but has its own operation queue: operations added to the + block object run inside the block, before ``__aexit__``. Operations + added to *asyncobj* run after the block, as they would in Python. + + Once ``__aenter__`` has completed, ``__aexit__`` is always awaited, + whether the block completes normally, *result_callback* or an + operation inside the block raises, an exception is thrown into the + object (including :exc:`asyncio.CancelledError`), or the object is + closed. On an exception, ``__aexit__`` receives the exception and may + suppress it by returning a true value. An exception from + ``__aenter__`` propagates without invoking ``__aexit__``. If + ``__aexit__`` itself raises, the block's exception becomes the + ``__context__`` of the new one. + + *error_callback* handles exceptions from the operation as a whole, + including any exception left unsuppressed by ``__aexit__``. It receives + *asyncobj* and the exception. If it is ``NULL``, the exception + propagates to the caller awaiting *asyncobj*. - *error_callback* handles exceptions from the operation, including any - exception left unsuppressed by ``__aexit__``. It receives *asyncobj* - and the exception. If it is ``NULL``, the exception propagates to the - caller awaiting *asyncobj*. + Blocks may be nested by calling this function on a block object. Return ``0`` on success, or ``-1`` with an exception set on failure. @@ -307,6 +384,7 @@ Value storage Each :c:type:`PyAsync` object has one object slot and one arbitrary pointer slot for storing state between callbacks. Both slots are initially ``NULL``. +Code that needs more than one object stores a tuple or other container. .. c:function:: int PyAsync_SetValue(PyObject *asyncobj, PyObject *value) @@ -342,7 +420,10 @@ slot for storing state between callbacks. Both slots are initially ``NULL``. Limited API ----------- -All APIs in this PEP will be made part of the :ref:`Stable ABI ` in Python 3.16. +All APIs in this PEP are part of the :ref:`Limited API ` +and the :ref:`Stable ABI ` as of Python 3.16. +:c:func:`PyAsync_CheckExact` is a function rather than a macro for this +reason. Rationale @@ -444,7 +525,7 @@ for the extension. return 1; } - // Equivalent to def call_async(func): return await func() + // Equivalent to: async def call_async(func): return await func() static PyObject * call_async(PyObject *self, PyObject *func) /* METH_O */ { From 4282a6ac75d675d2bd5f881945c1cb3d72282d0c Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 9 Nov 2026 23:49:55 -0500 Subject: [PATCH 14/15] Several additions and changes. --- peps/pep-0852.rst | 142 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 119 insertions(+), 23 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 72a72cd38fe..0c3bb87f156 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -58,6 +58,24 @@ users. For example, ``mypyc`` still `does not support all asynchronous features `__, because they are forced to build coroutine types from scratch, which is difficult to do correctly. +This problem is not unique to ``mypyc``; it shows up across the extension +ecosystem: + +- PyO3 has had an `open tracking issue + `__ for ``async fn`` support since + 2021. Its experimental support relies on a coroutine type implemented by + PyO3 itself, which still has gaps; for example, it `does not support + `__ the asynchronous context + manager protocol. +- A contributor to ``pydantic-core`` `asked + `__ how to await a + Python coroutine from Rust, and was told by a maintainer that moving the + asynchronous logic out of Python "is *really* hard". +- pybind11 has had an `open feature request + `__ for defining + asynchronous functions since 2020, in which the author resorted to + hand-writing their own awaitable class in C++. + For extension developers, the current solution to this problem is to do one of three things: @@ -90,6 +108,8 @@ To `quote `__ Marc-André Lemburg: the interactions with the asyncio (and other) event loops. +.. _pep-852-no-third-party: + A third-party library is bad for both maintainers and users ----------------------------------------------------------- @@ -107,7 +127,7 @@ using C API binding libraries and not the C API itself. Unfortunately, these binding libraries cannot wrap APIs that don't come directly from CPython, so native libraries are effectively unusable to projects using other C API bindings. In fact, PyAwaitable uses a vendoring technique that would make it functionally -impossible to use it from something such as PyO3. +impossible to use from something such as PyO3. There have been `some `__ `discussions `__ about making this easier (both of which @@ -122,15 +142,15 @@ to quote :pep:`20`, "practicality beats purity". Why was a third-party library created in the first place? ********************************************************** -When PyAwaitable was first created in early 2024, the author of the implementation -(who is also the author of this proposal) was new to CPython contribution and -thus not in a great place to author a PEP or oversee an implementation of PyAwaitable -in CPython. A relatively simple implementation was possible without changes to -the CPython core, and as such, PyAwaitable was better fit as a library for the -time being -- though always with the intent of eventually becoming part of the -public C API. The author has since become a core developer, and as such, it is -now much more feasible for them to maintain a variant of PyAwaitable in CPython -directly. +When PyAwaitable was first created in early 2024, a relatively simple +implementation was possible without changes to the CPython core, and as +such, PyAwaitable was better fit as a library for the time being -- though +always with the intent of eventually becoming part of the public C API. +The idea was that a third-party library would iron out potential design +flaws in the API as a whole before they were immortalized into CPython, +and this was largely successful (see :ref:`the deviations from PyAwaitable +`); as such, the third-party library has served its purpose, +and it is now time to add it to the C API. It is also worth noting that the reference implementation of this proposal does require the use of some internal helper functions (that PyAwaitable, as a @@ -139,6 +159,41 @@ is a good fit for CPython, as one of the original arguments against adding such an API was that it was implementable using only the public C API. +The C API is allowed to be ergonomic where possible +*************************************************** + +When discussed in early 2024, the main argument against this proposal was that +it was implementable using only Python's public C API. To `quote `_ +Petr Viktorin: + + To set expectations: I don’t think this needs to be in CPython itself. + I know that sounds disappointing, but after all, Python is made to be + extendable; we want to have interesting projects outside the core. + +Since then, there is a much larger precedent for adding convenience functions +to Python's C API (as in, functions that can be implemented in terms of the +existing C API or in standard C). For example: + +- There have been numerous attribute access helpers added in recent years, such + as :c:func:`PyObject_GetOptionalAttr`, :c:func:`PySys_GetAttr`, + :c:func:`PyImport_ImportModuleAttr`, and related functions. +- :c:type:`PyMutex`, while useful for the :term:`free threading` effort, is + a generic locking interface that could have been implemented outside CPython's + core. +- Several private APIs that were promoted to public, such as the + :ref:`c-api-time`. +- The APIs described by :pep:`788` can largely be implemented through + :mod:`atexit` callbacks. + +APIs like these were added because Python's C API should be easy to use where +possible -- merely because something *can* be implemented using the +public C API does not mean that it's the best option for users, especially +given the difficulty of distributing native code that wraps Python's C API. +When something is tightly coupled to a core language feature (in this case, +``async``/``await``), it is typically a better fit for the C API and not for +a third-party library. + + Specification ============= @@ -219,22 +274,18 @@ A :c:type:`PyAsync` object behaves as a native coroutine object does: operation's error callback is invoked with the exception, and so on as described under :c:func:`PyAsync_AddAwait`. If the object has not started, or is between operations, the exception propagates and the object - completes. This is how :mod:`asyncio` cancellation reaches an awaiting - :c:type:`PyAsync` object: a C error callback can observe - :exc:`asyncio.CancelledError` exactly as an ``except`` clause in a - coroutine would. + completes. * ``close()`` throws :exc:`GeneratorExit` into the object. The currently awaited operation is closed, pending error callbacks and :keyword:`async with` exits observe :exc:`GeneratorExit`, and the object - completes. As with native coroutines, if the object suspends again while - handling :exc:`GeneratorExit`, ``close()`` raises :exc:`RuntimeError`. - Closing an object that has not started or has completed does nothing. + completes. Closing an object that has not started or has completed does + nothing. A :c:type:`PyAsync` object may not be resumed while it is already executing; doing so raises :exc:`ValueError`. Queue and storage mutations are -serialized per object, so on the :term:`free-threaded ` -build it is safe to add operations or set values from another thread, but -an object can only be driven by one thread at a time. +serialized per object, so on the :term:`free-threaded build` it is safe to +add operations or set values from another thread, but an object can only be +driven by one thread at a time. Awaiting and callbacks @@ -422,8 +473,6 @@ Limited API All APIs in this PEP are part of the :ref:`Limited API ` and the :ref:`Stable ABI ` as of Python 3.16. -:c:func:`PyAsync_CheckExact` is a function rather than a macro for this -reason. Rationale @@ -439,9 +488,17 @@ be reason for adding an ``asyncio`` C API in the future, its only goal is to make :keyword:`async def` functions possible to implement in the C API. +.. _pep-852-deviations: + Deviations from PyAwaitable --------------------------- +The primary purpose of making PyAwaitable a third-party library and not a +formal C API proposal (yet) was to identify the best design choices based on +user experience and feedback. As such, this proposal does not copy PyAwaitable's +API exactly, and instead describes a new API without the original design flaws +of PyAwaitable. + ``PyAsync`` instead of ``PyAwaitable`` ************************************** @@ -690,7 +747,46 @@ A reference implementation of this PEP can be found Rejected Ideas ============== -TBD. + +A single callback with a state number +------------------------------------- + +Instead of one callback per :keyword:`await`, Serhiy Storchaka suggested in +the `2023 discussion `__ a single +callback that is re-entered with a state number and dispatches with a +``switch``. This is closer to a generator API than to :keyword:`await`, and +pushes the state machine onto the user, which is the burden this PEP aims to +remove. Users who prefer that style can implement it on top of the proposed +API. + +In addition, a single function with a ``switch`` would likely encourage users +to use ``static`` variables, which have thread-safety implications that would +likely go against the efforts of the :term:`free threading` project. + + +C++20 coroutines +---------------- + +Steve Dower suggested in the same discussion that native coroutine support +is better left to languages that have it, such as C++20 via pybind11. The C +API is a C API, and most of the extension ecosystem (including binding +generators) cannot consume C++ interfaces. Nothing in this PEP prevents a +C++ wrapper from offering ``co_await`` on top of :c:type:`PyAsync`. + + +Reusing native coroutine objects +-------------------------------- + +Native coroutine objects (instances of :class:`types.CoroutineType`) are bound +to a Python frame with bytecode, and thus there is no way to construct one +around C code without emulating a frame. A new type is simpler and avoids +exposing frame internals. + + +Keeping the API in a third-party library +---------------------------------------- + +See :ref:`pep-852-no-third-party`. Open Issues From bf52e17fb7fb239b8f7cbabd78d82839d66293c1 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 9 Nov 2026 23:56:21 -0500 Subject: [PATCH 15/15] Final touch-ups. --- peps/pep-0852.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0852.rst b/peps/pep-0852.rst index 0c3bb87f156..bf430a89798 100644 --- a/peps/pep-0852.rst +++ b/peps/pep-0852.rst @@ -154,7 +154,7 @@ and it is now time to add it to the C API. It is also worth noting that the reference implementation of this proposal does require the use of some internal helper functions (that PyAwaitable, as a -third-party, had to copy and work around), which is more evidence that PyAwaitable +third-party library, had to copy and work around), which is more evidence that PyAwaitable is a good fit for CPython, as one of the original arguments against adding such an API was that it was implementable using only the public C API. @@ -163,7 +163,7 @@ The C API is allowed to be ergonomic where possible *************************************************** When discussed in early 2024, the main argument against this proposal was that -it was implementable using only Python's public C API. To `quote `_ +it was implementable using only Python's public C API. To `quote `__ Petr Viktorin: To set expectations: I don’t think this needs to be in CPython itself. @@ -278,7 +278,7 @@ A :c:type:`PyAsync` object behaves as a native coroutine object does: * ``close()`` throws :exc:`GeneratorExit` into the object. The currently awaited operation is closed, pending error callbacks and :keyword:`async with` exits observe :exc:`GeneratorExit`, and the object - completes. Closing an object that has not started or has completed does + completes. Closing an object that has not started or has completed does nothing. A :c:type:`PyAsync` object may not be resumed while it is already executing; @@ -521,7 +521,7 @@ This API uses a single ``PyObject *`` reference and ``void *`` pointer for value storage between callbacks instead of variadic argument lists that were stored as arrays. PyAwaitable used the latter because it was optimized for ergonomics when writing pure C code, but since this API is also designed to be accessible from -bindings generators (and FFIs) as well, a single pointer for each felt like the +binding generators (and FFIs) as well, a single pointer for each felt like the better call. PyAwaitable's array system also turned out to be a source of complexity and bugs, and avoiding extra complexity for CPython where possible is ideal. @@ -582,7 +582,7 @@ for the extension. return 1; } - // Equivalent to: async def call_async(func): return await func() + /* Equivalent to: async def call_async(func): return await func() */ static PyObject * call_async(PyObject *self, PyObject *func) /* METH_O */ {