feat: expose coroutine constructor

Cargo.toml

@@ -68,6 +68,9 @@ default = ["macros"]
 # Enables support for `async fn` for `#[pyfunction]` and `#[pymethods]`.
 experimental-async = ["macros", "pyo3-macros/experimental-async"]
 
+# Switch coroutine implementation to anyio instead of asyncio
+anyio = ["experimental-async"]
+
 # Enables pyo3::inspect module and additional type information on FromPyObject
 # and IntoPy traits
 experimental-inspect = []

guide/src/SUMMARY.md

@@ -6,35 +6,36 @@
 
 - [Getting started](getting-started.md)
 - [Using Rust from Python](rust-from-python.md)
-  - [Python modules](module.md)
-  - [Python functions](function.md)
-    - [Function signatures](function/signature.md)
-    - [Error handling](function/error-handling.md)
-  - [Python classes](class.md)
-    - [Class customizations](class/protocols.md)
-      - [Basic object customization](class/object.md)
-      - [Emulating numeric types](class/numeric.md)
-      - [Emulating callable objects](class/call.md)
+    - [Python modules](module.md)
+    - [Python functions](function.md)
+        - [Function signatures](function/signature.md)
+        - [Error handling](function/error-handling.md)
+    - [Python classes](class.md)
+        - [Class customizations](class/protocols.md)
+            - [Basic object customization](class/object.md)
+            - [Emulating numeric types](class/numeric.md)
+            - [Emulating callable objects](class/call.md)
 - [Calling Python from Rust](python-from-rust.md)
-  - [Python object types](types.md)
-  - [Python exceptions](exception.md)
-  - [Calling Python functions](python-from-rust/function-calls.md)
-  - [Executing existing Python code](python-from-rust/calling-existing-code.md)
+    - [Python object types](types.md)
+    - [Python exceptions](exception.md)
+    - [Calling Python functions](python-from-rust/function-calls.md)
+    - [Executing existing Python code](python-from-rust/calling-existing-code.md)
 - [Type conversions](conversions.md)
-  - [Mapping of Rust types to Python types](conversions/tables.md)
-  - [Conversion traits](conversions/traits.md)
+    - [Mapping of Rust types to Python types](conversions/tables.md)
+    - [Conversion traits](conversions/traits.md)
 - [Using `async` and `await`](async-await.md)
+    - [Awaiting Python awaitables](async-await/awaiting_python_awaitables)
 - [Parallelism](parallelism.md)
 - [Debugging](debugging.md)
 - [Features reference](features.md)
 - [Memory management](memory.md)
 - [Performance](performance.md)
 - [Advanced topics](advanced.md)
 - [Building and distribution](building-and-distribution.md)
-  - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md)
+    - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md)
 - [Useful crates](ecosystem.md)
-  - [Logging](ecosystem/logging.md)
-  - [Using `async` and `await`](ecosystem/async-await.md)
+    - [Logging](ecosystem/logging.md)
+    - [Using `async` and `await`](ecosystem/async-await.md)
 - [FAQ and troubleshooting](faq.md)
 
 ---

guide/src/async-await.md

@@ -5,7 +5,7 @@
 `#[pyfunction]` and `#[pymethods]` attributes also support `async fn`.
 
 ```rust
-# #![allow(dead_code)]
+# # ![allow(dead_code)]
 # #[cfg(feature = "experimental-async")] {
 use std::{thread, time::Duration};
 use futures::channel::oneshot;
@@ -24,25 +24,35 @@ async fn sleep(seconds: f64, result: Option<PyObject>) -> Option<PyObject> {
 # }
 ```
 
-*Python awaitables instantiated with this method can only be awaited in *asyncio* context. Other Python async runtime may be supported in the future.*
-
 ## `Send + 'static` constraint
 
-Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python object.
+Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python
+object.
 
-As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`.
+As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a
+signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`.
 
-However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` over exclusive ones `&mut self` to avoid racy borrow check failures at runtime.
+However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this
+means that the class instance is borrowed for as long as the returned future is not completed, even across yield points
+and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future
+is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust
+code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared
+borrows `&self` over exclusive ones `&mut self` to avoid racy borrow check failures at runtime.
 
 ## Implicit GIL holding
 
-Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet the GIL is held.
+Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the
+execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet
+the GIL is held.
 
-It is still possible to get a `Python` marker using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is reentrant and optimized, the cost will be negligible.
+It is still possible to get a `Python` marker
+using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is
+reentrant and optimized, the cost will be negligible.
 
 ## Release the GIL across `.await`
 
-There is currently no simple way to release the GIL when awaiting a future, *but solutions are currently in development*.
+There is currently no simple way to release the GIL when awaiting a future, *but solutions are currently in
+development*.
 
 Here is the advised workaround for now:
 
@@ -74,10 +84,12 @@ where
 
 ## Cancellation
 
-Cancellation on the Python side can be caught using [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) type, by annotating a function parameter with `#[pyo3(cancel_handle)]`.
+Cancellation on the Python side can be caught
+using [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) type, by annotating a function
+parameter with `#[pyo3(cancel_handle)]`.
 
 ```rust
-# #![allow(dead_code)]
+# # ![allow(dead_code)]
 # #[cfg(feature = "experimental-async")] {
 use futures::FutureExt;
 use pyo3::prelude::*;
@@ -93,10 +105,44 @@ async fn cancellable(#[pyo3(cancel_handle)] mut cancel: CancelHandle) {
 # }
 ```
 
+## *asyncio* vs. *anyio*
+
+By default, Python awaitables instantiated with `async fn` can only be awaited in *asyncio* context.
+
+PyO3 can also target [*anyio*](https://github.com/agronholm/anyio) with the dedicated `anyio` Cargo feature. With it
+enabled, `async fn` become awaitable both in *asyncio* or [*trio*](https://github.com/python-trio/trio) context.
+However, it requires to have the [*sniffio*](https://github.com/python-trio/sniffio) (or *anyio*) library installed.
+
 ## The `Coroutine` type
 
-To make a Rust future awaitable in Python, PyO3 defines a [`Coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.Coroutine.html) type, which implements the Python [coroutine protocol](https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine).
+To make a Rust future awaitable in Python, PyO3 defines
+a [`Coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.Coroutine.html) type, which implements the
+Python [coroutine protocol](https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine).
+
+Each `coroutine.send` call is translated to a `Future::poll` call. If
+a [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) parameter is declared, the exception
+passed to `coroutine.throw` call is stored in it and can be retrieved
+with [`CancelHandle::cancelled`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html#method.cancelled);
+otherwise, it cancels the Rust future, and the exception is reraised;
 
-Each `coroutine.send` call is translated to a `Future::poll` call. If a [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) parameter is declared, the exception passed to `coroutine.throw` call is stored in it and can be retrieved with [`CancelHandle::cancelled`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html#method.cancelled); otherwise, it cancels the Rust future, and the exception is reraised;
+Coroutine can also be instantiated directly
 
-*The type does not yet have a public constructor until the design is finalized.*
+```rust
+# # ![allow(dead_code)]
+use pyo3::prelude::*;
+use pyo3::coroutine::{CancelHandle, Coroutine};
+
+#[pyfunction]
+fn new_coroutine(py: Python<'_>) -> Coroutine {
+    let mut cancel = CancelHandle::new();
+    let throw_callback = cancel.throw_callback();
+    let future = async move {
+        cancel.cancelled().await;
+        PyResult::Ok(())
+    };
+    Coroutine::new("my_coro", future)
+        .with_qualname_prefix("MyClass")
+        .with_throw_callback(throw_callback)
+        .with_allow_threads(true)
+}
+```

guide/src/async-await/awaiting_python_awaitables.md

@@ -0,0 +1,62 @@
+# Awaiting Python awaitables
+
+Python awaitable can be awaited on Rust side
+using [`await_in_coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/function.await_in_coroutine).
+
+```rust
+# # ![allow(dead_code)]
+# #[cfg(feature = "experimental-async")] {
+use pyo3::{prelude::*, coroutine::await_in_coroutine};
+
+#[pyfunction]
+async fn wrap_awaitable(awaitable: PyObject) -> PyResult<PyObject> {
+    Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?.await
+}
+# }
+```
+
+Behind the scene, `await_in_coroutine` calls the `__await__` method of the Python awaitable (or `__iter__` for
+generator-based coroutine).
+
+## Restrictions
+
+As the name suggests, `await_in_coroutine` resulting future can only be awaited in coroutine context. Otherwise, it
+panics.
+
+```rust
+# # ![allow(dead_code)]
+# #[cfg(feature = "experimental-async")] {
+use pyo3::{prelude::*, coroutine::await_in_coroutine};
+
+#[pyfunction]
+fn block_on(awaitable: PyObject) -> PyResult<PyObject> {
+    let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?;
+    futures::executor::block_on(future) // ERROR: PyFuture must be awaited in coroutine context
+}
+# }
+```
+
+The future must also be the only one to be awaited at a time; it means that it's forbidden to await it in a `select!`.
+Otherwise, it panics.
+
+```rust
+# # ![allow(dead_code)]
+# #[cfg(feature = "experimental-async")] {
+use futures::FutureExt;
+use pyo3::{prelude::*, coroutine::await_in_coroutine};
+
+#[pyfunction]
+async fn select(awaitable: PyObject) -> PyResult<PyObject> {
+    let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?;
+    futures::select_biased! {
+        _ = std::future::pending::<()>().fuse() => unreachable!(),
+        res = future.fuse() => res, // ERROR: Python awaitable mixed with Rust future
+    }
+}
+# }
+```
+
+These restrictions exist because awaiting a `await_in_coroutine` future strongly binds it to the
+enclosing coroutine. The coroutine will then delegate its `send`/`throw`/`close` methods to the
+awaited future. If it was awaited in a `select!`, `Coroutine::send` would no able to know if
+the value passed would have to be delegated or not.

guide/src/building-and-distribution.md

@@ -62,6 +62,7 @@ There are many ways to go about this: it is possible to use `cargo` to build the
 PyO3 has some Cargo features to configure projects for building Python extension modules:
  - The `extension-module` feature, which must be enabled when building Python extension modules.
  - The `abi3` feature and its version-specific `abi3-pyXY` companions, which are used to opt-in to the limited Python API in order to support multiple Python versions in a single wheel.
+- The `anyio` feature, making PyO3 coroutines target [*anyio*](https://github.com/agronholm/anyio) instead of *asyncio*; either [*sniffio*](https://github.com/python-trio/sniffio) or *anyio* should be added as dependency of the Python extension.
 
 This section describes each of these packaging tools before describing how to build manually without them. It then proceeds with an explanation of the `extension-module` feature. Finally, there is a section describing PyO3's `abi3` features.
 

newsfragments/3610.added.md

@@ -0,0 +1 @@
+Add `#[pyo3(allow_threads)]` to release the GIL in (async) functions

newsfragments/3611.added.md

@@ -0,0 +1 @@
+Add `coroutine::await_in_coroutine` to await awaitables in coroutine context

newsfragments/3612.added.md

@@ -0,0 +1 @@
+Support anyio with a Cargo feature

newsfragments/3613.added.md

@@ -0,0 +1 @@
+Expose `Coroutine` constructor

pyo3-ffi/src/abstract_.rs

@@ -129,7 +129,11 @@ extern "C" {
     pub fn PyIter_Next(arg1: *mut PyObject) -> *mut PyObject;
     #[cfg(all(not(PyPy), Py_3_10))]
     #[cfg_attr(PyPy, link_name = "PyPyIter_Send")]
-    pub fn PyIter_Send(iter: *mut PyObject, arg: *mut PyObject, presult: *mut *mut PyObject);
+    pub fn PyIter_Send(
+        iter: *mut PyObject,
+        arg: *mut PyObject,
+        presult: *mut *mut PyObject,
+    ) -> c_int;
 
     #[cfg_attr(PyPy, link_name = "PyPyNumber_Check")]
     pub fn PyNumber_Check(o: *mut PyObject) -> c_int;

pyo3-macros-backend/src/attributes.rs

@@ -9,6 +9,7 @@ use syn::{
 };
 
 pub mod kw {
+    syn::custom_keyword!(allow_threads);
     syn::custom_keyword!(annotation);
     syn::custom_keyword!(attribute);
     syn::custom_keyword!(cancel_handle);