diff --git a/Cargo.toml b/Cargo.toml index 90bcc03c80c..ddb20021c8e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -68,6 +68,9 @@ default = ["macros"] # Enables support for `async fn` for `#[pyfunction]` and `#[pymethods]`. experimental-async = ["macros", "pyo3-macros/experimental-async"] +# Switch coroutine implementation to anyio instead of asyncio +anyio = ["experimental-async"] + # Enables pyo3::inspect module and additional type information on FromPyObject # and IntoPy traits experimental-inspect = [] diff --git a/guide/src/SUMMARY.md b/guide/src/SUMMARY.md index 4c22c26f587..f1e3eccea8a 100644 --- a/guide/src/SUMMARY.md +++ b/guide/src/SUMMARY.md @@ -6,24 +6,25 @@ - [Getting started](getting-started.md) - [Using Rust from Python](rust-from-python.md) - - [Python modules](module.md) - - [Python functions](function.md) - - [Function signatures](function/signature.md) - - [Error handling](function/error-handling.md) - - [Python classes](class.md) - - [Class customizations](class/protocols.md) - - [Basic object customization](class/object.md) - - [Emulating numeric types](class/numeric.md) - - [Emulating callable objects](class/call.md) + - [Python modules](module.md) + - [Python functions](function.md) + - [Function signatures](function/signature.md) + - [Error handling](function/error-handling.md) + - [Python classes](class.md) + - [Class customizations](class/protocols.md) + - [Basic object customization](class/object.md) + - [Emulating numeric types](class/numeric.md) + - [Emulating callable objects](class/call.md) - [Calling Python from Rust](python-from-rust.md) - - [Python object types](types.md) - - [Python exceptions](exception.md) - - [Calling Python functions](python-from-rust/function-calls.md) - - [Executing existing Python code](python-from-rust/calling-existing-code.md) + - [Python object types](types.md) + - [Python exceptions](exception.md) + - [Calling Python functions](python-from-rust/function-calls.md) + - [Executing existing Python code](python-from-rust/calling-existing-code.md) - [Type conversions](conversions.md) - - [Mapping of Rust types to Python types](conversions/tables.md) - - [Conversion traits](conversions/traits.md) + - [Mapping of Rust types to Python types](conversions/tables.md) + - [Conversion traits](conversions/traits.md) - [Using `async` and `await`](async-await.md) + - [Awaiting Python awaitables](async-await/awaiting_python_awaitables) - [Parallelism](parallelism.md) - [Debugging](debugging.md) - [Features reference](features.md) @@ -31,10 +32,10 @@ - [Performance](performance.md) - [Advanced topics](advanced.md) - [Building and distribution](building-and-distribution.md) - - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md) + - [Supporting multiple Python versions](building-and-distribution/multiple-python-versions.md) - [Useful crates](ecosystem.md) - - [Logging](ecosystem/logging.md) - - [Using `async` and `await`](ecosystem/async-await.md) + - [Logging](ecosystem/logging.md) + - [Using `async` and `await`](ecosystem/async-await.md) - [FAQ and troubleshooting](faq.md) --- diff --git a/guide/src/async-await.md b/guide/src/async-await.md index 06fa1580ad7..a8a00a4e2dc 100644 --- a/guide/src/async-await.md +++ b/guide/src/async-await.md @@ -5,7 +5,7 @@ `#[pyfunction]` and `#[pymethods]` attributes also support `async fn`. ```rust -# #![allow(dead_code)] +# # ![allow(dead_code)] # #[cfg(feature = "experimental-async")] { use std::{thread, time::Duration}; use futures::channel::oneshot; @@ -24,25 +24,35 @@ async fn sleep(seconds: f64, result: Option) -> Option { # } ``` -*Python awaitables instantiated with this method can only be awaited in *asyncio* context. Other Python async runtime may be supported in the future.* - ## `Send + 'static` constraint -Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python object. +Resulting future of an `async fn` decorated by `#[pyfunction]` must be `Send + 'static` to be embedded in a Python +object. -As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`. +As a consequence, `async fn` parameters and return types must also be `Send + 'static`, so it is not possible to have a +signature like `async fn does_not_compile<'py>(arg: Bound<'py, PyAny>) -> Bound<'py, PyAny>`. -However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this means that the class instance is borrowed for as long as the returned future is not completed, even across yield points and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared borrows `&self` over exclusive ones `&mut self` to avoid racy borrow check failures at runtime. +However, there is an exception for method receivers, so async methods can accept `&self`/`&mut self`. Note that this +means that the class instance is borrowed for as long as the returned future is not completed, even across yield points +and while waiting for I/O operations to complete. Hence, other methods cannot obtain exclusive borrows while the future +is still being polled. This is the same as how async methods in Rust generally work but it is more problematic for Rust +code interfacing with Python code due to pervasive shared mutability. This strongly suggests to prefer shared +borrows `&self` over exclusive ones `&mut self` to avoid racy borrow check failures at runtime. ## Implicit GIL holding -Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet the GIL is held. +Even if it is not possible to pass a `py: Python<'py>` parameter to `async fn`, the GIL is still held during the +execution of the future – it's also the case for regular `fn` without `Python<'py>`/`Bound<'py, PyAny>` parameter, yet +the GIL is held. -It is still possible to get a `Python` marker using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is reentrant and optimized, the cost will be negligible. +It is still possible to get a `Python` marker +using [`Python::with_gil`]({{#PYO3_DOCS_URL}}/pyo3/marker/struct.Python.html#method.with_gil); because `with_gil` is +reentrant and optimized, the cost will be negligible. ## Release the GIL across `.await` -There is currently no simple way to release the GIL when awaiting a future, *but solutions are currently in development*. +There is currently no simple way to release the GIL when awaiting a future, *but solutions are currently in +development*. Here is the advised workaround for now: @@ -74,10 +84,12 @@ where ## Cancellation -Cancellation on the Python side can be caught using [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) type, by annotating a function parameter with `#[pyo3(cancel_handle)]`. +Cancellation on the Python side can be caught +using [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) type, by annotating a function +parameter with `#[pyo3(cancel_handle)]`. ```rust -# #![allow(dead_code)] +# # ![allow(dead_code)] # #[cfg(feature = "experimental-async")] { use futures::FutureExt; use pyo3::prelude::*; @@ -93,10 +105,44 @@ async fn cancellable(#[pyo3(cancel_handle)] mut cancel: CancelHandle) { # } ``` +## *asyncio* vs. *anyio* + +By default, Python awaitables instantiated with `async fn` can only be awaited in *asyncio* context. + +PyO3 can also target [*anyio*](https://github.com/agronholm/anyio) with the dedicated `anyio` Cargo feature. With it +enabled, `async fn` become awaitable both in *asyncio* or [*trio*](https://github.com/python-trio/trio) context. +However, it requires to have the [*sniffio*](https://github.com/python-trio/sniffio) (or *anyio*) library installed. + ## The `Coroutine` type -To make a Rust future awaitable in Python, PyO3 defines a [`Coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.Coroutine.html) type, which implements the Python [coroutine protocol](https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine). +To make a Rust future awaitable in Python, PyO3 defines +a [`Coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.Coroutine.html) type, which implements the +Python [coroutine protocol](https://docs.python.org/3/library/collections.abc.html#collections.abc.Coroutine). + +Each `coroutine.send` call is translated to a `Future::poll` call. If +a [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) parameter is declared, the exception +passed to `coroutine.throw` call is stored in it and can be retrieved +with [`CancelHandle::cancelled`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html#method.cancelled); +otherwise, it cancels the Rust future, and the exception is reraised; -Each `coroutine.send` call is translated to a `Future::poll` call. If a [`CancelHandle`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html) parameter is declared, the exception passed to `coroutine.throw` call is stored in it and can be retrieved with [`CancelHandle::cancelled`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/struct.CancelHandle.html#method.cancelled); otherwise, it cancels the Rust future, and the exception is reraised; +Coroutine can also be instantiated directly -*The type does not yet have a public constructor until the design is finalized.* +```rust +# # ![allow(dead_code)] +use pyo3::prelude::*; +use pyo3::coroutine::{CancelHandle, Coroutine}; + +#[pyfunction] +fn new_coroutine(py: Python<'_>) -> Coroutine { + let mut cancel = CancelHandle::new(); + let throw_callback = cancel.throw_callback(); + let future = async move { + cancel.cancelled().await; + PyResult::Ok(()) + }; + Coroutine::new("my_coro", future) + .with_qualname_prefix("MyClass") + .with_throw_callback(throw_callback) + .with_allow_threads(true) +} +``` diff --git a/guide/src/async-await/awaiting_python_awaitables.md b/guide/src/async-await/awaiting_python_awaitables.md new file mode 100644 index 00000000000..febecc6a286 --- /dev/null +++ b/guide/src/async-await/awaiting_python_awaitables.md @@ -0,0 +1,62 @@ +# Awaiting Python awaitables + +Python awaitable can be awaited on Rust side +using [`await_in_coroutine`]({{#PYO3_DOCS_URL}}/pyo3/coroutine/function.await_in_coroutine). + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +async fn wrap_awaitable(awaitable: PyObject) -> PyResult { + Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?.await +} +# } +``` + +Behind the scene, `await_in_coroutine` calls the `__await__` method of the Python awaitable (or `__iter__` for +generator-based coroutine). + +## Restrictions + +As the name suggests, `await_in_coroutine` resulting future can only be awaited in coroutine context. Otherwise, it +panics. + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +fn block_on(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::executor::block_on(future) // ERROR: PyFuture must be awaited in coroutine context +} +# } +``` + +The future must also be the only one to be awaited at a time; it means that it's forbidden to await it in a `select!`. +Otherwise, it panics. + +```rust +# # ![allow(dead_code)] +# #[cfg(feature = "experimental-async")] { +use futures::FutureExt; +use pyo3::{prelude::*, coroutine::await_in_coroutine}; + +#[pyfunction] +async fn select(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + _ = std::future::pending::<()>().fuse() => unreachable!(), + res = future.fuse() => res, // ERROR: Python awaitable mixed with Rust future + } +} +# } +``` + +These restrictions exist because awaiting a `await_in_coroutine` future strongly binds it to the +enclosing coroutine. The coroutine will then delegate its `send`/`throw`/`close` methods to the +awaited future. If it was awaited in a `select!`, `Coroutine::send` would no able to know if +the value passed would have to be delegated or not. diff --git a/guide/src/building-and-distribution.md b/guide/src/building-and-distribution.md index 780f135e211..df2155d5266 100644 --- a/guide/src/building-and-distribution.md +++ b/guide/src/building-and-distribution.md @@ -62,6 +62,7 @@ There are many ways to go about this: it is possible to use `cargo` to build the PyO3 has some Cargo features to configure projects for building Python extension modules: - The `extension-module` feature, which must be enabled when building Python extension modules. - The `abi3` feature and its version-specific `abi3-pyXY` companions, which are used to opt-in to the limited Python API in order to support multiple Python versions in a single wheel. +- The `anyio` feature, making PyO3 coroutines target [*anyio*](https://github.com/agronholm/anyio) instead of *asyncio*; either [*sniffio*](https://github.com/python-trio/sniffio) or *anyio* should be added as dependency of the Python extension. This section describes each of these packaging tools before describing how to build manually without them. It then proceeds with an explanation of the `extension-module` feature. Finally, there is a section describing PyO3's `abi3` features. diff --git a/newsfragments/3610.added.md b/newsfragments/3610.added.md new file mode 100644 index 00000000000..3b1493c29c0 --- /dev/null +++ b/newsfragments/3610.added.md @@ -0,0 +1 @@ +Add `#[pyo3(allow_threads)]` to release the GIL in (async) functions \ No newline at end of file diff --git a/newsfragments/3611.added.md b/newsfragments/3611.added.md new file mode 100644 index 00000000000..75aa9aee8fd --- /dev/null +++ b/newsfragments/3611.added.md @@ -0,0 +1 @@ +Add `coroutine::await_in_coroutine` to await awaitables in coroutine context diff --git a/newsfragments/3612.added.md b/newsfragments/3612.added.md new file mode 100644 index 00000000000..4f5f2f24014 --- /dev/null +++ b/newsfragments/3612.added.md @@ -0,0 +1 @@ +Support anyio with a Cargo feature \ No newline at end of file diff --git a/newsfragments/3613.added.md b/newsfragments/3613.added.md new file mode 100644 index 00000000000..6016f54ffa2 --- /dev/null +++ b/newsfragments/3613.added.md @@ -0,0 +1 @@ +Expose `Coroutine` constructor \ No newline at end of file diff --git a/pyo3-ffi/src/abstract_.rs b/pyo3-ffi/src/abstract_.rs index b5bf9cc3d35..3cc5cd346ad 100644 --- a/pyo3-ffi/src/abstract_.rs +++ b/pyo3-ffi/src/abstract_.rs @@ -129,7 +129,11 @@ extern "C" { pub fn PyIter_Next(arg1: *mut PyObject) -> *mut PyObject; #[cfg(all(not(PyPy), Py_3_10))] #[cfg_attr(PyPy, link_name = "PyPyIter_Send")] - pub fn PyIter_Send(iter: *mut PyObject, arg: *mut PyObject, presult: *mut *mut PyObject); + pub fn PyIter_Send( + iter: *mut PyObject, + arg: *mut PyObject, + presult: *mut *mut PyObject, + ) -> c_int; #[cfg_attr(PyPy, link_name = "PyPyNumber_Check")] pub fn PyNumber_Check(o: *mut PyObject) -> c_int; diff --git a/pyo3-macros-backend/src/attributes.rs b/pyo3-macros-backend/src/attributes.rs index e91b3b8d9a2..000ec4b5979 100644 --- a/pyo3-macros-backend/src/attributes.rs +++ b/pyo3-macros-backend/src/attributes.rs @@ -9,6 +9,7 @@ use syn::{ }; pub mod kw { + syn::custom_keyword!(allow_threads); syn::custom_keyword!(annotation); syn::custom_keyword!(attribute); syn::custom_keyword!(cancel_handle); diff --git a/pyo3-macros-backend/src/method.rs b/pyo3-macros-backend/src/method.rs index 982cf62946e..ae80835477e 100644 --- a/pyo3-macros-backend/src/method.rs +++ b/pyo3-macros-backend/src/method.rs @@ -6,6 +6,7 @@ use syn::{ext::IdentExt, spanned::Spanned, Ident, Result}; use crate::utils::Ctx; use crate::{ + attributes, attributes::{FromPyWithAttribute, TextSignatureAttribute, TextSignatureAttributeValue}, deprecations::{Deprecation, Deprecations}, params::{impl_arg_params, Holders}, @@ -379,6 +380,7 @@ pub struct FnSpec<'a> { pub asyncness: Option, pub unsafety: Option, pub deprecations: Deprecations<'a>, + pub allow_threads: Option, } pub fn parse_method_receiver(arg: &syn::FnArg) -> Result { @@ -416,6 +418,7 @@ impl<'a> FnSpec<'a> { text_signature, name, signature, + allow_threads, .. } = options; @@ -461,6 +464,7 @@ impl<'a> FnSpec<'a> { asyncness: sig.asyncness, unsafety: sig.unsafety, deprecations, + allow_threads, }) } @@ -603,6 +607,21 @@ impl<'a> FnSpec<'a> { bail_spanned!(name.span() => "`cancel_handle` may only be specified once"); } } + if let Some(FnArg::Py(py_arg)) = self + .signature + .arguments + .iter() + .find(|arg| matches!(arg, FnArg::Py(_))) + { + ensure_spanned!( + self.asyncness.is_none(), + py_arg.ty.span() => "GIL token cannot be passed to async function" + ); + ensure_spanned!( + self.allow_threads.is_none(), + py_arg.ty.span() => "GIL cannot be held in function annotated with `allow_threads`" + ); + } if self.asyncness.is_some() { ensure_spanned!( @@ -612,8 +631,21 @@ impl<'a> FnSpec<'a> { } let rust_call = |args: Vec, holders: &mut Holders| { - let mut self_arg = || self.tp.self_arg(cls, ExtractErrorMode::Raise, holders, ctx); - + let allow_threads = self.allow_threads.is_some(); + let mut self_arg = || { + let self_arg = self.tp.self_arg(cls, ExtractErrorMode::Raise, holders, ctx); + if self_arg.is_empty() { + self_arg + } else { + let self_checker = holders.push_gil_refs_checker(self_arg.span()); + quote! { + #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), + } + } + }; + let arg_names = (0..args.len()) + .map(|i| format_ident!("arg_{}", i)) + .collect::>(); let call = if self.asyncness.is_some() { let throw_callback = if cancel_handle.is_some() { quote! { Some(__throw_callback) } @@ -625,9 +657,6 @@ impl<'a> FnSpec<'a> { Some(cls) => quote!(Some(<#cls as #pyo3_path::PyTypeInfo>::NAME)), None => quote!(None), }; - let arg_names = (0..args.len()) - .map(|i| format_ident!("arg_{}", i)) - .collect::>(); let future = match self.tp { FnType::Fn(SelfType::Receiver { mutable: false, .. }) => { quote! {{ @@ -645,28 +674,18 @@ impl<'a> FnSpec<'a> { } _ => { let self_arg = self_arg(); - if self_arg.is_empty() { - quote! { function(#(#args),*) } - } else { - let self_checker = holders.push_gil_refs_checker(self_arg.span()); - quote! { - function( - // NB #self_arg includes a comma, so none inserted here - #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), - #(#args),* - ) - } - } + quote!(function(#self_arg #(#args),*)) } }; let mut call = quote! {{ let future = #future; - #pyo3_path::impl_::coroutine::new_coroutine( - #pyo3_path::intern!(py, stringify!(#python_name)), - #qualname_prefix, - #throw_callback, + #pyo3_path::coroutine::Coroutine::new( + stringify!(#python_name), async move { #pyo3_path::impl_::wrap::OkWrap::wrap(future.await) }, ) + .with_qualname_prefix(#qualname_prefix) + .with_throw_callback(#throw_callback) + .with_allow_threads(#allow_threads) }}; if cancel_handle.is_some() { call = quote! {{ @@ -676,20 +695,21 @@ impl<'a> FnSpec<'a> { }}; } call - } else { + } else if allow_threads { let self_arg = self_arg(); - if self_arg.is_empty() { - quote! { function(#(#args),*) } + let (self_arg_name, self_arg_decl) = if self_arg.is_empty() { + (quote!(), quote!()) } else { - let self_checker = holders.push_gil_refs_checker(self_arg.span()); - quote! { - function( - // NB #self_arg includes a comma, so none inserted here - #pyo3_path::impl_::deprecations::inspect_type(#self_arg &#self_checker), - #(#args),* - ) - } - } + (quote!(__self,), quote! { let (__self,) = (#self_arg); }) + }; + quote! {{ + #self_arg_decl + #(let #arg_names = #args;)* + py.allow_threads(|| function(#self_arg_name #(#arg_names),*)) + }} + } else { + let self_arg = self_arg(); + quote!(function(#self_arg #(#args),*)) }; quotes::map_result_into_ptr(quotes::ok_wrap(call, ctx), ctx) }; diff --git a/pyo3-macros-backend/src/pyclass.rs b/pyo3-macros-backend/src/pyclass.rs index d9c84655b42..8ee1821e954 100644 --- a/pyo3-macros-backend/src/pyclass.rs +++ b/pyo3-macros-backend/src/pyclass.rs @@ -1174,6 +1174,7 @@ fn complex_enum_struct_variant_new<'a>( asyncness: None, unsafety: None, deprecations: Deprecations::new(ctx), + allow_threads: None, }; crate::pymethod::impl_py_method_def_new(&variant_cls_type, &spec, ctx) @@ -1199,6 +1200,7 @@ fn complex_enum_variant_field_getter<'a>( asyncness: None, unsafety: None, deprecations: Deprecations::new(ctx), + allow_threads: None, }; let property_type = crate::pymethod::PropertyType::Function { diff --git a/pyo3-macros-backend/src/pyfunction.rs b/pyo3-macros-backend/src/pyfunction.rs index 7c355533b83..f65a0597b58 100644 --- a/pyo3-macros-backend/src/pyfunction.rs +++ b/pyo3-macros-backend/src/pyfunction.rs @@ -91,6 +91,7 @@ pub struct PyFunctionOptions { pub signature: Option, pub text_signature: Option, pub krate: Option, + pub allow_threads: Option, } impl Parse for PyFunctionOptions { @@ -99,7 +100,8 @@ impl Parse for PyFunctionOptions { while !input.is_empty() { let lookahead = input.lookahead1(); - if lookahead.peek(attributes::kw::name) + if lookahead.peek(attributes::kw::allow_threads) + || lookahead.peek(attributes::kw::name) || lookahead.peek(attributes::kw::pass_module) || lookahead.peek(attributes::kw::signature) || lookahead.peek(attributes::kw::text_signature) @@ -121,6 +123,7 @@ impl Parse for PyFunctionOptions { } pub enum PyFunctionOption { + AllowThreads(attributes::kw::allow_threads), Name(NameAttribute), PassModule(attributes::kw::pass_module), Signature(SignatureAttribute), @@ -131,7 +134,9 @@ pub enum PyFunctionOption { impl Parse for PyFunctionOption { fn parse(input: ParseStream<'_>) -> Result { let lookahead = input.lookahead1(); - if lookahead.peek(attributes::kw::name) { + if lookahead.peek(attributes::kw::allow_threads) { + input.parse().map(PyFunctionOption::AllowThreads) + } else if lookahead.peek(attributes::kw::name) { input.parse().map(PyFunctionOption::Name) } else if lookahead.peek(attributes::kw::pass_module) { input.parse().map(PyFunctionOption::PassModule) @@ -171,6 +176,7 @@ impl PyFunctionOptions { } for attr in attrs { match attr { + PyFunctionOption::AllowThreads(allow_threads) => set_option!(allow_threads), PyFunctionOption::Name(name) => set_option!(name), PyFunctionOption::PassModule(pass_module) => set_option!(pass_module), PyFunctionOption::Signature(signature) => set_option!(signature), @@ -198,6 +204,7 @@ pub fn impl_wrap_pyfunction( ) -> syn::Result { check_generic(&func.sig)?; let PyFunctionOptions { + allow_threads, pass_module, name, signature, @@ -247,6 +254,7 @@ pub fn impl_wrap_pyfunction( python_name, signature, text_signature, + allow_threads, asyncness: func.sig.asyncness, unsafety: func.sig.unsafety, deprecations: Deprecations::new(ctx), diff --git a/pyo3-macros/src/lib.rs b/pyo3-macros/src/lib.rs index 64756a1c73b..65456334ac1 100644 --- a/pyo3-macros/src/lib.rs +++ b/pyo3-macros/src/lib.rs @@ -121,6 +121,7 @@ pub fn pymethods(attr: TokenStream, input: TokenStream) -> TokenStream { /// | `#[pyo3(name = "...")]` | Defines the name of the function in Python. | /// | `#[pyo3(text_signature = "...")]` | Defines the `__text_signature__` attribute of the function in Python. | /// | `#[pyo3(pass_module)]` | Passes the module containing the function as a `&PyModule` first argument to the function. | +/// | `#[pyo3(allow_threads)]` | Release the GIL in the function body, or each time the returned future is polled for `async fn` | /// /// For more on exposing functions see the [function section of the guide][1]. /// diff --git a/pytests/Cargo.toml b/pytests/Cargo.toml index 255094a6c40..d5865573dcd 100644 --- a/pytests/Cargo.toml +++ b/pytests/Cargo.toml @@ -7,7 +7,8 @@ edition = "2021" publish = false [dependencies] -pyo3 = { path = "../", features = ["extension-module"] } +futures = "0.3.29" +pyo3 = { path = "../", features = ["extension-module", "anyio"] } [build-dependencies] pyo3-build-config = { path = "../pyo3-build-config" } diff --git a/pytests/pyproject.toml b/pytests/pyproject.toml index aace57dd4d4..fe2e7361237 100644 --- a/pytests/pyproject.toml +++ b/pytests/pyproject.toml @@ -20,6 +20,7 @@ classifiers = [ [project.optional-dependencies] dev = [ + "anyio[trio]>=4.0", "gevent>=22.10.2; implementation_name == 'cpython'", "hypothesis>=3.55", "pytest-asyncio>=0.21", diff --git a/pytests/src/anyio.rs b/pytests/src/anyio.rs new file mode 100644 index 00000000000..8fbe6cf685e --- /dev/null +++ b/pytests/src/anyio.rs @@ -0,0 +1,34 @@ +use std::{task::Poll, thread, time::Duration}; + +use futures::{channel::oneshot, future::poll_fn}; +use pyo3::prelude::*; + +#[pyfunction] +async fn sleep(seconds: f64, result: Option) -> Option { + if seconds <= 0.0 { + let mut ready = false; + poll_fn(|cx| { + if ready { + return Poll::Ready(()); + } + ready = true; + cx.waker().wake_by_ref(); + Poll::Pending + }) + .await; + } else { + let (tx, rx) = oneshot::channel(); + thread::spawn(move || { + thread::sleep(Duration::from_secs_f64(seconds)); + tx.send(()).unwrap(); + }); + rx.await.unwrap(); + } + result +} + +#[pymodule] +pub fn anyio(_py: Python<'_>, m: &PyModule) -> PyResult<()> { + m.add_function(wrap_pyfunction!(sleep, m)?)?; + Ok(()) +} diff --git a/pytests/src/lib.rs b/pytests/src/lib.rs index cbd65c8012c..9d03ee719c6 100644 --- a/pytests/src/lib.rs +++ b/pytests/src/lib.rs @@ -2,6 +2,7 @@ use pyo3::prelude::*; use pyo3::types::PyDict; use pyo3::wrap_pymodule; +pub mod anyio; pub mod awaitable; pub mod buf_and_str; pub mod comparisons; @@ -19,6 +20,7 @@ pub mod subclassing; #[pymodule] fn pyo3_pytests(py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add_wrapped(wrap_pymodule!(anyio::anyio))?; m.add_wrapped(wrap_pymodule!(awaitable::awaitable))?; #[cfg(not(Py_LIMITED_API))] m.add_wrapped(wrap_pymodule!(buf_and_str::buf_and_str))?; @@ -41,6 +43,7 @@ fn pyo3_pytests(py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> { let sys = PyModule::import_bound(py, "sys")?; let sys_modules = sys.getattr("modules")?.downcast_into::()?; + sys_modules.set_item("pyo3_pytests.anyio", m.getattr("anyio")?)?; sys_modules.set_item("pyo3_pytests.awaitable", m.getattr("awaitable")?)?; sys_modules.set_item("pyo3_pytests.buf_and_str", m.getattr("buf_and_str")?)?; sys_modules.set_item("pyo3_pytests.comparisons", m.getattr("comparisons")?)?; diff --git a/pytests/tests/test_anyio.py b/pytests/tests/test_anyio.py new file mode 100644 index 00000000000..c48435bd2fc --- /dev/null +++ b/pytests/tests/test_anyio.py @@ -0,0 +1,14 @@ +import asyncio + +from pyo3_pytests.anyio import sleep +import trio + + +def test_asyncio(): + assert asyncio.run(sleep(0)) is None + assert asyncio.run(sleep(0.1, 42)) == 42 + + +def test_trio(): + assert trio.run(sleep, 0) is None + assert trio.run(sleep, 0.1, 42) == 42 diff --git a/src/coroutine.rs b/src/coroutine.rs index f2feab4af16..c870cb06376 100644 --- a/src/coroutine.rs +++ b/src/coroutine.rs @@ -1,5 +1,6 @@ //! Python coroutine implementation, used notably when wrapping `async fn` //! with `#[pyfunction]`/`#[pymethods]`. +use std::borrow::Cow; use std::{ future::Future, panic, @@ -11,28 +12,74 @@ use std::{ use pyo3_macros::{pyclass, pymethods}; use crate::{ - coroutine::{cancel::ThrowCallback, waker::AsyncioWaker}, - exceptions::{PyAttributeError, PyRuntimeError, PyStopIteration}, + coroutine::waker::CoroutineWaker, + exceptions::{PyGeneratorExit, PyRuntimeError, PyStopIteration}, + marker::Ungil, panic::PanicException, - types::{string::PyStringMethods, PyIterator, PyString}, - Bound, IntoPy, Py, PyAny, PyErr, PyObject, PyResult, Python, + types::PyString, + Bound, IntoPy, Py, PyErr, PyObject, PyResult, Python, }; -pub(crate) mod cancel; +#[cfg(feature = "anyio")] +mod anyio; +mod asyncio; +mod awaitable; +mod cancel; +#[cfg(feature = "anyio")] +mod trio; mod waker; -pub use cancel::CancelHandle; +pub use awaitable::await_in_coroutine; +pub use cancel::{CancelHandle, ThrowCallback}; const COROUTINE_REUSED_ERROR: &str = "cannot reuse already awaited coroutine"; +pub(crate) enum CoroOp { + Send(PyObject), + Throw(PyObject), + Close, +} + +trait CoroutineFuture: Send { + fn poll( + self: Pin<&mut Self>, + py: Python<'_>, + waker: &Waker, + allow_threads: bool, + ) -> Poll>; +} + +impl CoroutineFuture for F +where + F: Future> + Send + Ungil, + T: IntoPy + Send + Ungil, + E: Into + Send + Ungil, +{ + fn poll( + self: Pin<&mut Self>, + py: Python<'_>, + waker: &Waker, + allow_threads: bool, + ) -> Poll> { + if allow_threads { + py.allow_threads(|| self.poll(&mut Context::from_waker(waker))) + } else { + self.poll(&mut Context::from_waker(waker)) + } + .map_ok(|obj| obj.into_py(py)) + .map_err(Into::into) + } +} + /// Python coroutine wrapping a [`Future`]. #[pyclass(crate = "crate")] pub struct Coroutine { - name: Option>, + future: Option>>, + name: Cow<'static, str>, qualname_prefix: Option<&'static str>, throw_callback: Option, - future: Option> + Send>>>, - waker: Option>, + allow_threads: bool, + waker: Option>, } impl Coroutine { @@ -41,120 +88,125 @@ impl Coroutine { /// Coroutine `send` polls the wrapped future, ignoring the value passed /// (should always be `None` anyway). /// - /// `Coroutine `throw` drop the wrapped future and reraise the exception passed - pub(crate) fn new( - name: Option>, - qualname_prefix: Option<&'static str>, - throw_callback: Option, - future: F, - ) -> Self + /// `Coroutine `throw` drop the wrapped future and reraise the exception passed. + pub fn new(name: impl Into>, future: F) -> Self where - F: Future> + Send + 'static, - T: IntoPy, - E: Into, + F: Future> + Send + Ungil + 'static, + T: IntoPy + Send + Ungil, + E: Into + Send + Ungil, { - let wrap = async move { - let obj = future.await.map_err(Into::into)?; - // SAFETY: GIL is acquired when future is polled (see `Coroutine::poll`) - Ok(obj.into_py(unsafe { Python::assume_gil_acquired() })) - }; Self { - name, - qualname_prefix, - throw_callback, - future: Some(Box::pin(wrap)), + future: Some(Box::pin(future)), + name: name.into(), + qualname_prefix: None, + throw_callback: None, + allow_threads: false, waker: None, } } - fn poll(&mut self, py: Python<'_>, throw: Option) -> PyResult { + /// Set a prefix for `__qualname__`, which will be joined with a "." + pub fn with_qualname_prefix(mut self, prefix: impl Into>) -> Self { + self.qualname_prefix = prefix.into(); + self + } + + /// Register a callback for coroutine `throw` method. + /// + /// The exception passed to `throw` is then redirected to this callback, notifying the + /// associated [`CancelHandle`], without being reraised. + pub fn with_throw_callback(mut self, callback: impl Into>) -> Self { + self.throw_callback = callback.into(); + self + } + + /// Release the GIL while polling the future wrapped. + pub fn with_allow_threads(mut self, allow_threads: bool) -> Self { + self.allow_threads = allow_threads; + self + } + + fn poll_inner(&mut self, py: Python<'_>, mut op: CoroOp) -> PyResult { // raise if the coroutine has already been run to completion let future_rs = match self.future { Some(ref mut fut) => fut, None => return Err(PyRuntimeError::new_err(COROUTINE_REUSED_ERROR)), }; - // reraise thrown exception it - match (throw, &self.throw_callback) { - (Some(exc), Some(cb)) => cb.throw(exc), - (Some(exc), None) => { - self.close(); - return Err(PyErr::from_value_bound(exc.into_bound(py))); - } - (None, _) => {} + // if the future is not pending on a Python awaitable, + // execute throw callback or complete on close + if !matches!(self.waker, Some(ref w) if w.is_delegated(py)) { + match op { + send @ CoroOp::Send(_) => op = send, + CoroOp::Throw(exc) => match &self.throw_callback { + Some(cb) => { + cb.throw(exc.clone_ref(py)); + op = CoroOp::Send(py.None()); + } + None => return Err(PyErr::from_value_bound(exc.into_bound(py))), + }, + CoroOp::Close => return Err(PyGeneratorExit::new_err(py.None())), + }; } // create a new waker, or try to reset it in place if let Some(waker) = self.waker.as_mut().and_then(Arc::get_mut) { - waker.reset(); + waker.reset(op); } else { - self.waker = Some(Arc::new(AsyncioWaker::new())); + self.waker = Some(Arc::new(CoroutineWaker::new(op))); } - let waker = Waker::from(self.waker.clone().unwrap()); - // poll the Rust future and forward its results if ready + // poll the future and forward its results if ready; otherwise, yield from waker // polling is UnwindSafe because the future is dropped in case of panic - let poll = || future_rs.as_mut().poll(&mut Context::from_waker(&waker)); + let waker = Waker::from(self.waker.clone().unwrap()); + let poll = || future_rs.as_mut().poll(py, &waker, self.allow_threads); match panic::catch_unwind(panic::AssertUnwindSafe(poll)) { - Ok(Poll::Ready(res)) => { - self.close(); - return Err(PyStopIteration::new_err(res?)); - } - Err(err) => { - self.close(); - return Err(PanicException::from_panic_payload(err)); - } - _ => {} + Err(err) => Err(PanicException::from_panic_payload(err)), + Ok(Poll::Ready(res)) => Err(PyStopIteration::new_err(res?)), + Ok(Poll::Pending) => match self.waker.as_ref().unwrap().yield_(py) { + Ok(to_yield) => Ok(to_yield), + Err(err) => Err(err), + }, } - // otherwise, initialize the waker `asyncio.Future` - if let Some(future) = self.waker.as_ref().unwrap().initialize_future(py)? { - // `asyncio.Future` must be awaited; fortunately, it implements `__iter__ = __await__` - // and will yield itself if its result has not been set in polling above - if let Some(future) = PyIterator::from_bound_object(&future.as_borrowed()) - .unwrap() - .next() - { - // future has not been leaked into Python for now, and Rust code can only call - // `set_result(None)` in `Wake` implementation, so it's safe to unwrap - return Ok(future.unwrap().into()); - } + } + + fn poll(&mut self, py: Python<'_>, op: CoroOp) -> PyResult { + let result = self.poll_inner(py, op); + if result.is_err() { + // the Rust future is dropped, and the field set to `None` + // to indicate the coroutine has been run to completion + drop(self.future.take()); } - // if waker has been waken during future polling, this is roughly equivalent to - // `await asyncio.sleep(0)`, so just yield `None`. - Ok(py.None().into_py(py)) + result } } #[pymethods(crate = "crate")] impl Coroutine { #[getter] - fn __name__(&self, py: Python<'_>) -> PyResult> { - match &self.name { - Some(name) => Ok(name.clone_ref(py)), - None => Err(PyAttributeError::new_err("__name__")), - } + fn __name__<'py>(&self, py: Python<'py>) -> Bound<'py, PyString> { + PyString::new_bound(py, &self.name) } #[getter] - fn __qualname__(&self, py: Python<'_>) -> PyResult> { - match (&self.name, &self.qualname_prefix) { - (Some(name), Some(prefix)) => Ok(format!("{}.{}", prefix, name.bind(py).to_cow()?) - .as_str() - .into_py(py)), - (Some(name), None) => Ok(name.clone_ref(py)), - (None, _) => Err(PyAttributeError::new_err("__qualname__")), - } + fn __qualname__<'py>(&self, py: Python<'py>) -> PyResult> { + Ok(match &self.qualname_prefix { + Some(prefix) => PyString::new_bound(py, &format!("{}.{}", prefix, self.name)), + None => self.__name__(py), + }) } - fn send(&mut self, py: Python<'_>, _value: &Bound<'_, PyAny>) -> PyResult { - self.poll(py, None) + fn send(&mut self, py: Python<'_>, value: PyObject) -> PyResult { + self.poll(py, CoroOp::Send(value)) } fn throw(&mut self, py: Python<'_>, exc: PyObject) -> PyResult { - self.poll(py, Some(exc)) + self.poll(py, CoroOp::Throw(exc)) } - fn close(&mut self) { - // the Rust future is dropped, and the field set to `None` - // to indicate the coroutine has been run to completion - drop(self.future.take()); + fn close(&mut self, py: Python<'_>) -> PyResult<()> { + match self.poll(py, CoroOp::Close) { + Ok(_) => Ok(()), + Err(err) if err.is_instance_of::(py) => Ok(()), + Err(err) => Err(err), + } } fn __await__(self_: Py) -> Py { @@ -162,6 +214,6 @@ impl Coroutine { } fn __next__(&mut self, py: Python<'_>) -> PyResult { - self.poll(py, None) + self.poll(py, CoroOp::Send(py.None())) } } diff --git a/src/coroutine/anyio.rs b/src/coroutine/anyio.rs new file mode 100644 index 00000000000..d8d47fbd6a2 --- /dev/null +++ b/src/coroutine/anyio.rs @@ -0,0 +1,70 @@ +//! Coroutine implementation using sniffio to select the appropriate implementation, +//! compatible with anyio. +use crate::{ + coroutine::{asyncio::AsyncioWaker, trio::TrioWaker}, + exceptions::PyRuntimeError, + sync::GILOnceCell, + types::PyAnyMethods, + Bound, PyAny, PyErr, PyObject, PyResult, Python, +}; + +fn current_async_library(py: Python<'_>) -> PyResult> { + static CURRENT_ASYNC_LIBRARY: GILOnceCell = GILOnceCell::new(); + let import = || -> PyResult<_> { + let module = py.import_bound("sniffio")?; + Ok(module.getattr("current_async_library")?.into()) + }; + CURRENT_ASYNC_LIBRARY + .get_or_try_init(py, import)? + .bind(py) + .call0() +} + +fn unsupported(runtime: &str) -> PyErr { + PyRuntimeError::new_err(format!("unsupported runtime {rt}", rt = runtime)) +} + +/// Sniffio/anyio-compatible coroutine waker. +/// +/// Polling a Rust future calls `sniffio.current_async_library` to select the appropriate +/// implementation, either asyncio or trio. +pub(super) enum AnyioWaker { + /// [`AsyncioWaker`] + Asyncio(AsyncioWaker), + /// [`TrioWaker`] + Trio(TrioWaker), +} + +impl AnyioWaker { + pub(super) fn new(py: Python<'_>) -> PyResult { + let sniffed = current_async_library(py)?; + match sniffed.extract()? { + "asyncio" => Ok(Self::Asyncio(AsyncioWaker::new(py)?)), + "trio" => Ok(Self::Trio(TrioWaker::new(py)?)), + rt => Err(unsupported(rt)), + } + } + + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + match self { + AnyioWaker::Asyncio(w) => w.yield_(py), + AnyioWaker::Trio(w) => w.yield_(py), + } + } + + pub(super) fn yield_waken(py: Python<'_>) -> PyResult { + let sniffed = current_async_library(py)?; + match sniffed.extract()? { + "asyncio" => AsyncioWaker::yield_waken(py), + "trio" => TrioWaker::yield_waken(py), + rt => Err(unsupported(rt)), + } + } + + pub(super) fn wake(&self, py: Python<'_>) -> PyResult<()> { + match self { + AnyioWaker::Asyncio(w) => w.wake(py), + AnyioWaker::Trio(w) => w.wake(py), + } + } +} diff --git a/src/coroutine/asyncio.rs b/src/coroutine/asyncio.rs new file mode 100644 index 00000000000..0049b56c0b8 --- /dev/null +++ b/src/coroutine/asyncio.rs @@ -0,0 +1,96 @@ +//! Coroutine implementation compatible with asyncio. +use pyo3_macros::pyfunction; + +use crate::{ + intern, + sync::GILOnceCell, + types::{PyAnyMethods, PyCFunction, PyIterator}, + wrap_pyfunction_bound, Bound, IntoPy, Py, PyAny, PyObject, PyResult, Python, +}; + +/// `asyncio.get_running_loop` +fn get_running_loop(py: Python<'_>) -> PyResult> { + static GET_RUNNING_LOOP: GILOnceCell = GILOnceCell::new(); + let import = || -> PyResult<_> { + let module = py.import_bound("asyncio")?; + Ok(module.getattr("get_running_loop")?.into()) + }; + GET_RUNNING_LOOP + .get_or_try_init(py, import)? + .bind(py) + .call0() +} + +/// Asyncio-compatible coroutine waker. +/// +/// Polling a Rust future yields an `asyncio.Future`, whose `set_result` method is called +/// when `Waker::wake` is called. +pub(super) struct AsyncioWaker { + event_loop: PyObject, + future: PyObject, +} + +impl AsyncioWaker { + pub(super) fn new(py: Python<'_>) -> PyResult { + let event_loop = get_running_loop(py)?.into_py(py); + let future = event_loop.call_method0(py, "create_future")?; + Ok(Self { event_loop, future }) + } + + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + let __await__; + // `asyncio.Future` must be awaited; in normal case, it implements `__iter__ = __await__`, + // but `create_future` may have been overriden + let mut iter = match PyIterator::from_bound_object(self.future.bind(py)) { + Ok(iter) => iter, + Err(_) => { + __await__ = self.future.call_method0(py, intern!(py, "__await__"))?; + PyIterator::from_bound_object(__await__.bind(py))? + } + }; + // future has not been wakened (because `yield_waken` would have been called + // otherwise), so it is expected to yield itself + Ok(iter.next().expect("future didn't yield")?.into_py(py)) + } + + #[allow(clippy::unnecessary_wraps)] + pub(super) fn yield_waken(py: Python<'_>) -> PyResult { + Ok(py.None()) + } + + pub(super) fn wake(&self, py: Python<'_>) -> PyResult<()> { + static RELEASE_WAITER: GILOnceCell> = GILOnceCell::new(); + let release_waiter = RELEASE_WAITER.get_or_try_init(py, || { + wrap_pyfunction_bound!(release_waiter, py).map(Into::into) + })?; + // `Future.set_result` must be called in event loop thread, + // so it requires `call_soon_threadsafe` + let call_soon_threadsafe = self.event_loop.call_method1( + py, + intern!(py, "call_soon_threadsafe"), + (release_waiter, &self.future), + ); + if let Err(err) = call_soon_threadsafe { + // `call_soon_threadsafe` will raise if the event loop is closed; + // instead of catching an unspecific `RuntimeError`, check directly if it's closed. + let is_closed = self.event_loop.call_method0(py, "is_closed")?; + if !is_closed.extract(py)? { + return Err(err); + } + } + Ok(()) + } +} + +/// Call `future.set_result` if the future is not done. +/// +/// Future can be cancelled by the event loop before being wakened. +/// See +#[pyfunction(crate = "crate")] +fn release_waiter(future: &Bound<'_, PyAny>) -> PyResult<()> { + let done = future.call_method0(intern!(future.py(), "done"))?; + if !done.extract::()? { + future.call_method1(intern!(future.py(), "set_result"), (future.py().None(),))?; + } + Ok(()) +} diff --git a/src/coroutine/awaitable.rs b/src/coroutine/awaitable.rs new file mode 100644 index 00000000000..9eee03a0f07 --- /dev/null +++ b/src/coroutine/awaitable.rs @@ -0,0 +1,161 @@ +use std::{ + future::Future, + pin::Pin, + task::{Context, Poll}, +}; + +use super::waker::try_delegate; +use crate::{ + coroutine::CoroOp, + exceptions::{PyAttributeError, PyTypeError}, + intern, + sync::GILOnceCell, + types::{PyAnyMethods, PyIterator, PyTypeMethods}, + Bound, PyAny, PyErr, PyObject, PyResult, Python, +}; + +const NOT_IN_COROUTINE_CONTEXT: &str = "PyFuture must be awaited in coroutine context"; + +fn is_awaitable(obj: &Bound<'_, PyAny>) -> PyResult { + static IS_AWAITABLE: GILOnceCell = GILOnceCell::new(); + let import = || { + PyResult::Ok( + obj.py() + .import_bound("inspect")? + .getattr("isawaitable")? + .into(), + ) + }; + IS_AWAITABLE + .get_or_try_init(obj.py(), import)? + .call1(obj.py(), (obj,))? + .extract(obj.py()) +} + +pub(crate) enum YieldOrReturn { + Return(PyObject), + Yield(PyObject), +} + +pub(crate) fn delegate( + py: Python<'_>, + await_impl: PyObject, + op: &CoroOp, +) -> PyResult { + match op { + CoroOp::Send(obj) => { + cfg_if::cfg_if! { + if #[cfg(all(Py_3_10, not(PyPy), not(Py_LIMITED_API)))] { + let mut result = std::ptr::null_mut(); + match unsafe { crate::ffi::PyIter_Send(await_impl.as_ptr(), obj.as_ptr(), &mut result) } + { + -1 => Err(PyErr::take(py).unwrap()), + 0 => Ok(YieldOrReturn::Return(unsafe { + PyObject::from_owned_ptr(py, result) + })), + 1 => Ok(YieldOrReturn::Yield(unsafe { + PyObject::from_owned_ptr(py, result) + })), + _ => unreachable!(), + } + } else { + let send = intern!(py, "send"); + if obj.is_none(py) || !await_impl.bind(py).hasattr(send).unwrap_or(false) { + await_impl.call_method0(py, intern!(py, "__next__")) + } else { + await_impl.call_method1(py, send, (obj,)) + } + .map(YieldOrReturn::Yield) + } + } + } + CoroOp::Throw(exc) => { + let throw = intern!(py, "throw"); + if await_impl.bind(py).hasattr(throw).unwrap_or(false) { + await_impl + .call_method1(py, throw, (exc,)) + .map(YieldOrReturn::Yield) + } else { + Err(PyErr::from_value_bound(exc.bind(py).clone())) + } + } + CoroOp::Close => { + let close = intern!(py, "close"); + if await_impl.bind(py).hasattr(close).unwrap_or(false) { + await_impl + .call_method0(py, close) + .map(YieldOrReturn::Return) + } else { + Ok(YieldOrReturn::Return(py.None())) + } + } + } +} + +struct AwaitImpl(PyObject); + +impl AwaitImpl { + fn new(obj: &Bound<'_, PyAny>) -> PyResult { + let __await__ = intern!(obj.py(), "__await__"); + match obj.call_method0(__await__) { + Ok(iter) => Ok(Self(iter.unbind())), + Err(err) if err.is_instance_of::(obj.py()) => { + if obj.hasattr(__await__)? { + Err(err) + } else if is_awaitable(obj)? { + Ok(Self( + PyIterator::from_bound_object(obj)?.unbind().into_any(), + )) + } else { + Err(PyTypeError::new_err(format!( + "object {tp} can't be used in 'await' expression", + tp = obj.get_type().name()? + ))) + } + } + Err(err) => Err(err), + } + } +} + +impl Future for AwaitImpl { + type Output = PyResult; + + fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + match try_delegate(cx.waker(), self.0.clone()) { + Some(poll) => poll, + None => panic!("{}", NOT_IN_COROUTINE_CONTEXT), + } + } +} + +/// Allows awaiting arbitrary Python awaitable inside PyO3 coroutine context, e.g. async pyfunction. +/// +/// Awaiting the resulting future will panic if it's not done in coroutine context. +/// However, the future can be instantiated outside of coroutine context. +/// +/// ```rust +/// use pyo3::{coroutine::await_in_coroutine, prelude::*, py_run, wrap_pyfunction_bound}; +/// +/// # fn main() { +/// #[pyfunction] +/// async fn wrap_awaitable(awaitable: PyObject) -> PyResult { +/// let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; +/// future.await +/// } +/// Python::with_gil(|py| { +/// let wrap_awaitable = wrap_pyfunction_bound!(wrap_awaitable, py).unwrap(); +/// let test = r#" +/// import asyncio +/// assert asyncio.run(wrap_awaitable(asyncio.sleep(1, result=42))) == 42 +/// "#; +/// py_run!(py, wrap_awaitable, test); +/// }) +/// # } +/// ``` +/// ```rust +pub fn await_in_coroutine( + obj: &Bound<'_, PyAny>, +) -> PyResult> + Send + Sync + 'static> { + AwaitImpl::new(obj) +} diff --git a/src/coroutine/cancel.rs b/src/coroutine/cancel.rs index 2b10fb9a438..7ca52de8958 100644 --- a/src/coroutine/cancel.rs +++ b/src/coroutine/cancel.rs @@ -1,9 +1,13 @@ -use crate::{Py, PyAny, PyObject}; +use std::{ + future::Future, + pin::Pin, + sync::Arc, + task::{Context, Poll, Waker}, +}; + use parking_lot::Mutex; -use std::future::Future; -use std::pin::Pin; -use std::sync::Arc; -use std::task::{Context, Poll, Waker}; + +use crate::PyObject; #[derive(Debug, Default)] struct Inner { @@ -45,30 +49,29 @@ impl CancelHandle { /// Retrieve the exception thrown in the associated coroutine. pub async fn cancelled(&mut self) -> PyObject { + // TODO use `std::future::poll_fn` with MSRV 1.64+ + struct Cancelled<'a>(&'a mut CancelHandle); + + impl Future for Cancelled<'_> { + type Output = PyObject; + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + self.0.poll_cancelled(cx) + } + } Cancelled(self).await } - #[doc(hidden)] + /// Instantiate a [`ThrowCallback`] associated to this cancel handle. pub fn throw_callback(&self) -> ThrowCallback { ThrowCallback(self.0.clone()) } } -// Because `poll_fn` is not available in MSRV -struct Cancelled<'a>(&'a mut CancelHandle); - -impl Future for Cancelled<'_> { - type Output = PyObject; - fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - self.0.poll_cancelled(cx) - } -} - -#[doc(hidden)] +/// Callback for coroutine `throw` method, notifying the associated [`CancelHandle`] pub struct ThrowCallback(Arc>); impl ThrowCallback { - pub(super) fn throw(&self, exc: Py) { + pub(super) fn throw(&self, exc: PyObject) { let mut inner = self.0.lock(); inner.exception = Some(exc); if let Some(waker) = inner.waker.take() { diff --git a/src/coroutine/trio.rs b/src/coroutine/trio.rs new file mode 100644 index 00000000000..45361cdc0c7 --- /dev/null +++ b/src/coroutine/trio.rs @@ -0,0 +1,89 @@ +//! Coroutine implementation compatible with trio. +use pyo3_macros::pyfunction; + +use crate::{ + intern, + sync::GILOnceCell, + types::{PyAnyMethods, PyCFunction, PyIterator}, + wrap_pyfunction_bound, Bound, Py, PyAny, PyObject, PyResult, Python, +}; + +struct Trio { + cancel_shielded_checkpoint: PyObject, + current_task: PyObject, + current_trio_token: PyObject, + reschedule: PyObject, + succeeded: PyObject, + wait_task_rescheduled: PyObject, +} +impl Trio { + fn get(py: Python<'_>) -> PyResult<&Self> { + static TRIO: GILOnceCell = GILOnceCell::new(); + TRIO.get_or_try_init(py, || { + let module = py.import_bound("trio.lowlevel")?; + Ok(Self { + cancel_shielded_checkpoint: module.getattr("cancel_shielded_checkpoint")?.into(), + current_task: module.getattr("current_task")?.into(), + current_trio_token: module.getattr("current_trio_token")?.into(), + reschedule: module.getattr("reschedule")?.into(), + succeeded: module.getattr("Abort")?.getattr("SUCCEEDED")?.into(), + wait_task_rescheduled: module.getattr("wait_task_rescheduled")?.into(), + }) + }) + } +} + +fn yield_from(coro_func: &PyAny) -> PyResult { + PyIterator::from_object(coro_func.call_method0("__await__")?)? + .next() + .expect("cancel_shielded_checkpoint didn't yield") + .map(Into::into) +} + +/// Asyncio-compatible coroutine waker. +/// +/// Polling a Rust future yields `trio.lowlevel.wait_task_rescheduled()`, while `Waker::wake` +/// reschedule the current task. +pub(super) struct TrioWaker { + task: PyObject, + token: PyObject, +} + +impl TrioWaker { + pub(super) fn new(py: Python<'_>) -> PyResult { + let trio = Trio::get(py)?; + let task = trio.current_task.call0(py)?; + let token = trio.current_trio_token.call0(py)?; + Ok(Self { task, token }) + } + + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + static ABORT_FUNC: GILOnceCell> = GILOnceCell::new(); + let abort_func = ABORT_FUNC.get_or_try_init(py, || { + wrap_pyfunction_bound!(abort_func, py).map(Into::into) + })?; + let wait_task_rescheduled = Trio::get(py)? + .wait_task_rescheduled + .call1(py, (abort_func,))?; + yield_from(wait_task_rescheduled.as_ref(py)) + } + + pub(super) fn yield_waken(py: Python<'_>) -> PyResult { + let checkpoint = Trio::get(py)?.cancel_shielded_checkpoint.call0(py)?; + yield_from(checkpoint.as_ref(py)) + } + + pub(super) fn wake(&self, py: Python<'_>) -> PyResult<()> { + self.token.call_method1( + py, + intern!(py, "run_sync_soon"), + (&Trio::get(py)?.reschedule, &self.task), + )?; + Ok(()) + } +} + +#[pyfunction(crate = "crate")] +fn abort_func(py: Python<'_>, _arg: &Bound<'_, PyAny>) -> PyResult { + Ok(Trio::get(py)?.succeeded.clone()) +} diff --git a/src/coroutine/waker.rs b/src/coroutine/waker.rs index fc7c54e1f5a..4cc7ab1fc8a 100644 --- a/src/coroutine/waker.rs +++ b/src/coroutine/waker.rs @@ -1,106 +1,126 @@ -use crate::sync::GILOnceCell; -use crate::types::any::PyAnyMethods; -use crate::types::PyCFunction; -use crate::{intern, wrap_pyfunction_bound, Bound, Py, PyAny, PyObject, PyResult, Python}; -use pyo3_macros::pyfunction; -use std::sync::Arc; -use std::task::Wake; +use std::{ + cell::Cell, + sync::Arc, + task::{Poll, Wake, Waker}, +}; -/// Lazy `asyncio.Future` wrapper, implementing [`Wake`] by calling `Future.set_result`. -/// -/// asyncio future is let uninitialized until [`initialize_future`][1] is called. -/// If [`wake`][2] is called before future initialization (during Rust future polling), -/// [`initialize_future`][1] will return `None` (it is roughly equivalent to `asyncio.sleep(0)`) -/// -/// [1]: AsyncioWaker::initialize_future -/// [2]: AsyncioWaker::wake -pub struct AsyncioWaker(GILOnceCell>); +use crate::{ + coroutine::{ + awaitable::{delegate, YieldOrReturn}, + CoroOp, + }, + exceptions::PyStopIteration, + intern, + sync::GILOnceCell, + types::PyAnyMethods, + Bound, PyObject, PyResult, Python, +}; -impl AsyncioWaker { - pub(super) fn new() -> Self { - Self(GILOnceCell::new()) +cfg_if::cfg_if! { + if #[cfg(feature = "anyio")] { + type WakerImpl = crate::coroutine::anyio::AnyioWaker; + } else { + type WakerImpl = crate::coroutine::asyncio::AsyncioWaker; + } +} + +const MIXED_AWAITABLE_AND_FUTURE_ERROR: &str = "Python awaitable mixed with Rust future"; + +enum State { + Pending(WakerImpl), + Waken, + Delegated(PyObject), +} + +pub(super) struct CoroutineWaker { + state: GILOnceCell, + op: CoroOp, +} + +impl CoroutineWaker { + pub(super) fn new(op: CoroOp) -> Self { + Self { + state: GILOnceCell::new(), + op, + } + } + + pub(super) fn reset(&mut self, op: CoroOp) { + self.state.take(); + self.op = op; } - pub(super) fn reset(&mut self) { - self.0.take(); + pub(super) fn is_delegated(&self, py: Python<'_>) -> bool { + matches!(self.state.get(py), Some(State::Delegated(_))) } - pub(super) fn initialize_future<'py>( - &self, - py: Python<'py>, - ) -> PyResult>> { - let init = || LoopAndFuture::new(py).map(Some); - let loop_and_future = self.0.get_or_try_init(py, init)?.as_ref(); - Ok(loop_and_future.map(|LoopAndFuture { future, .. }| future.bind(py))) + pub(super) fn yield_(&self, py: Python<'_>) -> PyResult { + let init = || PyResult::Ok(State::Pending(WakerImpl::new(py)?)); + let state = self.state.get_or_try_init(py, init)?; + match state { + State::Pending(waker) => waker.yield_(py), + State::Waken => WakerImpl::yield_waken(py), + State::Delegated(obj) => Ok(obj.clone_ref(py)), + } + } + + fn delegate(&self, py: Python<'_>, await_impl: PyObject) -> Poll> { + match delegate(py, await_impl, &self.op) { + Ok(YieldOrReturn::Yield(obj)) => { + let delegated = self.state.set(py, State::Delegated(obj)); + assert!(delegated.is_ok(), "{}", MIXED_AWAITABLE_AND_FUTURE_ERROR); + Poll::Pending + } + Ok(YieldOrReturn::Return(obj)) => Poll::Ready(Ok(obj)), + Err(err) if err.is_instance_of::(py) => Poll::Ready( + err.value_bound(py) + .getattr(intern!(py, "value")) + .map(Bound::unbind), + ), + Err(err) => Poll::Ready(Err(err)), + } } } -impl Wake for AsyncioWaker { +impl Wake for CoroutineWaker { fn wake(self: Arc) { self.wake_by_ref() } fn wake_by_ref(self: &Arc) { - Python::with_gil(|gil| { - if let Some(loop_and_future) = self.0.get_or_init(gil, || None) { - loop_and_future - .set_result(gil) - .expect("unexpected error in coroutine waker"); - } - }); + Python::with_gil(|gil| match WAKER_HACK.with(|cell| cell.take()) { + Some(WakerHack::Argument(await_impl)) => WAKER_HACK.with(|cell| { + let res = self.delegate(gil, await_impl); + cell.set(Some(WakerHack::Result(res))) + }), + Some(WakerHack::Result(_)) => unreachable!(), + None => match self.state.get_or_init(gil, || State::Waken) { + State::Pending(waker) => waker.wake(gil).expect("wake error"), + State::Waken => {} + State::Delegated(_) => panic!("{}", MIXED_AWAITABLE_AND_FUTURE_ERROR), + }, + }) } } -struct LoopAndFuture { - event_loop: PyObject, - future: PyObject, +enum WakerHack { + Argument(PyObject), + Result(Poll>), } -impl LoopAndFuture { - fn new(py: Python<'_>) -> PyResult { - static GET_RUNNING_LOOP: GILOnceCell = GILOnceCell::new(); - let import = || -> PyResult<_> { - let module = py.import_bound("asyncio")?; - Ok(module.getattr("get_running_loop")?.into()) - }; - let event_loop = GET_RUNNING_LOOP.get_or_try_init(py, import)?.call0(py)?; - let future = event_loop.call_method0(py, "create_future")?; - Ok(Self { event_loop, future }) - } - - fn set_result(&self, py: Python<'_>) -> PyResult<()> { - static RELEASE_WAITER: GILOnceCell> = GILOnceCell::new(); - let release_waiter = RELEASE_WAITER.get_or_try_init(py, || { - wrap_pyfunction_bound!(release_waiter, py).map(Bound::unbind) - })?; - // `Future.set_result` must be called in event loop thread, - // so it requires `call_soon_threadsafe` - let call_soon_threadsafe = self.event_loop.call_method1( - py, - intern!(py, "call_soon_threadsafe"), - (release_waiter, self.future.bind(py)), - ); - if let Err(err) = call_soon_threadsafe { - // `call_soon_threadsafe` will raise if the event loop is closed; - // instead of catching an unspecific `RuntimeError`, check directly if it's closed. - let is_closed = self.event_loop.call_method0(py, "is_closed")?; - if !is_closed.extract(py)? { - return Err(err); - } - } - Ok(()) - } +thread_local! { + static WAKER_HACK: Cell> = Cell::new(None); } -/// Call `future.set_result` if the future is not done. -/// -/// Future can be cancelled by the event loop before being waken. -/// See -#[pyfunction(crate = "crate")] -fn release_waiter(future: &Bound<'_, PyAny>) -> PyResult<()> { - let done = future.call_method0(intern!(future.py(), "done"))?; - if !done.extract::()? { - future.call_method1(intern!(future.py(), "set_result"), (future.py().None(),))?; +pub(crate) fn try_delegate( + waker: &Waker, + await_impl: PyObject, +) -> Option>> { + WAKER_HACK.with(|cell| cell.set(Some(WakerHack::Argument(await_impl)))); + waker.wake_by_ref(); + match WAKER_HACK.with(|cell| cell.take()) { + Some(WakerHack::Result(poll)) => Some(poll), + Some(WakerHack::Argument(_)) => None, + None => unreachable!(), } - Ok(()) } diff --git a/src/gil.rs b/src/gil.rs index e2f36037755..64c82d50221 100644 --- a/src/gil.rs +++ b/src/gil.rs @@ -1,14 +1,18 @@ //! Interaction with Python's global interpreter lock -use crate::impl_::not_send::{NotSend, NOT_SEND}; -use crate::{ffi, Python}; -use parking_lot::{const_mutex, Mutex, Once}; -use std::cell::Cell; #[cfg(debug_assertions)] use std::cell::RefCell; #[cfg(not(debug_assertions))] use std::cell::UnsafeCell; -use std::{mem, ptr::NonNull}; +use std::{cell::Cell, mem, ptr::NonNull}; + +use parking_lot::{const_mutex, Mutex, Once}; + +use crate::{ + ffi, + impl_::not_send::{NotSend, NOT_SEND}, + Python, +}; static START: Once = Once::new(); @@ -799,9 +803,10 @@ mod tests { #[test] #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled fn test_clone_without_gil() { - use crate::{Py, PyAny}; use std::{sync::Arc, thread}; + use crate::{Py, PyAny}; + // Some events for synchronizing static GIL_ACQUIRED: Event = Event::new(); static OBJECT_CLONED: Event = Event::new(); @@ -864,9 +869,10 @@ mod tests { #[test] #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled fn test_clone_in_other_thread() { - use crate::Py; use std::{sync::Arc, thread}; + use crate::Py; + // Some events for synchronizing static OBJECT_CLONED: Event = Event::new(); @@ -939,4 +945,47 @@ mod tests { POOL.update_counts(py); }) } + + #[cfg(feature = "macros")] + #[test] + fn allow_threads_fn() { + #[crate::pyfunction(allow_threads, crate = "crate")] + fn without_gil(_arg1: PyObject, _arg2: PyObject) { + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + } + Python::with_gil(|gil| { + let without_gil = crate::wrap_pyfunction_bound!(without_gil, gil).unwrap(); + crate::py_run!(gil, without_gil, "without_gil(..., ...)"); + }) + } + + #[cfg(feature = "experimental-async")] + #[cfg(not(target_arch = "wasm32"))] + #[test] + fn allow_threads_async_fn() { + #[crate::pyfunction(allow_threads, crate = "crate")] + async fn without_gil(_arg1: PyObject, _arg2: PyObject) { + use std::task::Poll; + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + let mut ready = false; + futures::future::poll_fn(|cx| { + if ready { + return Poll::Ready(()); + } + ready = true; + cx.waker().wake_by_ref(); + Poll::Pending + }) + .await; + GIL_COUNT.with(|c| assert_eq!(c.get(), 0)); + } + Python::with_gil(|gil| { + let without_gil = crate::wrap_pyfunction_bound!(without_gil, gil).unwrap(); + crate::py_run!( + gil, + without_gil, + "import asyncio; asyncio.run(without_gil(..., ...))" + ); + }) + } } diff --git a/src/impl_/coroutine.rs b/src/impl_/coroutine.rs index 1d3119400a0..50a6669be1f 100644 --- a/src/impl_/coroutine.rs +++ b/src/impl_/coroutine.rs @@ -1,36 +1,10 @@ -use std::{ - future::Future, - ops::{Deref, DerefMut}, -}; +use std::ops::{Deref, DerefMut}; use crate::{ - coroutine::{cancel::ThrowCallback, Coroutine}, - instance::Bound, - pycell::impl_::PyClassBorrowChecker, - pyclass::boolean_struct::False, - types::{PyAnyMethods, PyString}, - IntoPy, Py, PyAny, PyClass, PyErr, PyObject, PyResult, Python, + instance::Bound, pycell::impl_::PyClassBorrowChecker, pyclass::boolean_struct::False, + types::PyAnyMethods, Py, PyAny, PyClass, PyResult, Python, }; -pub fn new_coroutine( - name: &Bound<'_, PyString>, - qualname_prefix: Option<&'static str>, - throw_callback: Option, - future: F, -) -> Coroutine -where - F: Future> + Send + 'static, - T: IntoPy, - E: Into, -{ - Coroutine::new( - Some(name.clone().into()), - qualname_prefix, - throw_callback, - future, - ) -} - fn get_ptr(obj: &Py) -> *mut T { obj.get_class_object().get_ptr() } diff --git a/src/lib.rs b/src/lib.rs index e444912a63d..d4ce65c30e5 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -312,27 +312,29 @@ //! [Rust from Python]: https://github.com/PyO3/pyo3#using-rust-from-python //! [Features chapter of the guide]: https://pyo3.rs/latest/features.html#features-reference "Features Reference - PyO3 user guide" //! [`Ungil`]: crate::marker::Ungil -pub use crate::class::*; -pub use crate::conversion::{AsPyPointer, FromPyObject, IntoPy, ToPyObject}; #[allow(deprecated)] pub use crate::conversion::{FromPyPointer, PyTryFrom, PyTryInto}; -pub use crate::err::{ - DowncastError, DowncastIntoError, PyDowncastError, PyErr, PyErrArguments, PyResult, ToPyErr, -}; #[allow(deprecated)] pub use crate::gil::GILPool; #[cfg(not(any(PyPy, GraalPy)))] pub use crate::gil::{prepare_freethreaded_python, with_embedded_python_interpreter}; -pub use crate::instance::{Borrowed, Bound, Py, PyNativeType, PyObject}; -pub use crate::marker::Python; #[allow(deprecated)] pub use crate::pycell::PyCell; -pub use crate::pycell::{PyRef, PyRefMut}; -pub use crate::pyclass::PyClass; -pub use crate::pyclass_init::PyClassInitializer; -pub use crate::type_object::{PyTypeCheck, PyTypeInfo}; -pub use crate::types::PyAny; -pub use crate::version::PythonVersionInfo; +pub use crate::{ + class::*, + conversion::{AsPyPointer, FromPyObject, IntoPy, ToPyObject}, + err::{ + DowncastError, DowncastIntoError, PyDowncastError, PyErr, PyErrArguments, PyResult, ToPyErr, + }, + instance::{Borrowed, Bound, Py, PyNativeType, PyObject}, + marker::Python, + pycell::{PyRef, PyRefMut}, + pyclass::PyClass, + pyclass_init::PyClassInitializer, + type_object::{PyTypeCheck, PyTypeInfo}, + types::PyAny, + version::PythonVersionInfo, +}; pub(crate) mod ffi_ptr_ext; pub(crate) mod py_result_ext; @@ -347,7 +349,6 @@ pub(crate) mod sealed; /// once is resolved. pub mod class { pub use self::gc::{PyTraverseError, PyVisit}; - #[doc(hidden)] pub use self::methods::{ PyClassAttributeDef, PyGetterDef, PyMethodDef, PyMethodDefType, PyMethodType, PySetterDef, @@ -410,16 +411,15 @@ pub mod class { } } +#[cfg(all(feature = "macros", feature = "multiple-pymethods"))] +#[doc(hidden)] +pub use inventory; #[cfg(feature = "macros")] #[doc(hidden)] pub use { indoc, // Re-exported for py_run unindent, // Re-exported for py_run -}; - -#[cfg(all(feature = "macros", feature = "multiple-pymethods"))] -#[doc(hidden)] -pub use inventory; // Re-exported for `#[pyclass]` and `#[pymethods]` with `multiple-pymethods`. +}; // Re-exported for `#[pyclass]` and `#[pymethods]` with `multiple-pymethods`. /// Tests and helpers which reside inside PyO3's main library. Declared first so that macros /// are available in unit tests. @@ -462,14 +462,7 @@ pub mod type_object; pub mod types; mod version; -#[allow(unused_imports)] // with no features enabled this module has no public exports -pub use crate::conversions::*; - -#[cfg(feature = "macros")] -pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; - /// A proc macro used to expose Rust structs and fieldless enums as Python objects. -/// #[doc = include_str!("../guide/pyclass-parameters.md")] /// /// For more on creating Python classes, @@ -478,6 +471,11 @@ pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; /// [1]: https://pyo3.rs/latest/class.html #[cfg(feature = "macros")] pub use pyo3_macros::pyclass; +#[cfg(feature = "macros")] +pub use pyo3_macros::{pyfunction, pymethods, pymodule, FromPyObject}; + +#[allow(unused_imports)] // with no features enabled this module has no public exports +pub use crate::conversions::*; #[cfg(feature = "macros")] #[macro_use] @@ -512,6 +510,7 @@ pub mod doc_test { "README.md" => readme_md, "guide/src/advanced.md" => guide_advanced_md, "guide/src/async-await.md" => guide_async_await_md, + "guide/src/async-await/awaiting_python_awaitables.md" => guide_async_await_awaiting_python_awaitable_md, "guide/src/building-and-distribution.md" => guide_building_and_distribution_md, "guide/src/building-and-distribution/multiple-python-versions.md" => guide_bnd_multiple_python_versions_md, "guide/src/class.md" => guide_class_md, diff --git a/src/tests/common.rs b/src/tests/common.rs index 854d73e4d7b..78962d68cd0 100644 --- a/src/tests/common.rs +++ b/src/tests/common.rs @@ -6,13 +6,14 @@ #[macro_use] mod inner { + use pyo3::{ + prelude::*, + types::{IntoPyDict, PyList}, + }; + #[allow(unused_imports)] // pulls in `use crate as pyo3` in `test_utils.rs` use super::*; - use pyo3::prelude::*; - - use pyo3::types::{IntoPyDict, PyList}; - #[macro_export] macro_rules! py_assert { ($py:expr, $($val:ident)+, $assertion:literal) => { @@ -156,6 +157,17 @@ mod inner { .unwrap(); }}; } + + // see https://stackoverflow.com/questions/60359157/valueerror-set-wakeup-fd-only-works-in-main-thread-on-windows-on-python-3-8-wit + #[cfg(feature = "macros")] + pub fn asyncio_windows(test: &str) -> String { + let set_event_loop_policy = r#" + import asyncio, sys + if sys.platform == "win32": + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) + "#; + pyo3::unindent::unindent(set_event_loop_policy) + &pyo3::unindent::unindent(test) + } } #[allow(unused_imports)] // some tests use just the macros and none of the other functionality diff --git a/tests/test_await_in_coroutine.rs b/tests/test_await_in_coroutine.rs new file mode 100644 index 00000000000..3f835c27588 --- /dev/null +++ b/tests/test_await_in_coroutine.rs @@ -0,0 +1,177 @@ +#![cfg(feature = "experimental-async")] + +use std::task::Poll; + +use futures::{future::poll_fn, FutureExt}; +use pyo3::{ + coroutine::{await_in_coroutine, CancelHandle}, + exceptions::{PyAttributeError, PyTypeError}, + prelude::*, + py_run, +}; + +#[path = "../src/tests/common.rs"] +mod common; + +#[pyfunction] +async fn wrap_awaitable(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + future.await +} + +#[test] +fn awaitable() { + Python::with_gil(|gil| { + let wrap_awaitable = wrap_pyfunction_bound!(wrap_awaitable, gil).unwrap(); + let test = r#" + import types + import asyncio; + + class BadAwaitable: + def __await__(self): + raise AttributeError("__await__") + + @types.coroutine + def gen_coro(): + yield None + + async def main(): + await wrap_awaitable(...) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("wrap_awaitable", wrap_awaitable).unwrap(); + let run = |awaitable| { + gil.run_bound( + &common::asyncio_windows(test).replace("...", awaitable), + Some(&globals), + None, + ) + }; + run("asyncio.sleep(0.001)").unwrap(); + run("gen_coro()").unwrap(); + assert!(run("None").unwrap_err().is_instance_of::(gil)); + assert!(run("BadAwaitable()") + .unwrap_err() + .is_instance_of::(gil)); + }) +} + +#[test] +fn cancel_delegation() { + #[pyfunction] + async fn wrap_cancellable(awaitable: PyObject, #[pyo3(cancel_handle)] cancel: CancelHandle) { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil))).unwrap(); + let result = future.await; + Python::with_gil(|gil| { + assert_eq!( + result.unwrap_err().get_type_bound(gil).name().unwrap(), + "CancelledError" + ) + }); + assert!(!cancel.is_cancelled()); + } + Python::with_gil(|gil| { + let wrap_cancellable = wrap_pyfunction_bound!(wrap_cancellable, gil).unwrap(); + let test = r#" + import asyncio; + + async def main(): + task = asyncio.create_task(wrap_cancellable(asyncio.sleep(0.001))) + await asyncio.sleep(0) + task.cancel() + await task + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals + .set_item("wrap_cancellable", wrap_cancellable) + .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} + +#[test] +#[should_panic(expected = "PyFuture must be awaited in coroutine context")] +fn pyfuture_without_coroutine() { + #[pyfunction] + fn block_on(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::executor::block_on(future) + } + Python::with_gil(|gil| { + let block_on = wrap_pyfunction_bound!(block_on, gil).unwrap(); + let test = r#" + async def coro(): + ... + block_on(coro()) + "#; + py_run!(gil, block_on, &common::asyncio_windows(test)); + }) +} + +async fn checkpoint() { + let mut ready = false; + poll_fn(|cx| { + if ready { + return Poll::Ready(()); + } + ready = true; + cx.waker().wake_by_ref(); + Poll::Pending + }) + .await +} + +#[test] +#[should_panic(expected = "Python awaitable mixed with Rust future")] +fn pyfuture_in_select() { + #[pyfunction] + async fn select(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + _ = checkpoint().fuse() => unreachable!(), + res = future.fuse() => res, + } + } + Python::with_gil(|gil| { + let select = wrap_pyfunction_bound!(select, gil).unwrap(); + let test = r#" + import asyncio; + async def main(): + return await select(asyncio.sleep(1)) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("select", select).unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} + +#[test] +#[should_panic(expected = "Python awaitable mixed with Rust future")] +fn pyfuture_in_select2() { + #[pyfunction] + async fn select2(awaitable: PyObject) -> PyResult { + let future = Python::with_gil(|gil| await_in_coroutine(awaitable.bind(gil)))?; + futures::select_biased! { + res = future.fuse() => res, + _ = checkpoint().fuse() => unreachable!(), + } + } + Python::with_gil(|gil| { + let select2 = wrap_pyfunction_bound!(select2, gil).unwrap(); + let test = r#" + import asyncio; + async def main(): + return await select2(asyncio.sleep(1)) + asyncio.run(main()) + "#; + let globals = gil.import_bound("__main__").unwrap().dict(); + globals.set_item("select2", select2).unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); + }) +} diff --git a/tests/test_compile_error.rs b/tests/test_compile_error.rs index 44049620598..48069a59d75 100644 --- a/tests/test_compile_error.rs +++ b/tests/test_compile_error.rs @@ -52,6 +52,6 @@ fn test_compile_errors() { t.compile_fail("tests/ui/invalid_pymodule_two_pymodule_init.rs"); #[cfg(feature = "experimental-async")] #[cfg(any(not(Py_LIMITED_API), Py_3_10))] // to avoid PyFunctionArgument for &str - t.compile_fail("tests/ui/invalid_cancel_handle.rs"); + t.compile_fail("tests/ui/invalid_async_pyfunction.rs"); t.pass("tests/ui/pymodule_missing_docs.rs"); } diff --git a/tests/test_coroutine.rs b/tests/test_coroutine.rs index 4abba9f36b4..d95372fe771 100644 --- a/tests/test_coroutine.rs +++ b/tests/test_coroutine.rs @@ -1,28 +1,20 @@ #![cfg(feature = "experimental-async")] #![cfg(not(target_arch = "wasm32"))] -use std::{task::Poll, thread, time::Duration}; +use std::{sync::Arc, task::Poll, thread, time::Duration}; use futures::{channel::oneshot, future::poll_fn, FutureExt}; use portable_atomic::{AtomicBool, Ordering}; use pyo3::{ - coroutine::CancelHandle, + coroutine::{CancelHandle, Coroutine}, prelude::*, py_run, + sync::GILOnceCell, types::{IntoPyDict, PyType}, }; #[path = "../src/tests/common.rs"] mod common; -fn handle_windows(test: &str) -> String { - let set_event_loop_policy = r#" - import asyncio, sys - if sys.platform == "win32": - asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) - "#; - pyo3::unindent::unindent(set_event_loop_policy) + &pyo3::unindent::unindent(test) -} - #[test] fn noop_coroutine() { #[pyfunction] @@ -32,7 +24,7 @@ fn noop_coroutine() { Python::with_gil(|gil| { let noop = wrap_pyfunction_bound!(noop, gil).unwrap(); let test = "import asyncio; assert asyncio.run(noop()) == 42"; - py_run!(gil, noop, &handle_windows(test)); + py_run!(gil, noop, &common::asyncio_windows(test)); }) } @@ -76,7 +68,7 @@ fn test_coroutine_qualname() { ("MyClass", gil.get_type_bound::().as_any()), ] .into_py_dict_bound(gil); - py_run!(gil, *locals, &handle_windows(test)); + py_run!(gil, *locals, &common::asyncio_windows(test)); }) } @@ -98,7 +90,7 @@ fn sleep_0_like_coroutine() { Python::with_gil(|gil| { let sleep_0 = wrap_pyfunction_bound!(sleep_0, gil).unwrap(); let test = "import asyncio; assert asyncio.run(sleep_0()) == 42"; - py_run!(gil, sleep_0, &handle_windows(test)); + py_run!(gil, sleep_0, &common::asyncio_windows(test)); }) } @@ -117,7 +109,7 @@ fn sleep_coroutine() { Python::with_gil(|gil| { let sleep = wrap_pyfunction_bound!(sleep, gil).unwrap(); let test = r#"import asyncio; assert asyncio.run(sleep(0.1)) == 42"#; - py_run!(gil, sleep, &handle_windows(test)); + py_run!(gil, sleep, &common::asyncio_windows(test)); }) } @@ -137,11 +129,7 @@ fn cancelled_coroutine() { let globals = gil.import_bound("__main__").unwrap().dict(); globals.set_item("sleep", sleep).unwrap(); let err = gil - .run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) + .run_bound(&common::asyncio_windows(test), Some(&globals), None) .unwrap_err(); assert_eq!( err.value_bound(gil).get_type().qualname().unwrap(), @@ -177,12 +165,8 @@ fn coroutine_cancel_handle() { globals .set_item("cancellable_sleep", cancellable_sleep) .unwrap(); - gil.run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) - .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); }) } @@ -207,12 +191,8 @@ fn coroutine_is_cancelled() { "#; let globals = gil.import_bound("__main__").unwrap().dict(); globals.set_item("sleep_loop", sleep_loop).unwrap(); - gil.run_bound( - &pyo3::unindent::unindent(&handle_windows(test)), - Some(&globals), - None, - ) - .unwrap(); + gil.run_bound(&common::asyncio_windows(test), Some(&globals), None) + .unwrap(); }) } @@ -241,7 +221,7 @@ fn coroutine_panic() { else: assert False; "#; - py_run!(gil, panic, &handle_windows(test)); + py_run!(gil, panic, &common::asyncio_windows(test)); }) } @@ -338,6 +318,57 @@ fn test_async_method_receiver_with_other_args() { assert asyncio.run(v.get_value_plus_with(1, 1)) == 12 "#; let locals = [("Value", gil.get_type_bound::())].into_py_dict_bound(gil); - py_run!(gil, *locals, test); + py_run!(gil, *locals, &common::asyncio_windows(test)); }); } + +#[test] +fn multi_thread_event_loop() { + Python::with_gil(|gil| { + let sleep = wrap_pyfunction_bound!(sleep, gil).unwrap(); + let test = r#" + import asyncio + import threading + loop = asyncio.new_event_loop() + # spawn the sleep task and run just one iteration of the event loop + # to schedule the sleep wakeup + task = loop.create_task(sleep(0.1)) + loop.stop() + loop.run_forever() + assert not task.done() + # spawn a thread to complete the execution of the sleep task + def target(loop, task): + loop.run_until_complete(task) + thread = threading.Thread(target=target, args=(loop, task)) + thread.start() + thread.join() + assert task.result() == 42 + "#; + py_run!(gil, sleep, &common::asyncio_windows(test)); + }) +} + +#[test] +fn closed_event_loop() { + let waker = Arc::new(GILOnceCell::new()); + let waker2 = waker.clone(); + let future = poll_fn(move |cx| { + Python::with_gil(|gil| waker2.set(gil, cx.waker().clone()).unwrap()); + Poll::Pending::> + }); + Python::with_gil(|gil| { + let register_waker = Coroutine::new("register_waker", future).into_py(gil); + let test = r#" + import asyncio + loop = asyncio.new_event_loop() + # register a waker by spawning a task and polling it once, then close the loop + task = loop.create_task(register_waker) + loop.stop() + loop.run_forever() + loop.close() + "#; + py_run!(gil, register_waker, &common::asyncio_windows(test)); + // asyncio waker can be used even if the event loop is closed + Python::with_gil(|gil| waker.get(gil).unwrap().wake_by_ref()) + }) +} diff --git a/tests/test_methods.rs b/tests/test_methods.rs index 2b5396e9ee4..f343dd3eb98 100644 --- a/tests/test_methods.rs +++ b/tests/test_methods.rs @@ -1158,3 +1158,23 @@ fn test_issue_2988() { ) { } } + +#[pyclass] +struct NoGILCounter(usize); + +#[pymethods] +impl NoGILCounter { + #[pyo3(allow_threads, signature = (other = 1))] + fn inc_no_gil(&mut self, other: usize) -> usize { + self.0 += other; + self.0 + } +} + +#[test] +fn test_method_allow_threads() { + Python::with_gil(|py| { + let counter = Py::new(py, NoGILCounter(42)).unwrap(); + py_run!(py, counter, "assert counter.inc_no_gil() == 43") + }) +} diff --git a/tests/ui/invalid_cancel_handle.rs b/tests/ui/invalid_async_pyfunction.rs similarity index 60% rename from tests/ui/invalid_cancel_handle.rs rename to tests/ui/invalid_async_pyfunction.rs index cff6c5dcbad..d39c017c0e7 100644 --- a/tests/ui/invalid_cancel_handle.rs +++ b/tests/ui/invalid_async_pyfunction.rs @@ -1,20 +1,26 @@ use pyo3::prelude::*; +#[pyfunction(allow_threads)] +async fn async_with_gil(_py: Python<'_>) {} + +#[pyfunction(allow_threads)] +async fn async_with_bound(_obj: &Bound<'_, PyAny>) {} + #[pyfunction] -async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: String) {} +async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: i32) {} #[pyfunction] async fn cancel_handle_repeated2( - #[pyo3(cancel_handle)] _param: String, - #[pyo3(cancel_handle)] _param2: String, + #[pyo3(cancel_handle)] _param: i32, + #[pyo3(cancel_handle)] _param2: i32, ) { } #[pyfunction] -fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: String) {} +fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: i32) {} #[pyfunction] -async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: String) {} +async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: i32) {} #[pyfunction] async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} diff --git a/tests/ui/invalid_async_pyfunction.stderr b/tests/ui/invalid_async_pyfunction.stderr new file mode 100644 index 00000000000..eb510421c09 --- /dev/null +++ b/tests/ui/invalid_async_pyfunction.stderr @@ -0,0 +1,141 @@ +error: GIL token cannot be passed to async function + --> tests/ui/invalid_async_pyfunction.rs:4:30 + | +4 | async fn async_with_gil(_py: Python<'_>) {} + | ^^^^^^ + +error: `cancel_handle` may only be specified once per argument + --> tests/ui/invalid_async_pyfunction.rs:10:55 + | +10 | async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: i32) {} + | ^^^^^^^^^^^^^ + +error: `cancel_handle` may only be specified once + --> tests/ui/invalid_async_pyfunction.rs:15:28 + | +15 | #[pyo3(cancel_handle)] _param2: i32, + | ^^^^^^^ + +error: `cancel_handle` attribute can only be used with `async fn` + --> tests/ui/invalid_async_pyfunction.rs:20:53 + | +20 | fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: i32) {} + | ^^^^^^ + +error: `from_py_with` and `cancel_handle` cannot be specified together + --> tests/ui/invalid_async_pyfunction.rs:30:12 + | +30 | #[pyo3(cancel_handle, from_py_with = "cancel_handle")] _param: pyo3::coroutine::CancelHandle, + | ^^^^^^^^^^^^^ + +error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely + --> tests/ui/invalid_async_pyfunction.rs:6:1 + | +6 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely + | + = help: within `pyo3::Bound<'_, pyo3::PyAny>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`, which is required by `{async block@$DIR/tests/ui/invalid_async_pyfunction.rs:6:1: 6:29}: Send` +note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `impl_::not_send::NotSend` + --> src/impl_/not_send.rs + | + | pub(crate) struct NotSend(PhantomData<*mut Python<'static>>); + | ^^^^^^^ + = note: required because it appears within the type `(&pyo3::gil::GILGuard, impl_::not_send::NotSend)` +note: required because it appears within the type `PhantomData<(&pyo3::gil::GILGuard, impl_::not_send::NotSend)>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `pyo3::Python<'_>` + --> src/marker.rs + | + | pub struct Python<'py>(PhantomData<(&'py GILGuard, NotSend)>); + | ^^^^^^ +note: required because it appears within the type `pyo3::Bound<'_, pyo3::PyAny>` + --> src/instance.rs + | + | pub struct Bound<'py, T>(Python<'py>, ManuallyDrop>); + | ^^^^^ + = note: required for `&pyo3::Bound<'_, pyo3::PyAny>` to implement `Send` +note: required because it's used within this `async` fn body + --> tests/ui/invalid_async_pyfunction.rs:7:52 + | +7 | async fn async_with_bound(_obj: &Bound<'_, PyAny>) {} + | ^^ +note: required because it's used within this `async` block + --> tests/ui/invalid_async_pyfunction.rs:6:1 + | +6 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +note: required by a bound in `pyo3::coroutine::Coroutine::new` + --> src/coroutine.rs + | + | pub fn new(name: impl Into>, future: F) -> Self + | --- required by a bound in this associated function + | where + | F: Future> + Send + Ungil + 'static, + | ^^^^ required by this bound in `Coroutine::new` + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) + +error[E0308]: mismatched types + --> tests/ui/invalid_async_pyfunction.rs:22:1 + | +22 | #[pyfunction] + | ^^^^^^^^^^^^^ + | | + | expected `i32`, found `CancelHandle` + | arguments to this function are incorrect + | +note: function defined here + --> tests/ui/invalid_async_pyfunction.rs:23:10 + | +23 | async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: i32) {} + | ^^^^^^^^^^^^^^^^^^^^^^^^ ----------- + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) + +error[E0277]: the trait bound `CancelHandle: PyClass` is not satisfied + --> tests/ui/invalid_async_pyfunction.rs:26:50 + | +26 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} + | ^^^^ the trait `PyClass` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` + | + = help: the trait `PyClass` is implemented for `pyo3::coroutine::Coroutine` + = note: required for `CancelHandle` to implement `FromPyObject<'_>` + = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` + = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` +note: required by a bound in `extract_argument` + --> src/impl_/extract_argument.rs + | + | pub fn extract_argument<'a, 'py, T>( + | ---------------- required by a bound in this function +... + | T: PyFunctionArgument<'a, 'py>, + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` + +error[E0277]: the trait bound `CancelHandle: Clone` is not satisfied + --> tests/ui/invalid_async_pyfunction.rs:26:50 + | +26 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} + | ^^^^ the trait `Clone` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` + | + = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: + Option<&'a pyo3::Bound<'py, T>> + &'a pyo3::Bound<'py, T> + &'a pyo3::coroutine::Coroutine + &'a mut pyo3::coroutine::Coroutine + = note: required for `CancelHandle` to implement `FromPyObject<'_>` + = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` + = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` +note: required by a bound in `extract_argument` + --> src/impl_/extract_argument.rs + | + | pub fn extract_argument<'a, 'py, T>( + | ---------------- required by a bound in this function +... + | T: PyFunctionArgument<'a, 'py>, + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` diff --git a/tests/ui/invalid_cancel_handle.stderr b/tests/ui/invalid_cancel_handle.stderr deleted file mode 100644 index 41a2c0854b7..00000000000 --- a/tests/ui/invalid_cancel_handle.stderr +++ /dev/null @@ -1,81 +0,0 @@ -error: `cancel_handle` may only be specified once per argument - --> tests/ui/invalid_cancel_handle.rs:4:55 - | -4 | async fn cancel_handle_repeated(#[pyo3(cancel_handle, cancel_handle)] _param: String) {} - | ^^^^^^^^^^^^^ - -error: `cancel_handle` may only be specified once - --> tests/ui/invalid_cancel_handle.rs:9:28 - | -9 | #[pyo3(cancel_handle)] _param2: String, - | ^^^^^^^ - -error: `cancel_handle` attribute can only be used with `async fn` - --> tests/ui/invalid_cancel_handle.rs:14:53 - | -14 | fn cancel_handle_synchronous(#[pyo3(cancel_handle)] _param: String) {} - | ^^^^^^ - -error: `from_py_with` and `cancel_handle` cannot be specified together - --> tests/ui/invalid_cancel_handle.rs:24:12 - | -24 | #[pyo3(cancel_handle, from_py_with = "cancel_handle")] _param: pyo3::coroutine::CancelHandle, - | ^^^^^^^^^^^^^ - -error[E0308]: mismatched types - --> tests/ui/invalid_cancel_handle.rs:16:1 - | -16 | #[pyfunction] - | ^^^^^^^^^^^^^ - | | - | expected `String`, found `CancelHandle` - | arguments to this function are incorrect - | -note: function defined here - --> tests/ui/invalid_cancel_handle.rs:17:10 - | -17 | async fn cancel_handle_wrong_type(#[pyo3(cancel_handle)] _param: String) {} - | ^^^^^^^^^^^^^^^^^^^^^^^^ -------------- - = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info) - -error[E0277]: the trait bound `CancelHandle: PyClass` is not satisfied - --> tests/ui/invalid_cancel_handle.rs:20:50 - | -20 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} - | ^^^^ the trait `PyClass` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` - | - = help: the trait `PyClass` is implemented for `pyo3::coroutine::Coroutine` - = note: required for `CancelHandle` to implement `FromPyObject<'_>` - = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` - = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` -note: required by a bound in `extract_argument` - --> src/impl_/extract_argument.rs - | - | pub fn extract_argument<'a, 'py, T>( - | ---------------- required by a bound in this function -... - | T: PyFunctionArgument<'a, 'py>, - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` - -error[E0277]: the trait bound `CancelHandle: Clone` is not satisfied - --> tests/ui/invalid_cancel_handle.rs:20:50 - | -20 | async fn missing_cancel_handle_attribute(_param: pyo3::coroutine::CancelHandle) {} - | ^^^^ the trait `Clone` is not implemented for `CancelHandle`, which is required by `CancelHandle: PyFunctionArgument<'_, '_>` - | - = help: the following other types implement trait `PyFunctionArgument<'a, 'py>`: - Option<&'a pyo3::Bound<'py, T>> - &'a pyo3::Bound<'py, T> - &'a pyo3::coroutine::Coroutine - &'a mut pyo3::coroutine::Coroutine - = note: required for `CancelHandle` to implement `FromPyObject<'_>` - = note: required for `CancelHandle` to implement `FromPyObjectBound<'_, '_>` - = note: required for `CancelHandle` to implement `PyFunctionArgument<'_, '_>` -note: required by a bound in `extract_argument` - --> src/impl_/extract_argument.rs - | - | pub fn extract_argument<'a, 'py, T>( - | ---------------- required by a bound in this function -... - | T: PyFunctionArgument<'a, 'py>, - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `extract_argument` diff --git a/tests/ui/invalid_pyfunction_signatures.stderr b/tests/ui/invalid_pyfunction_signatures.stderr index dbca169d8ea..d48e21fa1ea 100644 --- a/tests/ui/invalid_pyfunction_signatures.stderr +++ b/tests/ui/invalid_pyfunction_signatures.stderr @@ -16,7 +16,7 @@ error: expected argument from function definition `y` but got argument `x` 13 | #[pyo3(signature = (x))] | ^ -error: expected one of: `name`, `pass_module`, `signature`, `text_signature`, `crate` +error: expected one of: `allow_threads`, `name`, `pass_module`, `signature`, `text_signature`, `crate` --> tests/ui/invalid_pyfunction_signatures.rs:18:14 | 18 | #[pyfunction(x)] diff --git a/tests/ui/invalid_pyfunctions.rs b/tests/ui/invalid_pyfunctions.rs index 1a95c9e4a34..4c09ffd5608 100644 --- a/tests/ui/invalid_pyfunctions.rs +++ b/tests/ui/invalid_pyfunctions.rs @@ -35,4 +35,10 @@ fn first_argument_not_module<'a, 'py>( module.name() } +#[pyfunction(allow_threads)] +fn allow_threads_with_gil(_py: Python<'_>) {} + +#[pyfunction(allow_threads)] +fn allow_threads_with_bound(_obj: &Bound<'_, PyAny>) {} + fn main() {} diff --git a/tests/ui/invalid_pyfunctions.stderr b/tests/ui/invalid_pyfunctions.stderr index 893d7cbec2c..af49195e0de 100644 --- a/tests/ui/invalid_pyfunctions.stderr +++ b/tests/ui/invalid_pyfunctions.stderr @@ -47,6 +47,12 @@ error: expected `&PyModule` or `Py` as first argument with `pass_modul 28 | fn pass_module_but_no_arguments<'py>() {} | ^^ +error: GIL cannot be held in function annotated with `allow_threads` + --> tests/ui/invalid_pyfunctions.rs:39:32 + | +39 | fn allow_threads_with_gil(_py: Python<'_>) {} + | ^^^^^^ + error[E0277]: the trait bound `&str: From>` is not satisfied --> tests/ui/invalid_pyfunctions.rs:32:13 | @@ -61,3 +67,54 @@ error[E0277]: the trait bound `&str: From> > = note: required for `BoundRef<'_, '_, pyo3::prelude::PyModule>` to implement `Into<&str>` + +error[E0277]: `*mut pyo3::Python<'static>` cannot be shared between threads safely + --> tests/ui/invalid_pyfunctions.rs:41:1 + | +41 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `*mut pyo3::Python<'static>` cannot be shared between threads safely + | + = help: within `&pyo3::Bound<'_, pyo3::PyAny>`, the trait `Sync` is not implemented for `*mut pyo3::Python<'static>`, which is required by `{closure@$DIR/tests/ui/invalid_pyfunctions.rs:41:1: 41:29}: Ungil` +note: required because it appears within the type `PhantomData<*mut pyo3::Python<'static>>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `impl_::not_send::NotSend` + --> src/impl_/not_send.rs + | + | pub(crate) struct NotSend(PhantomData<*mut Python<'static>>); + | ^^^^^^^ + = note: required because it appears within the type `(&pyo3::gil::GILGuard, impl_::not_send::NotSend)` +note: required because it appears within the type `PhantomData<(&pyo3::gil::GILGuard, impl_::not_send::NotSend)>` + --> $RUST/core/src/marker.rs + | + | pub struct PhantomData; + | ^^^^^^^^^^^ +note: required because it appears within the type `pyo3::Python<'_>` + --> src/marker.rs + | + | pub struct Python<'py>(PhantomData<(&'py GILGuard, NotSend)>); + | ^^^^^^ +note: required because it appears within the type `pyo3::Bound<'_, pyo3::PyAny>` + --> src/instance.rs + | + | pub struct Bound<'py, T>(Python<'py>, ManuallyDrop>); + | ^^^^^ + = note: required because it appears within the type `&pyo3::Bound<'_, pyo3::PyAny>` + = note: required for `&&pyo3::Bound<'_, pyo3::PyAny>` to implement `Send` +note: required because it's used within this closure + --> tests/ui/invalid_pyfunctions.rs:41:1 + | +41 | #[pyfunction(allow_threads)] + | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + = note: required for `{closure@$DIR/tests/ui/invalid_pyfunctions.rs:41:1: 41:29}` to implement `Ungil` +note: required by a bound in `pyo3::Python::<'py>::allow_threads` + --> src/marker.rs + | + | pub fn allow_threads(self, f: F) -> T + | ------------- required by a bound in this associated function + | where + | F: Ungil + FnOnce() -> T, + | ^^^^^ required by this bound in `Python::<'py>::allow_threads` + = note: this error originates in the attribute macro `pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info)