PEP 489 Compatibility

[davidhewitt]

Inspired by #2269 (comment), this is an issue to track all of the places which we know need to change for full PEP 489 compatibility (draft implementation in #2245).

The latest documentation about supporting isolated extensions can be found at https://docs.python.org/3.14/howto/isolating-extensions.html

• Rust statics - #[pyclass] type objects, most implementations of PyTypeInfo, intern!

[davidhewitt]

The above is currently incomplete - please post / edit things as you think of them

[messense]

Linking some related PEPs here:

• PEP 489 – Multi-phase extension module initialization
• PEP 573 – Module State Access from C Extension Methods
• PEP 630 – Isolating Extension Modules
• PEP 684 - A Per-Interpreter GIL
• PEP 687 – Isolating modules in the standard library

[davidhewitt]

Beginning to think that this is something that we may not want to do for now - see #2346 (comment)

[davidhewitt]

@bazaah following up from #5525 (comment)

Let's use this issue to track follow-up work.

Defining a way to set per-module state as part of the #[pymodule] macro will allow users to get a long way towards isolating their own modules. There is potentially a lot of work we need to do inside various function macro machinery to make that state accessible.

One other alternative to removing statics is to have interpreter-safe datastructures (e.g. store multiple values in some weakly-referenced form, indexed by interpreter ID).

[bazaah]

Yeah, my experimenting so far has lead me to the realization that exposing generics sanely while interacting with PyModule_GetState and #[pymodule] is pretty difficult to expose via an (e.g) PyModuleState<T>, as we have to erase and then "remember" the T later, inside the closure callbacks. I'm currently exploring a typemap approach instead, which is worse ergonomically, but shields pyo3 from having to care about the user's state T at all.

I haven't yet really started to look at what statics will need to be updated / moved. In general I think it will be pretty difficult to handle m_traverse stuff without storing stuff directly in the module state, but we probably just don't have to do that immediately either.

[bazaah]

@davidhewitt -- I'm trying to get my first go at this into a reviewable shape, and have a question around what exactly Python::attach / detach mean. These are referring to the "thread state" (ala python/cpython#127989), yes? More specifically, once a module (and it's PyModule_GetState state) is initialized, that module instance cannot move across threads, right? I have tried reading the cpython docs, but I'm still a bit confused 😆

[bazaah]

So, assuming something close to #5600 is merged, we are left with the question of how to expose this state to consumers.

My personal thoughts after experimenting with it for a bit:

1. The pyo3 side gets very annoying if the (module) state isn't a concrete type
2. I don't understand the semantics around PyInterpreterState_GetID + multiple interpreters (in the same os thread) + cpython ""thread state"" enough to know how this safely interacts with rust statics and thread locals
   • How does PyInterpreterState_Get work in the presence of >1 interpreters on a single thread?
     • From what I can tell, pyo3 encapsulates this in the AttachGuard machinery, which is not guaranteed to have a reference to the PyInterpreterState at all (something else could have initialized the state before pyo3 got called), so trying to use it instead is out
   • Can interpreters / modules / module state be moved across threads? Do we need to account for that or can we assume that if we are "attached" everything happening is thread local and doesn't need synchronization vis-a-vis the python parts

1 mandates type erasure of some kind, so pyo3 remains blissfully unaware of what consumer state types look like -- I've been using a vendored typemap impl from some of my internal stuff and it seems to work fine, we can decide if we want to keep that or use a crate.

If we assume the following for 2:

• There is a 1:1 relation between interpreter thread state and os thread
• interpreters / modules / module state don't move threads or are accessed across threads

Then I think there is a reasonable path forward to moving most of the pyo3 statics to thread_locals, and locking the remaining few inside sync primitives, plus maybe exposing a PyThreadLock (or something) that works like PyOnceLock, but initializes once per thread / python thread state, as that makes it very easy for (e.g) pyca/cryptography to move their lazy imports over to being safe in the presence subinterpreters.

But I can't find documentation that the above assumptions hold true.

Beyond all of this, lurks the specter of PyInterpreterState_Clear / PyInterpreterState_Delete but I think pyo3 as it stands today probably doesn't handle the destruction of an interpreter well, so we're not really making the situation worse.

[alex]

1. When you say concrete type, what do you mean? It seems like at a 10,000 ft level, we want to somehow introduce a generic T for our module if we can.

2. Isn't the whole (?) point of being able assosciate module state that we don't need thread locals, and that we can store relevant state on some type? It's not clear that there's any guarantee that a given thread can never change which interpreter its running on, unless that's documented somewhere.

[davidhewitt]

that module instance cannot move across threads, right

A module instance can move across threads freely. It cannot move between subinterpreters, which are a broader, more awkward concept. (One subinterpreter can have many threads & thread states). Any number of Python subinterpreters and thread states may be run on the same native thread (but only one at a time, TLS etc).

My assumption is that for any "globals" which need to be subinterpreter-specific we don't have much choice other than creating some kind of hashmap-like structure. Or having a way to retrieve that from interpreter-specific state (which I believe exists, but I am not very familiar with). pybind11 may have prior art for us to look at.

[bazaah]

(on 1)
What I mean is that pyo3 really doesn't want to deal with a PyModule<T>, it wants a PyModule, which means we have to type erase the T.

I currently expose access to the user state via some extra methods on PyModuleMethods that look like:

fn state_or_init<T, F>(&mut self, f: F) -> &mut T
where
	F: FnOnce() -> T
{ /* ... */ }

fn state_mut(&mut self) -> Option<&mut T>
{ /* ... */ }

// Used like
if let Some(state) = mymodule.state_mut::<MyStateType>()
{ /* ... */ }

Importantly, pyo3 itself doesn't need to carry around the type information, consumers can materialize their state's T at their leisure.

(on 2)
I'm very open to be corrected here as I would best characterize my understanding of (c)python's operating model as shaky 😅. That said, here how I understand it:

1. At the top, you have some sort of global process state (Py_Initialize & friends)
2. Since py3.7 (I think?) you have per-thread interpreter state (PyInterpreterState_New & friends)
3. Lastly, you have per-module / extension state (PyModule_GetState & friends)

I think the PyInterpreterState level corresponds to how imports are loaded / stored in python, ("once" per interpreter) and not on a module-by-module case. That said, importing a module triggers anything it does in the moral equivalent of __new__ (or pymodule_init), so every #[pymodule] will get to load its state on import.

So it goes something like:

erDiagram
	direction LR
	Py_Initialize ||--o{ PyInterpreterState_New : "has many"
	PyInterpreterState_New ||--o{ PyModule_GetState : "has many"

Loading

Which means we need both per-thread storage & module level storage to be able to properly represent how and where python stores stuff

It's not clear that there's any guarantee that a given thread can never change which interpreter its running on

Yup, hence why I'm asking because I can't find documentation either way

[bazaah]

A module instance can move across threads freely. It cannot move between subinterpreters, which are a broader, more awkward concept. (One subinterpreter can have many threads & thread states). Any number of Python subinterpreters and thread states may be run on the same native thread (but only one at a time, TLS etc).

So, subinterpreters exist above PyInterpreterState but below Py_Initialize? (That is, you can have one Py_Initialize with many subinterpreters(?), each of which can have many PyInterpreterStates that can each have many PyModules?

[davidhewitt]

Each subinterpreter will have

• one interpreter state
• many thread states
• many module states (which have no relation to thread states)