Next steps for type introspection and stub generation

[Tpt]

This is an incomplete list of next steps to build proper stubs from PyO3 codebases:

• introspect all class methods (including adding a test for all magic methods) (partial: Refine some magic methods introspection #5273)
• introspect classes associated constants (const) Introspect class associated constants #5272
• introspect modules associated constants (const) add declarative module level constants export #5096
• introspect simple enums built by #[pyclass]
• introspect complex enums built by #[pyclass]
• introspect exceptions built by create_exception!
• introspect class inheritance (#[pyclass(extends=)]) Introspection: Basic pyclass(extends) support #5331
• @typing.final decorator on class that cannot be subclasses (without #[pyclass(subclass)]) Introspection: add typing.final on final classes #5552
• introspect auto-generated magic methods (#[pyclass(eq, eq_int, ord, hash, str)]) Generate introspection for #[pyclass(eq, eq_int, ord, hash, str)] #5338
• introspect fields getter and setter (#[pyo3(get, set)], #[pyclass(get_all, set_all)]) Introspection: support #[pyo3(get, set)] and #[pyclass(get_all, set_all)] #5370
• implement return type annotation Introspection: implement output type #5208
• fill PYTHON_TYPE constant on all implementations of the relevant traits (FromPyObject and IntoPyObject) Introspection: simplify code by leveraging that building type hint for containers now work #5634 Introspection: Declare INPUT_TYPE and OUTPUT_TYPE nearly everywhere #5637 Introspection: Set INPUT_TYPE and OUTPUT_TYPE on Vec<_> and &[] #5639 Introspection: properly set OUTPUT_TYPE on IntoPyObject implementations for Bound, Borrow and Py and add a PyTypeCheck bound to them #5640
• fill INPUT_TYPE in #[derive(FromPyObject)] Basic introspection of #[derive(FromPyObject)] #5339
• fill OUTPUT_TYPE in #[derive(IntoPyObject)] Basic introspection of #[derive(IntoPyObject)] #5365
• figure out how to emit #[classattr] in the stubs
• support cross-modules types (class defined in module A and used in module B as an input/output type)
• add doc strings to the generated stubs
• proper test coverage
• proper formatting of stubs (not a blocker)
• integration into maturin
• proper type stubs for containers (list[T] instead of list). Introspection: Set INPUT_TYPE and OUTPUT_TYPE on Vec<_> and &[] #5639
• allow to set custom type annotations (both inputs and output) Introspection: allows to override annotations in signature #5241
• allow to set custom stubs for eg. protocols
• add _typeshed.Incomplete to relevant places (modules with a #[pymodule_init] function...) (doc) Introspection: properly tag modules as incomplete when needed #5207
• choose between _typeshed.Incomplete and typing.Any in type annotations
• make sure to properly gate introspection element with cfg macros
• provide a way to set @overloads
• Support WASM files in pyo3-introspection
• Allow custom type annotations without strings (example: #[pyo3(signature = (arg: list[int]) -> list[int])])
• Add version information to the generated JSON blobs
• Figure out if the function used to tag incomplete modules should be generated by pyo3-macros instead
• Make sure we don't try to import None from builtins, it does not work

Reference on stubs file

[davidhewitt]

For "custom type annotations", I think maybe adding it to #[pyo3(signature)] attribute is the way to go (xref #5138)

// `a` and return type set explicitly, b will be inferred
#[pyo3(pyfunction, signature = (a: int, b) -> int)]
fn foo(a: i32, b: i32) -> i32 {
    a + b
}

... I guess the alternative would be #[pyo3(type = "int")] annotations on each argument, but I think using the Python syntax is much better.

[yogevm15]

Hi,
I would like to help here, is there anything I should be aware of before I start working on it?

[Tpt]

@yogevm15 Thank you! It would be amazing!

I am currently working on function type annotations #5089

There are a bunch of topics that are quite independent from it like exposing const (without type annotation), supporting properly enums...

To help you deep dive in the code:

• Introspection data are emitted as const elements by the various PyO3 macros. Code for that is in pyo3-macros-backend/src/introspection.rs. Each element is a JSON string.
• The new pyo3-introspection parses the built dynamic libs objects, extracts the elements and builds the pyi stub files.
• Testing is done by building the pytests crate, run pyo3-introspection on it and check if the output is equal to the stub files in pytests/stubs. To run the test do nox -s test-introspection.

Note that introspection is only working with modules declared using inline Rust modules (mod XXX) and not with modules declared with functions (fn XXX)

[yogevm15]

Thanks,
I opened a PR for the modules associated consts #5150

I'm not sure about the classes associated consts. If I understood the code correctly, it exports them as class methods rather than actual class variables. So, how should we generate them in the stub?

[purepani]

Thanks for all this work; I'm quite excited for this!
Could you update this issue with the current in progress PRs, and the already merged ones?

[wiktor-k]

I've used experimental-inspect feature of pyo3 0.26 and pyo3_introspection and I must say it already works very, very well! 👏

Two issues found when running mypy on a code which uses generated stubs:

pysequoia.pyi:43: error: "__new__" must return a class instance (got "None")  [misc]

Which is caused by the following fragment of the auto-generated stub:

class Notation:
    def __new__(cls, /, key: str, value: str) -> None: ...

Generated from this Rust-code:

#[pymethods]
impl Notation {
    #[new]
    pub fn new(key: String, value: String) -> Self {
        Self { key, value }
    }
...

It seems the None in the stub case shouldn't be there.

The second issue is more subtle:

pysequoia.pyi:69: error: Argument 1 of "__eq__" is incompatible with supertype "builtins.object"; supertype defines the argument type as "object"  [override]
pysequoia.pyi:69: note: This violates the Liskov substitution principle
pysequoia.pyi:69: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
pysequoia.pyi:69: note: It is recommended for "__eq__" to work with arbitrary objects, for example:
pysequoia.pyi:69: note:     def __eq__(self, other: object) -> bool:
pysequoia.pyi:69: note:         if not isinstance(other, SignatureMode):
pysequoia.pyi:69: note:             return NotImplemented
pysequoia.pyi:69: note:         return <logic to compare two SignatureMode instances>
pysequoia.pyi:70: error: Argument 1 of "__ne__" is incompatible with supertype "builtins.object"; supertype defines the argument type as "object"  [override]
pysequoia.pyi:70: note: This violates the Liskov substitution principle
pysequoia.pyi:70: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides

Caused by this:

class SignatureMode:
    def __eq__(self, /, other: SignatureMode | int) -> bool: ...
    def __ne__(self, /, other: SignatureMode | int) -> bool: ...

Generated from this:

#[pyclass(eq, eq_int)]
#[derive(PartialEq)]
pub enum SignatureMode {
    #[pyo3(name = "INLINE")]
    Inline,
    #[pyo3(name = "DETACHED")]
    Detached,
    #[pyo3(name = "CLEAR")]
    Clear,
}

(Removing eq, eq_int helps). I'm not sure if mypy is overly sensitive here (it wants object instead of SignatureMode | int) as the type describe exactly what can go in there... but maybe it's a separate issue.

(Just in case it helps, here's my code: https://github.com/wiktor-k/pysequoia/pull/51/files#diff-ca800b2c3ea5475c099fdab5613417c051dd4435a00279d4220badc43537bf28R43)

I hope you don't mind me sharing these code fragments here, but if they should go in separate issues, then sorry for the noise. 🙏

Thanks for your work on this!