add #[text_signature = "..."] attribute

[programmerjake]

Based on design at #310 (comment)

Just implemented parsing #[text_signature = "..."] and adding to generated doc strings.
Still need to add docs, tests, changelog entries, etc.

Demo code:

/// doc string 1
#[pyclass]
#[text_signature = "(arg1, *, arg2=0)"]
/// doc string 2
struct MyClass {}

#[pymethods]
impl MyClass {
    #[new]
    #[args(arg1, "*", arg2 = "0")]
    fn new(obj: &PyRawObject, arg1: i32, arg2: i32) {
        obj.init(Self {});
    }

    /// doc string 1
    #[text_signature = "($self)"]
    /// doc string 2
    fn my_method(&self) {}

    #[staticmethod]
    #[text_signature = "()"]
    fn method_without_docs() {}
}

/// doc string 1
#[pyfunction]
#[text_signature = "(arg1, arg2)"]
/// doc string 2
fn my_function(arg1: i32, arg2: i32) -> i32 {
    0
}

[kngwyu]

Thank you.

• We already have 'doc comment to doc str' feature.
  E.g.,

>>> from rustapi_module.buf_and_str import BytesExtractor
>>> b = BytesExtractor()
>>> b.__doc__
'This is for confirming that PyBuffer does not cause memory leak'

So I think what this PR should do is just filling out __text__signature__.

• Why and when is this feature useful? When we generate docs using sphinx?

[programmerjake]

• We already have 'doc comment to doc str' feature.

Yes, I am building this pull request on top of the existing doc comment machinery (and improving error handling too by using syn::Error instead of panic) since both CPython and PyPy currently get __text_signature__ from the provided doc string.

• Why and when is this feature useful? When we generate docs using sphinx?

Being able to set __text_signature__ is useful because:

• it enables function argument auto-completion when using pyls (the Python language server -- like RLS for Python).
• it works with Python's built-in help() (doesn't currently work on #[new] methods, probably a CPython defect)
• it makes inspect.signature() work on functions/methods/classes implemented in Rust
  • used in a lot of Python doc generators (at least pdoc3, probably many more)

This feature is useful when you don't want the special __text_signature__ text to always be at the top of your Rust docs, such as:

• when your API is intended for Rust users as well as just Python users.
• when you use a auto-formatter on doc comments

This also does some checks to make sure the provided signature is valid enough to actually be put in __text_signature__ instead of __doc__ when CPython parses the doc-string.

This is also useful because the #[text_signature = "..."] annotation doesn't have to be at the top of the doc comments and can be ordered anywhere where PyO3 can parse it (much more useful when generating functions using macros):

/// my really long docs for `my_fn`
/// that cover many
/// many lines
#[pyfunction(b = 2, "*", c = "None")]
#[text_signature = "my_fn(a, b=2, *, c=None)"]
/// more docs here
fn my_fn(a: &str, b: i32, c: Option<u32>) -> i16 {
    unimplemented!()
}

This also is much nicer when you want to have __text_signature__ without __doc__:

#[pyfunction]
#[text_signature = "no_docs()"] // simple
fn no_docs() {}

vs.

// hope this is right -- easy to mess up:
// must have exactly 2 trailing "\n"

/// no_docs()
/// --
///
///
#[pyfunction]
fn no_docs() {}

Additional useful property: should CPython (or any other supported Python interpreter) switch to getting __text_signature__ some other way, all that needs to be changed is PyO3 and not all the code using #[text_signature = "..."].

When the code to implement automatically generating __text_signature__ #310 gets around to being implemented, it should be relatively easy to slot that code into the machinery I'm implementing.

[Alexander-N]

I think this makes a lot of sense since it gives a clear way to provide __text_signature__. I didn't know that the __text_signature__ attribute is built from the docstring and wouldn't have made the connection when reading a docstring with the manually constructed special string. Just too bad that type annotations are not supported with this approach. It would be nice if we could replace this mechanism by providing a proper __signature__ in the future.

[programmerjake]

I think this makes a lot of sense since it gives a clear way to provide __text_signature__. I didn't know that the __text_signature__ attribute is built from the docstring and wouldn't have made the connection when reading a docstring with the manually constructed special string. Just too bad that type annotations are not supported with this approach. It would be nice if we could replace this mechanism by providing a proper __signature__ in the future.

That's part of why I ended up picking #[text_signature] instead of #[signature], figuring that #[signature] can be added to support annotations using __signature__ and leaving #[text_signature] for backward-compatibility.

[kngwyu]

@programmerjake
Thank you for detailed explaination.

it enables function argument auto-completion when using pyls (the Python language server -- like RLS for Python).
it works with Python's built-in help() (doesn't currently work on #[new] methods, probably a CPython defect)
it makes inspect.signature() work on functions/methods/classes implemented in Rust
    used in a lot of Python doc generators (at least pdoc3, probably many more)

These are all good.

It would be nice if we could replace this mechanism by providing a proper signature in the future.

Is it a proper/efficient way to provide __annotations__?
CPython built-in types also don't have __signature__.

In [15]: object.__signature__                                                                                             
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-15-e544111936b4> in <module>
----> 1 object.__signature__

AttributeError: type object 'object' has no attribute '__signature__'

In [16]: list.__signature__                                                                                               
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-16-27c4b7eaead5> in <module>
----> 1 list.__signature__

AttributeError: type object 'list' has no attribute '__signature__'

I think what we need for type-annotation is just fill out the __annotations__ field.

And... #[text_signature] looks good, but isn't it better if we can generate __text_signature__ from function declaration?
Do you think we can do this?
If you think we cannot, what is the problem?

[programmerjake]

It would be nice if we could replace this mechanism by providing a proper signature in the future.

Yeah, but that's probably quite a bit harder to implement, so #[text_signature] is a stop-gap solution.

Is it a proper/efficient way to provide __annotations__?

I don't think so, it didn't work with argument annotations when I tried; a return type annotation causes CPython to not detect the signature (__text_signature__ unset, signature left at the beginning of __doc__) due to specifically searching the docstring for ")\n--\n\n".

In the latest CPython git, inspect.signature() ignores __attributes__ when __text_signature__ is set. Assuming CPython's implementation doesn't change, the only way that I can see to support annotations that inspect.signature() detects on builtin functions is to set __signature__.

CPython built-in types also don't have __signature__.

Yes, __signature__ is the mechanism used to override what inspect.signature() returns, none of the built-in functions/methods/types need to do that since __text_signature__ is sufficient for them.

I think what we need for type-annotation is just fill out the __annotations__ field.

And... #[text_signature] looks good, but isn't it better if we can generate __text_signature__ from function declaration?
Is it possible?

yes, but it's probably also much harder to implement, hence why #[text_signature] is useful as a stop-gap measure. Once __text_signature__ generation is implemented, #[text_signature] will also be useful for cases where the user wants a different value than is generated by PyO3.

[konstin]

Is it a proper/efficient way to provide annotations?

Unfortunately not, see #305, python#3208 amd #510 (comment).

I think what we need for type-annotation is just fill out the __annotations__ field.
And... #[text_signature] looks good, but isn't it better if we can generate __text_signature__ from function declaration?
Is it possible?

yes, but it's probably also much harder to implement, hence why #[text_signature] is useful as a stop-gap measure. Once __text_signature__ generation is implemented, #[text_signature] will also be useful for cases where the user wants a different value than is generated by PyO3.

Imo we should eventually autogenerate the text signature, even though this is pratically useful intermediate step.

---

How does this currently interact with rustdoc, i.e. does the signature show up when running cargo doc?

[programmerjake]

Imo we should eventually autogenerate the text signature, even though this is pratically useful intermediate step.

Agreed, though I don't think we should remove #[text_signature] once auto-generation is added since it's useful for overriding the autogenerated result as well as backward-compatibility.

How does this currently interact with rustdoc, i.e. does the signature show up when running cargo doc?

I haven't tested it yet, but the signature shouldn't show up in the Rust doc-string. It will show up as a #[text_signature] annotation if rustdoc shows the pre-macro-expanded code though, which I don't think it does.

[programmerjake]

I finished adding tests and documentation. I switched the syntax for #[text_signature] to mirror the value of __text_signature__ visible in Python:

#[pyfunction(a, b, c=3, "*", d=4, e=5]
// doesn't have function name in text_signature
#[text_signature = "(a, b, c=3, *, d=4, e=5)"]
fn my_function(a: i32, b: i32, c: i32, d: i32, e: i32) -> i32 {
    a + b + c + d + e
}

rather than

#[pyfunction(a, b, c=3, "*", d=4, e=5]
#[text_signature = "my_function(a, b, c=3, *, d=4, e=5)"]
fn my_function(a: i32, b: i32, c: i32, d: i32, e: i32) -> i32 {
    a + b + c + d + e
}

[kngwyu]

These three patterns do almost the same thing and thus we should shorten the code.
E.g.,

FnType::FnNew | FnType::FnCall |.. => {
    if let Some(type_signature) = utils::parse_text_signature_attrs(
        &mut *meth_attrs,
        name
    )? {
        return Err(syn::Error::new_spanned(
            type_signature,
            "type_signature not allowed to use here",
        ));
    }
}

[programmerjake]

the name does need some translation for __new__ and __call__, so it would probably be just as many lines and be more complex to understand.

[kngwyu]

Then you can use macro or closure to construct an error.
I didn't want to say 'you should treat all errors as the same', but this kind of common error throwing code should be packed compactly.

[programmerjake]

fixed

[kngwyu]

put it on the struct definition instead
What does this mean?

[programmerjake]

it means that instead of writing:

#[pyclass]
struct MyClass {
}

#[pymethods]
impl MyClass {
    #[new]
    #[text_signature = "(a, b, c)"]
    fn new(...) {...}
}

This should be written instead:

#[pyclass]
#[text_signature = "(a, b, c)"]
struct MyClass {
}

#[pymethods]
impl MyClass {
    #[new]
    fn new(...) {...}
}

Do you have any suggestions for how to rephrase the error message?

[kngwyu]

#[pyclass]
#[text_signature = "(a, b, c)"]
struct MyClass {
}

So this (a, b, c) is a signature of __init__?
Then I suggest adding 'If you want to add a signature to __init__, '

[programmerjake]

fixed, though I used __new__ instead of __init__ since PyO3 doesn't support __init__

[kngwyu]

Thank you.
It looks well, but since proc-macro code is often hard to maintain, I think we should be careful not to make the code too complex.

[kngwyu]

Thank you, it's a pretty good feature!

[LucaCappelletti94]

Is the notation with the default values in the signature actually supported or was it dropped?

[davidhewitt]

There is a test for that still present here:

pyo3/tests/test_text_signature.rs

Line 104 in 9ea4733

fn test_function() {