[ty] Synthetic function-like callables

[sharkdp]

Summary

We create Callable types for synthesized functions like the __init__ method of a dataclass. These generated functions are real functions though, with descriptor-like behavior. That is, they can bind self when accessed on an instance. This was modeled incorrectly so far.

Test Plan

Updated tests

[github-actions]

mypy_primer results

Changes were detected when running on open source projects

attrs (https://github.com/python-attrs/attrs)
- info[revealed-type] tests/dataclass_transform_example.py:13:13: Revealed type: `(a: str, b: int) -> None`
+ info[revealed-type] tests/dataclass_transform_example.py:13:13: Revealed type: `(self: Define, a: str, b: int) -> None`
- info[revealed-type] tests/dataclass_transform_example.py:21:13: Revealed type: `(with_converter: int = Unknown) -> None`
+ info[revealed-type] tests/dataclass_transform_example.py:21:13: Revealed type: `(self: DefineConverter, with_converter: int = Unknown) -> None`
- warning[possibly-unbound-attribute] tests/test_dunders.py:1008:13: Attribute `__code__` on type `Unknown | (() -> None)` is possibly unbound
- warning[possibly-unbound-attribute] tests/test_dunders.py:1020:13: Attribute `__code__` on type `Unknown | (() -> None)` is possibly unbound
- error[unresolved-attribute] tests/test_dunders.py:1032:13: Type `() -> None` has no attribute `__code__`
- Found 621 diagnostics
+ Found 618 diagnostics

[AlexWaygood]

- warning[possibly-unbound-attribute] tests/test_dunders.py:1008:13: Attribute `__code__` on type `Unknown | (() -> None)` is possibly unbound
+ warning[possibly-unbound-attribute] tests/test_dunders.py:1008:13: Attribute `__code__` on type `Unknown | ((self: C) -> None)` is possibly unbound

hmm, it seems incorrect that we're emitting this diagnostic. It definitely exists at runtime:

>>> def f(): ...
... 
>>> f.__code__
<code object f at 0x105d60b90, file "<python-input-0>", line 1>
>>> class Foo:
...     def f(): ...
...     
>>> Foo().f.__code__
<code object f at 0x105d60c60, file "<python-input-2>", line 2>

and it's annotated as existing on FunctionType instances in typeshed:

ruff/crates/ty_vendored/vendor/typeshed/stdlib/types.pyi

Lines 72 to 76 in cb9e669

	@final
	class FunctionType:
	@property
	def __closure__(self) -> tuple[CellType, ...] | None: ...
	__code__: CodeType

I suppose we need to fallback to Instance("FunctionType") for member access on CallableType inhabitants if is_function_like is set to true?

[AlexWaygood]

Are CallableType inhabitants with is_function_like: true modeled as (consistent) subtypes of Instance("types.FunctionType")? I think they should be -- could you add a test?

[AlexWaygood]

I guess I'm not totally sold that these should be modeled as a special case of CallableType rather than being modeled as a special case of FunctionLiteral (or being their own Type variant entirely) -- the subtyping/assignability and member-access questions seem to me like function-like CallableTypes behave pretty differently to non-function-like CallableTypes

[carljm]

I think it would be important to see some new subtyping and assignability tests (and the code to make them pass, which I also don't think I'm seeing here?), which should verify several things:

• A fully-static callable type that is function-like is a subtype of the same callable type that is not function-like, but not the other way around. (Unless I'm missing something, it seems like this PR currently doesn't consider is_function_like in callable-subtyping at all, which I think would mean it fails to recognize the "not" part of this?)
• Same for assignability.
• A function literal is a subtype of its own into_callable_type (this could almost be a property test?)

[carljm]

I think we could be a bit more precise in our description here, i.e. a callable type with is_function_like: True is inhabited by callable objects with the given signature that also have a __get__ method which acts like FunctionType.__get__ (returns a bound method).

[sharkdp]

```diff
+ warning[possibly-unbound-attribute] tests/test_dunders.py:1008:13: Attribute `__code__` on type `Unknown | ((self: C) -> None)` is possibly unbound

hmm, it seems incorrect that we're emitting this diagnostic. It definitely exists at runtime:

Yes, thanks! At runtime, it also exists for generated methods on dataclasses. This should be fixed now.

[sharkdp]

Are CallableType inhabitants with is_function_like: true modeled as (consistent) subtypes of Instance("types.FunctionType")? I think they should be -- could you add a test?

Yes, thanks. Added the test, but haven't fixed it yet.

I guess I'm not totally sold that these should be modeled as a special case of CallableType rather than being modeled as a special case of FunctionLiteral (or being their own Type variant entirely) -- the subtyping/assignability and member-access questions seem to me like function-like CallableTypes behave pretty differently to non-function-like CallableTypes

I originally planned it this way, but somehow accidentally did the "reverse" today and special-cased Type::Callable, and wondered why it was so much easier than I had anticipated 😄. I will re-evaluate this after quickly trying to implement subtyping with the existing approach.

[sharkdp]

I think it would be important to see some new subtyping and assignability tests (and the code to make them pass, which I also don't think I'm seeing here?), which should verify several things:

• A fully-static callable type that is function-like is a subtype of the same callable type that is not function-like, but not the other way around. (Unless I'm missing something, it seems like this PR currently doesn't consider is_function_like in callable-subtyping at all, which I think would mean it fails to recognize the "not" part of this?)

• Same for assignability.

So this is interesting. I added the tests, and they.. just pass. It works not because we're looking at the is_function_like attribute, but rather because the corresponding synthesized callable has positional-or-keyword parameters with names, whereas the equivalent callable type has positional-only parameters without any name.

type DunderInitType = TypeOf[C.__init__]  # (self: C, x: int) -> None
type EquivalentCallableType = Callable[[C, int], None]  # (C, int, /) -> None

This is probably not a great way to ensure proper subtyping/assignability properties, so I can look into incorporating is_function_like somehow.

[carljm]

I can look into incorporating is_function_like somehow

I think the implementation shouldn't be too difficult? Just a check in is_assignable_to_impl that the two is_function_like, if they differ, differ only in the right direction.

Writing a failing test may be more of the challenge, but I think if you also define EquivalentCallableType by writing an actual function and then using CallableTypeOf on it, that might work?

[AlexWaygood]

somehow accidentally did the "reverse" today and special-cased Type::Callable, and wondered why it was so much easier than I had anticipated

it being "easier" doesn't necessarily reassure me ;) that might just mean that we've accidentally omitted some branches in some places that we would otherwise be forced to add if it was its own variant 😆

[carljm]

If we do have this as a form of Callable type, it might mean that we could be "smart" about method_attr: Callable[[Self], int] = function_object vs non_method_attr: Callable[[], int] = other_non_function_like_callable_object, where we actually use the type of the RHS as a hint to decide whether we consider the declaration to be of a "method descriptor" or "non method descriptor" callable type. That could let us avoid some unsoundness? See #18167 (comment)

On the other hand, it's weird to interpret the same annotation differently depending on the RHS. So maybe we shouldn't do that.

[sharkdp]

Opening this up for review again. It passes all tests and resolves some ecosystem false positives. I am happy to spend (significantly?) more time on this, trying to rewrite it in terms of a customized FunctionType variant. I agree with @AlexWaygood that this might be less error-prone. But given that we currently only use this for one edge case (explicit calls to a synthetic dunder method on a dataclass, as in MyDataClass.__lt__(…)), I'm not sure it's worth it? The answer might depend on whether or not we foresee using this for purposes of resolving astral-sh/ty#491.

[carljm]

I think it's fine to go with this for now.

[AlexWaygood]

given that we currently only use this for one edge case (explicit calls to a synthetic dunder method on a dataclass, as in MyDataClass.__lt__(…)), I'm not sure it's worth it?

But I think we should use function-like callables for lambdas too, since they're also instances of FunctionType at runtime (they therefore have all those attributes that you can find on instances of types.FunctionType, and can be used as methods just fine if you assign them to symbols in class namespaces()