Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Since every class has `object` in it's MRO, the default implementations are `obj
- If neither `__new__` nor `__init__` are defined anywhere in the MRO of class (except for `object`)
\- no arguments are accepted and `TypeError` is raised if any are passed.
- If `__new__` is defined, but `__init__` is not - `object.__init__` will allow arbitrary arguments!
- If `__init__` is defined, but `__new__` is not - `object.__new__` will allow arbitrary arguments!

As of today there are a number of behaviors that we do not support:

Expand Down Expand Up @@ -323,3 +324,23 @@ reveal_type(Foo(1)) # revealed: Foo
# error: [too-many-positional-arguments] "Too many positional arguments to bound method `__init__`: expected 1, got 2"
reveal_type(Foo(1, 2)) # revealed: Foo
```

## `__new__` defined on a meta-class should not be used

We lookup `__new__` method using descriptor protocol, which technically allows it to be found on a
meta-class if `__new__` is not defined anywher in class MRO. At runtime this is never the case,
since all classes have `object` in their MRO, which has a `__new__` method. However, for the
purposes of type checking we use special lookup logic that ignores `object.__new__` as it's runtime
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
purposes of type checking we use special lookup logic that ignores `object.__new__` as it's runtime
purposes of type checking we use special lookup logic that ignores `object.__new__` as its runtime

behavior is not expressible in typeshed. This not cause false-positives when `__new__` is defined on
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
behavior is not expressible in typeshed. This not cause false-positives when `__new__` is defined on
behavior is not expressible in typeshed. This should not cause false-positives when `__new__` is defined on

a meta-class.
Comment on lines +330 to +335
Copy link
Member

@AlexWaygood AlexWaygood Apr 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We lookup `__new__` method using descriptor protocol, which technically allows it to be found on a
meta-class if `__new__` is not defined anywher in class MRO. At runtime this is never the case,
since all classes have `object` in their MRO, which has a `__new__` method. However, for the
purposes of type checking we use special lookup logic that ignores `object.__new__` as it's runtime
behavior is not expressible in typeshed. This not cause false-positives when `__new__` is defined on
a meta-class.
We lookup `__new__` methods using the descriptor protocol, which technically allows it to be found on a
metaclass if `__new__` is not defined anywhere in a class's MRO. At runtime this is never the case,
since all classes have `object` in their MRO, and `object` has a `__new__` method. However, for the
purposes of type checking we use special lookup logic that ignores `object.__new__` as its runtime
behavior is not expressible in typeshed. This does not cause false-positives when `__new__` is defined on
a metaclass.


```py
class Meta(type):
def __new__(mcls, name, bases, namespace, /, **kwargs):
return super().__new__(mcls, name, bases, namespace)

class Foo(metaclass=Meta): ...

# This should not raise an error
reveal_type(Foo()) # revealed: Foo
```
27 changes: 22 additions & 5 deletions crates/red_knot_python_semantic/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,7 @@ bitflags! {

/// When looking up an attribute on a class, we sometimes need to avoid
/// looking up attributes defined on `type` if this is the metaclass of the class.
/// For example, this is used to find if a meta-class implements a `__call__` method.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// For example, this is used to find if a meta-class implements a `__call__` method.
/// For example, this is used to find if a metaclass implements a `__call__` method.

///
/// This is similar to no object fallback above
const META_CLASS_NO_TYPE_FALLBACK = 1 << 2;
Expand Down Expand Up @@ -2641,12 +2642,28 @@ impl<'db> Type<'db> {
qualifiers: meta_attr_qualifiers,
},
meta_attr_kind,
) = Self::try_call_dunder_get_on_attribute(
db,
self.class_member_with_policy(db, name.into(), member_policy),
) = if matches!(
self,
self.to_meta_type(db),
);
Type::ClassLiteral(_) | Type::GenericAlias(_) | Type::SubclassOf(_)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This match on these specific type variants here looks suspicious to me; I think I need a clearer statement of why this is correct? Tests fail without it, so it's clearly not redundant.

It seems like we are using "is self one of these three variants" as a proxy for "are we currently looking up an attribute on the meta type", but I don't think that's really a correct proxy? Or at least, if it's correct today, it's making assumptions that could be invalidated in future. (For instance, Instance(<builtins.type>) could definitely be a meta-type -- I can't remember if we normalize that to SubclassOf(<builtins.object>) currently.)

I'd have to spend more time digging into details to make this more concrete, but it feels like there should be some place higher up the call chain where we should know for certain that we are about to look up the attribute on the meta type, and that's the place where we should be making a decision about this case (and if necessary, passing some policy or context down to here to reflect that fact).

) && member_policy.mro_no_object_fallback()
{
// When looking up the attribute on the meta-type with `NO_OBJECT_FALLBACK` policy, we
// imply that the looked up attribute exists on `object` even if we do not want to fall back
// back to it. This in turn means that descriptor protocol will not select same named
// data-descriptor on the meta-type - hence we return `SymbolAndQualifiers::default()` here,
// which is essentially a `Symbol::Unbound` with no qualifiers.
(
SymbolAndQualifiers::default(),
AttributeKind::NormalOrNonDataDescriptor,
)
} else {
Self::try_call_dunder_get_on_attribute(
db,
self.class_member_with_policy(db, name.into(), member_policy),
self,
self.to_meta_type(db),
)
};

let SymbolAndQualifiers {
symbol: fallback,
Expand Down
15 changes: 9 additions & 6 deletions crates/red_knot_python_semantic/src/types/class.rs
Original file line number Diff line number Diff line change
Expand Up @@ -879,12 +879,15 @@ impl<'db> ClassLiteral<'db> {
continue;
}

// HACK: we should implement some more general logic here that supports arbitrary custom
// metaclasses, not just `type` and `ABCMeta`.
if matches!(
class.known(db),
Some(KnownClass::Type | KnownClass::ABCMeta)
) && policy.meta_class_no_type_fallback()
// Some meta-class methods such as `__call__` need special treatment depending on
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Some meta-class methods such as `__call__` need special treatment depending on
// Some metaclass methods such as `__call__` need special treatment depending on

// whether they are explicitly defined, or inherited from `type.__call__`.
// We use `MemberLookupPolicy::META_CLASS_NO_TYPE_FALLBACK` flag and it's getter
// `policy.meta_class_no_type_fallback` to allow downstream code to check whether
// a custom implementation of such symbol exists on a type.
// E.g. this is currently used in `ClassLiteral::into_callable` which passes this
// policy here through a chain of calls.
if matches!(class.known(db), Some(KnownClass::Type))
&& policy.meta_class_no_type_fallback()
{
continue;
}
Comment on lines +889 to 893
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why we still need to add some special casing for type specifically here? I see some tests fail in is_subtype_of.md without this, but I don't understand why we need to continue if the class is type but not if it's some type subclass. Adding a comment here that explains this would be very helpful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexWaygood yeah, I initially removed it and then stumbled on failing tests and figured out it is not possible (at least without refactoring into_callable logic). I've updated the policy bit flags enum docs and mentioned the reason in the description of the PR, but agree that adding additional comment at the use site would be helpful for future readers. I'll wait for other reviews and will add.

Here is my line from the PR for the underlying reason:

We can't get rid of META_CLASS_NO_TYPE_FALLBACK as it is necessary to figure out if a given meta-class has an explicit __call__ method defined, not inherited from type. This is used in into_callable implementation

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is my line from the PR for the underlying reason:

I see -- I didn't realise that that note from your PR description was referring to this, since this method is called Type::class_member_from_mro rather than Type::into_callable :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do better with wording! Thanks for pointing this out. I've just pushed a commit with a comment, hopefully it is clear enough. Let me know, I can continue exercising my English if it doesn't!

Expand Down
Loading