chore: Fix `IntoSchema` typing, allow to pass `Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]` #3266

FBruzzesi · 2025-10-31T20:57:06Z

What type of PR is this? (check all applicable)

Related issues

Closes typing.IntoSchema too narrow #3257

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

FBruzzesi · 2025-10-31T20:58:31Z

narwhals/typing.py

-# TODO @dangotbanned: fix this?
-# Constructor allows tuples, but we don't support that *everywhere* yet
-IntoSchema: TypeAlias = "Mapping[str, dtypes.DType] | Schema"
+IntoSchema: TypeAlias = (
+    "Mapping[str, IntoDType] | Sequence[tuple[str, IntoDType]] | Schema"
+)


@dangotbanned I hope you are proud 😇

🥳

But now to ruin your day slightly ...

We got our typing for Schema.__init__ from polars, but even this is too strict.
Here's what typeshed has for dict, which we support (except **kwargs)

https://github.com/python/typeshed/blob/bf7214784877c52638844c065360d4814fae4c65/stdlib/builtins.pyi#L1158-L1187

Luckily we don't need to use any overloads, but we should have this in IntoSchema somewhere:

Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]

Interesting timing though, I just pushed a test where I passed a non-Mapping object to nw.Schema 😂

narwhals/tests/plan/schema_test.py

Lines 18 to 20 in 1579b32

# NOTE: Would type-check if `Schema.__init__` didn't make liskov unhappy

assert schema == nw.Schema(frozen_schema) # type: ignore[arg-type]

assert mapping == dict(frozen_schema)

Thanks @dangotbanned

Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]

SupportsKeysAndGetItem what the hell is this 😂 how's Mapping not enough anymore?

Regarding Iterable, I always have a hard time being comfortable with it, since it can be infinite (and lazy).

How does infinite converts to a dict?

Lazy makes it a bit more of a headache when we want to check it's content (e.g. first element is a string or a tuple), but that's alright

This is the reason why I went with Sequence, although polars Schema.__init__ signature is with the Iterable

We got our typing for Schema.init from polars, but even this is too strict.

can we resolve that later and just get the type[Dtype] / Dtype part sorted out in this one?

SupportsKeysAndGetItem what the hell is this 😂 how's Mapping not enough anymore?

Lol, it is an object that supports those methods ofc 😉

just import it in a TYPE_CHECKING block from here

https://github.com/python/typeshed/blob/be34e9201db75891a67d4d3ce5e5705ee6636f6f/stdlib/_typeshed/__init__.pyi#L161-L164

from _typeshed import SupportsKeysAndGetItem

Regarding Iterable, I always have a hard time being comfortable with it, since it can be infinite

If someone passes an infinite stream to an initializer - they need to learn the lesson that the call will never complete

(and lazy).

This is a desirable property for the caller and I think if we want to validate things, then that is our responsibility

Reducing scope

How about aligning just Schema.__init__ with dict.__init__?

We are only checking for None there, so the issues you have seem to not apply there

@dangotbanned attempt in 9ce61fe, which now fails for mypy when the dict has mix of initialized and un-initialized dtypes, which was the initial purpose of the PR 🤔

Edit: By changing Iterable into Sequence there are no issue (see 0e0d7a8). I am personally not interested in diving deeper than this.

Last to fix is the forward reference for tubular. Which I am unsure if it's our fault or them not importing something within a if TYPE_CHEKING: block

FBruzzesi · 2025-10-31T20:59:08Z

tests/frame/schema_test.py

Randomly removing some initialization to test the typechecking

FBruzzesi · 2025-10-31T20:59:24Z

narwhals/typing.py

+IntoNullableSchema: TypeAlias = (
+    "Mapping[str, IntoDType | None] | Sequence[tuple[str, IntoDType | None]]"
+)
+"""Schema specification with possible None values."""


TODO: Please let's come up with a better description

FBruzzesi · 2025-10-31T21:02:56Z

narwhals/_utils.py

    return (value,) * n_match if isinstance(value, bool) else tuple(value)
+
+
+class NullableSchema(OrderedDict[str, "IntoDType | None"]):


Reason for this class is mostly two folded:

Use as a utility to convert a Sequence[tuple[str, DType {|None}]] into a mapping. Hence make it easy to use the same API (key, dtype in obj.items()).

Easily set a flag to know if any value passed is None

The name Nullable* is making me think this is related to #3176 (comment) again 🫣

I'm not opposed to having classes to make our internal API cleaner btw 👍

The name Nullable* is making me think this is related to #3176 (comment) again 🫣

Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'm not opposed to having classes to make our internal API cleaner btw 👍

Any preference in having it prefixed by _?

Any preference in having it prefixed by _?

It is already in _utils, so no need for a prefix 🙂

The current _ names are an artifact from when we had everything in utils.py

Not sure I see how it's related. Passing None seems more like to be a free card, "live and let live" kind of behavior.

I'll try to give a more complete explanation later, but I was hinting at Nullable, Null, None being a bit overloaded.
Linking to (#3176 (comment)) was supposed to show that I made this mistake already 😂

narwhals/dataframe.py

FBruzzesi · 2025-11-01T08:21:02Z

Regarding CI failures: the only one related to this PR seem to be tubular 👀

MarcoGorelli · 2025-11-02T11:43:03Z

narwhals/typing.py

    "IntoFrameT",
    "IntoLazyFrame",
    "IntoLazyFrameT",
+    "IntoNullableSchema",


Are we sure about making this a public type?

Is it possible to only focus on Dtype -> intodtype changes here without introducing new public types?

Ok I honestly think there might be something wrong in the PR to begin with. If I refocus on the basic changes to fix #3257, then passing a dictionary with mixed initialized and un-initialized dtypes does still complain (pyright is ok with it, mypy is not)

chore: Fix IntoSchema typing, allow Sequence[tuple[str, IntoDType]

324a8dc

FBruzzesi added enhancement New feature or request typing labels Oct 31, 2025

FBruzzesi mentioned this pull request Oct 31, 2025

chore: IntoSchema overhaul #3262

Closed

10 tasks

Merge branch 'main' into chore/into-schema-overhaul-redo

983db99

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow Sequence[tuple[str, IntoDType]~~ chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] Oct 31, 2025

FBruzzesi commented Oct 31, 2025

View reviewed changes

tests/frame/schema_test.py

Copy link

Member Author

FBruzzesi Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Randomly removing some initialization to test the typechecking

FBruzzesi commented Oct 31, 2025

View reviewed changes

missed one annotation

ab0bfab

FBruzzesi commented Oct 31, 2025

View reviewed changes

narwhals/dataframe.py Outdated Show resolved Hide resolved

FBruzzesi and others added 2 commits November 1, 2025 09:20

pragma no cover exception

88136c3

Merge branch 'main' into chore/into-schema-overhaul-redo

12bd5e9

FBruzzesi added 4 commits November 1, 2025 09:22

tricking ruff for better formatting

476dc69

is exporting enough for forward reference?

c90c110

delay NullableSchema import

38bac0e

Use SupportsKeysAndGetItem and Iterable

9ce61fe

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]]~~ chore: Fix IntoSchema typing, allow to pass Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] Nov 1, 2025

one for two?

0e0d7a8

FBruzzesi changed the title ~~chore: Fix IntoSchema typing, allow to pass Iterable[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]~~ chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] Nov 1, 2025

merge main

15e08d7

MarcoGorelli reviewed Nov 2, 2025

View reviewed changes

lower diff, do not export SchemaDefinition

634bd33

	# NOTE: Would type-check if `Schema.__init__` didn't make liskov unhappy
	assert schema == nw.Schema(frozen_schema) # type: ignore[arg-type]
	assert mapping == dict(frozen_schema)

		return (value,) * n_match if isinstance(value, bool) else tuple(value)


		class NullableSchema(OrderedDict[str, "IntoDType \| None"]):

chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] #3266

Are you sure you want to change the base?

chore: Fix IntoSchema typing, allow to pass Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType] #3266

Uh oh!

Conversation

FBruzzesi commented Oct 31, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Reducing scope

Uh oh!

FBruzzesi Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FBruzzesi commented Nov 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chore: Fix `IntoSchema` typing, allow to pass `Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]` #3266

chore: Fix `IntoSchema` typing, allow to pass `Sequence[tuple[str, IntoDType]] | SupportsKeysAndGetItem[str, IntoDType]` #3266

dangotbanned Oct 31, 2025 •

edited

Loading

FBruzzesi Nov 1, 2025 •

edited

Loading

FBruzzesi Oct 31, 2025 •

edited

Loading

FBruzzesi Oct 31, 2025 •

edited

Loading

FBruzzesi Nov 1, 2025 •

edited

Loading

FBruzzesi Nov 2, 2025 •

edited

Loading