feat(DRAFT): Add get_supertype

[dangotbanned]

Description

Important

@FBruzzesi and I have been + are still iterating on this
Core functionality is there, focusing on readability, performance + shrinking the test suite

This PR implements polars' concept of supertyping - which more generally defines which types can be safely promoted/demoted/cast to other types.

I really like the DuckDB visualization of their version¹ of these rules, so here's that for an example:

Show Casting Operations Matrix

This is a preliminary step for implementing relaxed concat (#3386).
The aim is we own a consistent set of rules that all/most backends can participate in.
We've already dropped some supertypes that are valid in polars, but may prove challenging in other backends such as #121.
Some others are directly mentioned in comments (e.g. (Struct, DType) -> Struct)

Additional use-cases

Supertyping in polars is used for much more than just a subset of concat.
In (#2572), it is one of the larger concepts missing from the intermediate representation (see #3386 (comment)).

polars-plan::plans::conversion::type_coercion is full of examples of how deeply related the concept is with expressions.
My aim is not to reproduce all of that 😅 - but to be able to reason about DTypes between LazyFrame operations without querying the backend for a Schema between every step 🤞

Related issues

• Prep for [Enh]: Add support for concat(..., how={"vertical_relaxed", "diagonal_relaxed"}) #3386
• chore: Add DType.__call__ #3393
• feat: Improve support for Decimal DType #3377

Tasks

• Testing
  • Fix test id silently skipping with NOTSET (244537d)
  • Split up into more easily parametric tests
    • (443d9ccd)
    • Struct
    • Split out DType fixtures
      • (see thread)
  • Add cases to ensure e.g (v1.Datetime, Datetime) -> None
    • (see thread)
• Remove (now) unused things like _CACHE_SIZE_TP_HIGH
• Rename narwhals.dtypes.classes -> narwhals.dtypes._classes (d2c96fe)
• Rename & Move narwhals.stable.v1._dtypes -> narwhals.dtypes._classes_v1
  • (see thread)
• Try to balance set operation readability + performance
  • both are important, but "Readability counts"
  • _has_intersection
  • _first_excluding
• Update Decimal handling #3377
  • (see thread)
• Add the remaining String downcasts (see thread) 8d9e053
  • Primitive
  • Parametric (temporal weirdness)
  • Do we need to/can we support (Nested, String)?
• Add {List, Array} -> List 3ad1639
• Docs
  • Visualizing the rules, @FBruzzesi's "dirty" table (see comment and extra summary)
    • chore: Add promotion-rules.md script
    • docs: Manually generate promotion-rules.md
    • feat(DRAFT): Add get_supertype #3396 (comment)
    • feat(DRAFT): Add get_supertype #3396 (comment)
  • Convert comments into docstrings
    • docs: Convert _mixed_supertype comments
    • docs: Convert _numeric_supertype comments
    • docs: Some attempt at explaining Struct
    • refactor: merge dispatch tables, add docs
  • Module doc, to tie the functions together
• feat: Add concat(..., how="*_relaxed"}) #3398
• Add a small @lru_cache on a wrapper for DType.__eq__
  • Can be used in both _struct_fields_union and get_supertype

Footnotes

1. DuckDB also mentions another set of rules called Combination Casting - that is entirely implicit.
   The matrix doesn't relfect these and only one cast example is given, but it would apply to nw.concat:
   "This combination casting occurs for ..., set operations (UNION / EXCEPT / INTERSECT), and ..." ↩

[FBruzzesi]

@dangotbanned I am not a big fan of these naming to be honest. The TypeVar is already suggesting that two object will be the same.

I think renaming as:

-SameTemporalT = TypeVar("SameTemporalT", Datetime, DatetimeV1, Duration, DurationV1)
+TemporalT = TypeVar("TemporalT", Datetime, DatetimeV1, Duration, DurationV1)
 """Temporal data types, with a `time_unit` attribute."""
 
-SameDatetimeT = TypeVar("SameDatetimeT", Datetime, DatetimeV1)
-SameT = TypeVar(
-    "SameT", Array, List, Struct, Datetime, DatetimeV1, Duration, DurationV1, Enum
+DatetimeT = TypeVar("DatetimeT", Datetime, DatetimeV1)
+ParametricT = TypeVar(
+    "ParametricT", Array, List, Struct, Datetime, DatetimeV1, Duration, DurationV1, Enum
 )

would be better

[dangotbanned]

I'm trying to convey the difference between constrained and bound TypeVars.

• https://typing.python.org/en/latest/spec/generics.html#introduction
• https://typing.python.org/en/latest/spec/generics.html#type-variables-with-an-upper-bound

The only convention I've seen for constrained is like this:

AorB = TypeVar("AorB", A, B)

But it doesn't scale well to more than 2 types 😔

So I've been using Same* for this purpose.

A more verbose, but probably more accurate name would be:

EitherABCD = TypeVar("EitherABCD", A, B, C, D)

The main point is this kind of typing will reject A | B, meaning you need to have narrowed to exactly one of the constraints

[dangotbanned]

A few notes on this:

I've kept this as one big comment since we're already at 77! 😳

(1) Could we be lazy-er?

I would prefer if we defer generating this until it is needed.

E.g. I'd expect _integer_supertyping and _primitive_numeric_supertyping to be more commonly used - but even they don't exist at module-import-time

(2) DType vs type[DType] keys

I think this is the only place we have instances as mapping keys, not sure why?

For example, it means each call here instantiates more DTypes, when we could just use the type itself 😅

narwhals/narwhals/dtypes/_supertyping.py

Line 355 in 548e5b8

if integer in {UInt128(), Int128()}:

(3) NumericType.max?

I had a look upstream and it seems in the direction of (#3396 (comment)) and (#3396 (comment)).

What do you think about adding these maximums to the classes, (similar to _bits)?
That way we could compare directly and probably avoid the lookup table

(4) Minor tweak

I think this is the more efficient way to do these calculations.

Note

It would be exactly the second time I've found a use-case for bitshifting operators 😂

INT_MAX_MAP: Mapping[IntegerType, int] = {
-    UInt8(): (2**8) - 1,
-    UInt16(): (2**16) - 1,
-    UInt32(): (2**32) - 1,
-    UInt64(): (2**64) - 1,
-    Int8(): (2**7) - 1,
-    Int16(): (2**15) - 1,
-    Int32(): (2**31) - 1,
-    Int64(): (2**63) - 1,
+    UInt8(): (1 << 8) - 1,
+    UInt16(): (1 << 16) - 1,
+    UInt32(): (1 << 32) - 1,
+    UInt64(): (1 << 64) - 1,
+    Int8(): (1 << 7) - 1,
+    Int16(): (1 << 15) - 1,
+    Int32(): (1 << 31) - 1,
+    Int64(): (1 << 63) - 1,
}