feat: Add `DType.base_type` #2969

dangotbanned · 2025-08-10T20:44:36Z

What type of PR is this? (check all applicable)

Related issues

Should make this easier as well ([Enh]: Add DType.to_<backend>() #2486)

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

Incredibly simple feature, but made a big refactor possible (starting from 3070470)

Based on polars.DataType.base_type

See (#2969 (comment)) for details

Incredibly simple, but will allow a big a refactor https://github.com/pola-rs/polars/blob/34595af53dbdcbde52ab062746a42aa725e321a4/py-polars/polars/datatypes/classes.py#L145-L159

Either VSCode or `iPython` is showing me the wrong repr for classes 🤦‍♂️

dangotbanned · 2025-08-11T09:22:11Z

narwhals/dtypes.py

            >>> import narwhals as nw
            >>> nw.Datetime("us").base_type()
-            narwhals.dtypes.Datetime
+            <class 'narwhals.dtypes.Datetime'>


For reference, we could have the nicer repr from polars - but would need to add a metaclass for DType.
I think this is more user friendly, given that we tret the types and objects and interchangeable in most cases

@classmethod def base_type(cls) -> DataTypeClass: """ Return this DataType's fundamental/root type class. Examples -------- >>> pl.Datetime("ns").base_type() Datetime >>> pl.List(pl.Int32).base_type() List >>> pl.Struct([pl.Field("a", pl.Int64), pl.Field("b", pl.Boolean)]).base_type() Struct """ return cls

Fair enough though if not having this was an intentional choice

https://github.com/pola-rs/polars/blob/97fb4c8a49f259b3d614a6f1f3c58994bda5d7eb/py-polars/polars/datatypes/classes.py#L145-L159

https://github.com/pola-rs/polars/blob/97fb4c8a49f259b3d614a6f1f3c58994bda5d7eb/py-polars/polars/datatypes/classes.py#L51-L58

A tweak that comes to mind is this:

import narwhals as nw >>> nw.List nw.List >>> nw.List(nw.Int32) nw.List(nw.Int32)

And then we can qualify the class name a bit more for v1's Datetime, Duration and Enum only

dangotbanned · 2025-08-11T13:26:46Z

narwhals/_ibis/utils.py

+dtypes = Version.MAIN.dtypes
+NW_TO_IBIS_DTYPES: Mapping[type[DType], IbisDataType] = {
+    dtypes.Float64: ibis_dtypes.Float64(),
+    dtypes.Float32: ibis_dtypes.Float32(),
+    dtypes.Binary: ibis_dtypes.Binary(),
+    dtypes.String: ibis_dtypes.String(),
+    dtypes.Boolean: ibis_dtypes.Boolean(),
+    dtypes.Date: ibis_dtypes.Date(),
+    dtypes.Time: ibis_dtypes.Time(),
+    dtypes.Int8: ibis_dtypes.Int8(),
+    dtypes.Int16: ibis_dtypes.Int16(),
+    dtypes.Int32: ibis_dtypes.Int32(),
+    dtypes.Int64: ibis_dtypes.Int64(),
+    dtypes.UInt8: ibis_dtypes.UInt8(),
+    dtypes.UInt16: ibis_dtypes.UInt16(),
+    dtypes.UInt32: ibis_dtypes.UInt32(),
+    dtypes.UInt64: ibis_dtypes.UInt64(),
+    dtypes.Decimal: ibis_dtypes.Decimal(),
+}
+
+
+def narwhals_to_native_dtype(  # noqa: C901
    dtype: IntoDType, version: Version
 ) -> IbisDataType:
    dtypes = version.dtypes
-
-    if isinstance_or_issubclass(dtype, dtypes.Decimal):  # pragma: no cover
-        return ibis_dtypes.Decimal()
-    if isinstance_or_issubclass(dtype, dtypes.Float64):
-        return ibis_dtypes.Float64()
-    if isinstance_or_issubclass(dtype, dtypes.Float32):
-        return ibis_dtypes.Float32()
-    if isinstance_or_issubclass(dtype, dtypes.Int128):  # pragma: no cover
-        msg = "Int128 not supported by Ibis"
-        raise NotImplementedError(msg)
-    if isinstance_or_issubclass(dtype, dtypes.Int64):
-        return ibis_dtypes.Int64()
-    if isinstance_or_issubclass(dtype, dtypes.Int32):
-        return ibis_dtypes.Int32()
-    if isinstance_or_issubclass(dtype, dtypes.Int16):
-        return ibis_dtypes.Int16()
-    if isinstance_or_issubclass(dtype, dtypes.Int8):
-        return ibis_dtypes.Int8()
-    if isinstance_or_issubclass(dtype, dtypes.UInt128):  # pragma: no cover
-        msg = "UInt128 not supported by Ibis"
-        raise NotImplementedError(msg)
-    if isinstance_or_issubclass(dtype, dtypes.UInt64):
-        return ibis_dtypes.UInt64()
-    if isinstance_or_issubclass(dtype, dtypes.UInt32):
-        return ibis_dtypes.UInt32()
-    if isinstance_or_issubclass(dtype, dtypes.UInt16):
-        return ibis_dtypes.UInt16()
-    if isinstance_or_issubclass(dtype, dtypes.UInt8):
-        return ibis_dtypes.UInt8()
-    if isinstance_or_issubclass(dtype, dtypes.String):
-        return ibis_dtypes.String()
-    if isinstance_or_issubclass(dtype, dtypes.Boolean):
-        return ibis_dtypes.Boolean()
-    if isinstance_or_issubclass(dtype, dtypes.Categorical):
-        msg = "Categorical not supported by Ibis"
-        raise NotImplementedError(msg)
+    if ibis_type := NW_TO_IBIS_DTYPES.get(dtype.base_type()):
+        return ibis_type


Not sure if the diff paints a clear enough picture

Example (String() | String)

On main, we need to go through 13x failed isinstance_or_issubclass calls, before finally reaching the correct one.

Inside that, every time 2x isinstance calls are made

_ibis.utils.narwhals_to_native_dtype(nw.String(), ...) _ibis.utils.narwhals_to_native_dtype(nw.String, ...)

narwhals/narwhals/_utils.py

Lines 843 to 848 in 9e994f3

def isinstance_or_issubclass(obj_or_cls: Any, cls_or_tuple: Any) -> bool:

from narwhals.dtypes import DType

if isinstance(obj_or_cls, DType):

return isinstance(obj_or_cls, cls_or_tuple)

return isinstance(obj_or_cls, cls_or_tuple) or (

So either variation of nw.String currently always requires 26x failed isinstance calls, +2 once we get to the correct case

Optimization

Now most cases are now handled in a single dict lookup (16 in total for ibis) 🥳

We can do this for at least polars, pyarrow, ibis, duckdb - and probably with modifications in the other backends

- Unclear why very little of the *Types API* was being utilized before - Most cases can use a constant - https://duckdb.org/docs/stable/clients/python/types

- Converts previous `AttributeError` (on issubclass) into and unknown dtype - See (#2654 (comment)) - avoid using `Any` - focus on getting the dtype first

All of this is temporary until #2486

see #2982

All other backends make an attempt to retrieve their time units

narwhals/_pandas_like/utils.py

Hoping this makes things more visible

narwhals/_pandas_like/utils.py

MarcoGorelli

thanks @dangotbanned , really nice

narwhals/_pandas_like/utils.py

dangotbanned · 2025-08-13T18:46:13Z

Thanks @MarcoGorelli 😍

also 😂 at (2967c2d)

- Full revert of #2731 - Not needed since #2969 Now, we only have 1 case which uses the 2x `@overload`(s) but I'm keeping the 3x as well for now 🙂

dangotbanned added 2 commits August 10, 2025 20:16

feat: Add DType.base_type

5ccf476

Incredibly simple, but will allow a big a refactor https://github.com/pola-rs/polars/blob/34595af53dbdcbde52ab062746a42aa725e321a4/py-polars/polars/datatypes/classes.py#L145-L159

refactor: Simplify _arrow.utils.narwhals_to_native_dtype

3070470

dangotbanned added enhancement New feature or request dtypes labels Aug 10, 2025

test: fix doctests

3a1a257

Either VSCode or `iPython` is showing me the wrong repr for classes 🤦‍♂️

dangotbanned commented Aug 11, 2025

View reviewed changes

dangotbanned added 3 commits August 11, 2025 12:36

Merge branch 'main' into dtype-base-type

b725314

refactor: Simplify _polars.utils.narwhals_to_native_dtype

970cb8a

refactor: Simplify _ibis.utils.narwhals_to_native_dtype

9e994f3

dangotbanned commented Aug 11, 2025

View reviewed changes

dangotbanned added 12 commits August 12, 2025 18:21

Merge branch 'main' into dtype-base-type

52dfc47

refactor: Simplify _duckdb.utils.narwhals_to_native_dtype

7f5b00e

- Unclear why very little of the *Types API* was being utilized before - Most cases can use a constant - https://duckdb.org/docs/stable/clients/python/types

Merge branch 'main' into dtype-base-type

ee3d38d

Merge branch 'main' into dtype-base-type

dacd1db

refactor: tweak duckdb array conversion

6924757

- Converts previous `AttributeError` (on issubclass) into and unknown dtype - See (#2654 (comment)) - avoid using `Any` - focus on getting the dtype first

refactor: Simplify _dask.utils.narwhals_to_native_dtype

0d77180

refactor: Align unsupported across backends

5240e49

All of this is temporary until #2486

Merge remote-tracking branch 'upstream/main' into dtype-base-type

6212cda

docs: remove returns section

654bd01

see #2982

fix: Include v1 dtypes for dask

17508b2

All other backends make an attempt to retrieve their time units

refactor: Simplify _spark_like.utils.narwhals_to_native_dtype

b66891c

refactor: Simplify _pandas_like.utils.narwhals_to_native_dtype

cf83c9d

dangotbanned commented Aug 13, 2025

View reviewed changes

narwhals/_pandas_like/utils.py Show resolved Hide resolved

dangotbanned marked this pull request as ready for review August 13, 2025 17:26

dangotbanned marked this pull request as draft August 13, 2025 17:29

🧹🧹🧹

2465761

Hoping this makes things more visible

dangotbanned marked this pull request as ready for review August 13, 2025 17:54

dangotbanned added the performance label Aug 13, 2025

MarcoGorelli reviewed Aug 13, 2025

View reviewed changes

narwhals/_pandas_like/utils.py Show resolved Hide resolved

MarcoGorelli approved these changes Aug 13, 2025

View reviewed changes

actually raise the exception

61d2011

MarcoGorelli reviewed Aug 13, 2025

View reviewed changes

narwhals/_pandas_like/utils.py Outdated Show resolved Hide resolved

MarcoGorelli added 2 commits August 13, 2025 19:32

Update narwhals/_pandas_like/utils.py

d9e6c65

fix syntax (i shouldnt have tried modifying from github ui)

2967c2d

dangotbanned merged commit 64472d2 into main Aug 13, 2025
32 of 33 checks passed

dangotbanned deleted the dtype-base-type branch August 13, 2025 18:47

dangotbanned mentioned this pull request Aug 14, 2025

warn or raise on explicit incompatible TimeUnit #2964

Closed

dangotbanned mentioned this pull request Sep 18, 2025

refactor(typing): Remove now-unused isinstance_or_issubclass overloads #3139

Merged

10 tasks

MarcoGorelli mentioned this pull request Oct 4, 2025

[Bug]: narhwals fails to create polars dataframe from_dict with None in schema #3176

Closed

This was referenced Oct 10, 2025

perf: Add __slots__ to all DTypes #3194

Merged

test: Add serde tests for DTypes #3205

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add `DType.base_type` #2969

feat: Add `DType.base_type` #2969

Uh oh!

dangotbanned commented Aug 10, 2025 •

edited

Loading

Uh oh!

dangotbanned Aug 11, 2025 •

edited

Loading

Uh oh!

dangotbanned Aug 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

MarcoGorelli left a comment

Uh oh!

Uh oh!

dangotbanned commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	def isinstance_or_issubclass(obj_or_cls: Any, cls_or_tuple: Any) -> bool:
	from narwhals.dtypes import DType

	if isinstance(obj_or_cls, DType):
	return isinstance(obj_or_cls, cls_or_tuple)
	return isinstance(obj_or_cls, cls_or_tuple) or (

feat: Add DType.base_type #2969

feat: Add DType.base_type #2969

Uh oh!

Conversation

dangotbanned commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

dangotbanned Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Example (String() | String)

Optimization

Uh oh!

Uh oh!

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dangotbanned commented Aug 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: Add `DType.base_type` #2969

feat: Add `DType.base_type` #2969

dangotbanned commented Aug 10, 2025 •

edited

Loading

dangotbanned Aug 11, 2025 •

edited

Loading

dangotbanned Aug 11, 2025 •

edited

Loading

Example (`String() | String`)