[Enh]: Add `DType.to_<backend>()`

[dangotbanned]

Tracking

• Add IntoDType alias, replace DType | type[DType] #2626
• [Enh]: Add DType.{from,to}_python #2264
• feat: Add Schema.from_{native,<backend>} #2957

Description

Originally posted by @dangotbanned in #2460 (comment)

For all of the narwhals_to_native_dtype functions, I'd recommend the same solution.

Important

It'd be nice to also do this for native_to_narwhals, but the caching may be a challenge

It is a bit much (to implement) for (#2460), but hopefully it gives an example of where these kinds of changes can lead 🙂

General

We could instead have methods on each DType class like:

to_arrow
to_dask
to_duckdb
to_polars
# NOTE: Could be more granular for these
to_pandas_like
to_spark_like

• Each would be much shorter (the goal of the rules) and very easy to digest.
• They'd be more performant as we'd skip all of the isinstance checks
  • By just calling the same method for every single case
• We could also make these public very easily
  • I imagine they'd be helpful in the same way as ([Enh]: nw.(DType|Schema) conversion API #1912) 😉
• Unsupported conversions are handled by the abscence of methods

Implementation

We would want to borrow/adapt the classinstmethod descriptor from polars.
They use that for to_python which is the same kind of operation - and is something we might want in the future anyway (#2264) 😏

V1

Another big benefit of this is that we'd move the MAIN vs V1 logic into the V1 data types.
I think this would make it really clear for us what the behavior of Enum should be:

narwhals/narwhals/_polars/utils.py

Lines 184 to 191 in 162dd5f

	if isinstance_or_issubclass(dtype, dtypes.Enum):
	if version is Version.V1:
	msg = "Converting to Enum is not supported in narwhals.stable.v1"
	raise NotImplementedError(msg)
	if isinstance(dtype, dtypes.Enum):
	return pl.Enum(dtype.categories)
	msg = "Can not cast / initialize Enum without categories present"
	raise ValueError(msg)

• [Bug]: Version V1 got an unexpected keyword 'categories' when collecting duckdb enum dtype #2550

[dangotbanned]

If anyone takes a stab at this feel free to ping me 🙂

@classinstmethod and pl.DataType.to_python are similar to what we'd need.
However, there is more work here than simply lifting them from polars 😅

Some improvements I'm thinking about:

If we're using a descriptor decorator, the typing for it needs to be fixed:

• (error 1)
• (error 2)
• (error 3)
• (error 4)

---

Directly translating the implementation it might be tempting to do:

class DType:
    @classinstmethod
    def to_polars(self):
        from narwhals._polars.utils import narwhals_to_native_dtype

        return narwhals_to_native_dtype(self, ..., ...)

Warning

This doesn't solve the complexity problem

We could instead have methods on each DType class like
Each would be much shorter (the goal of the rules) and very easy to digest

So without any magic - which might be needed to handle type[DType] - I'd think something like this would make sense:

from __future__ import annotations

from typing import TYPE_CHECKING

from narwhals.dtypes import FloatType
from narwhals.dtypes import UnsignedIntegerType

if TYPE_CHECKING:
    import polars as pl


class UInt8(UnsignedIntegerType):
    def to_polars(self) -> pl.UInt8:
        import polars as pl

        return pl.UInt8()


class Float64(FloatType):
    def to_polars(self) -> pl.Float64:
        import polars as pl

        return pl.Float64()

[FBruzzesi]

Hey @dangotbanned thanks for putting this together - it is really great!

This doesn't solve the complexity problem

In principle, these are two steps or requests:

1. Refactor: simplify narwhals_to_native_dtype
2. Enhancement: introduce DType.to_<backend>

The main concern I have in using (the future refactored) narwhals_to_native_dtype, would be regarding return types, namely:

class UInt8(UnsignedIntegerType):
    def to_polars(self) -> pl.UInt8:
        ...

would not be possible, but it would be:

class UInt8(UnsignedIntegerType):
    def to_polars(self) -> pl.DataType:
        ...

unless... there is another way for such typing?

[dangotbanned]

👋 @FBruzzesi

In principle, these are two steps or requests:

1. Refactor: simplify `narwhals_to_native_dtype`

2. Enhancement: introduce `DType.to_<backend>`

So when I said

We could instead have methods on each DType class like:

I meant removing all of the narwhals_to_native_dtype functions and just calling .to_<backend> directly instead.

So I see the two as quite linked, since we simplify via the replacement - if that makes sense?

unless... there is another way for such typing?

Yeah that is the issue I was trying to illustrate in (#2486 (comment)).
It's not unsolvable, but would be the first hurdle you'd run into if trying to solve via lifting the polars implementation

[FBruzzesi]

I meant removing all of the narwhals_to_native_dtype functions and just calling .to_ directly instead.

So I see the two as quite linked, since we simplify via the replacement - if that makes sense?

Yes that was clear, it is certainly a way to address it! Sorry for creating confusion around it!

If I could add another challenge to the mix related to:

However, there is more work here than simply lifting them from polars 😅

is that we might actually need to decide if the method is a classmethod or just a method.

Consider the following case:

class Datetime(...):
    time_unit: TimeUnit
    time_zone: str | None

Now in principle Datetime.to_polars() should return pl.Datetime and Datetime(time_unit, time_zone).to_polars() should return pl.Datetime(time_unit, time_zone).

At least to me it's not clear if or how something similar polars classinstmethod can help with that considering that each class/type will have different arguments to forward, some of which are scalar values (such as time_unit in the example above), while in other cases, they are other narwhals types requiring to be converted to polars (basically all the nested types).

[dangotbanned]

Thanks for looking into this @FBruzzesi!
I had a similar thought but after reading (#2486 (comment)) I've gone back over the polars implementation and learned something new.

pl.DataType.to_python

import polars as pl

>>> pl.Datetime("ns").to_python() is pl.Datetime.to_python() is pl.Datetime("us").to_python()
True

>>> pl.Datetime("ms").to_python()
datetime.datetime

I wasn't expecting this, but it makes sense - python types aren't parametric like "dtypes".

Following up how exactly the polars implementation works is also interesting:

• DataType.to_python calls
• dtype_to_py_type which then calls
• DataType.base_type
  • which is also kinda defined as DataTypeClass.base_type
• DataTypeMappings.DTYPE_TO_PY_TYPE then provides the actual conversion

It isn't clear to me why they've used @classinstmethod for this, since the behavior seems identical to @classmethod 😅

Challenge 2

If I could add another challenge to the mix related to:

However, there is more work here than simply lifting them from polars 😅

is that we might actually need to decide if the method is a classmethod or just a method.

Consider the following case:

class Datetime(...):
    time_unit: TimeUnit
    time_zone: str | None

Now in principle Datetime.to_polars() should return pl.Datetime and Datetime(time_unit, time_zone).to_polars() should return pl.Datetime(time_unit, time_zone).
@FBruzzesi

Yeah this is another interesting part.
Currently we always return an instantiated DType/native DataType in all of the conversion functions.

polars has some variations, depending on what the conversion is for.

I think we should aim for just continuing with the current behavior, so if we define to_polars as:

to_polars

from functools import partial

import narwhals as nw
from narwhals._polars import utils as pl_utils
from narwhals.utils import Version


to_polars = partial(
    pl_utils.narwhals_to_native_dtype, version=Version.MAIN, backend_version=(1, 25)
)

Then the following are valid

(obviously there's many more)

happy_dtypes = (
    nw.Datetime,
    nw.Datetime("ms"),
    nw.Time,
    nw.Time(),
    nw.Enum(["a", "b", "c"]),
    nw.Int64,
    nw.List(nw.Float32),
)

>>> [to_polars(dtype) for dtype in happy_dtypes]
[Datetime(time_unit='us', time_zone=None),
 Datetime(time_unit='ms', time_zone=None),
 Time,
 Time,
 Enum(categories=['a', 'b', 'c']),
 Int64,
 List(Float32)]

But these are invalid

sad_dtypes = (
    nw.Enum,
    nw.List,
    nw.Struct,
    nw.Array,
    nw.Decimal,
    nw.Decimal(),
    nw.Field("hi", nw.String()),
    nw.Field,
)

for dtype in sad_dtypes:
    try:
        converted = to_polars(dtype)
        print(f"Converted: {dtype!r} -> {converted!r}")
    except (ValueError, NotImplementedError, AttributeError) as err:
        print(f"{err!r}")

ValueError('Can not cast / initialize Enum without categories present')
AttributeError("type object 'List' has no attribute 'inner'")
AttributeError("type object 'Struct' has no attribute 'fields'")
AttributeError("type object 'Array' has no attribute 'size'")
NotImplementedError('Casting to Decimal is not supported yet.')
NotImplementedError('Casting to Decimal is not supported yet.')
AttributeError("type object 'Float64' has no attribute 'name'")
Converted: <class 'narwhals.dtypes.Field'> -> Unknown

I think I might have revealed a bug or two for those 😄

This comment is already getting long, so I'll try to follow up with how we can define different behavior on the class vs instance later.

Hint: we have an example of it already in here

narwhals/narwhals/utils.py

Lines 1761 to 1822 in 96f46f6

	class not_implemented: # noqa: N801
	"""Mark some functionality as unsupported.
	Arguments:
	alias: optional name used instead of the data model hook [`__set_name__`].
	Returns:
	An exception-raising [descriptor].
	Notes:
	- Attribute/method name *doesn't* need to be declared twice
	- Allows different behavior when looked up on the class vs instance
	- Allows us to use `isinstance(...)` instead of monkeypatching an attribute to the function
	Examples:
	>>> from narwhals.utils import not_implemented
	>>> class Thing:
	... def totally_ready(self) -> str:
	... return "I'm ready!"
	...
	... not_ready_yet = not_implemented()
	>>>
	>>> thing = Thing()
	>>> thing.totally_ready()
	"I'm ready!"
	>>> thing.not_ready_yet()
	Traceback (most recent call last):
	...
	NotImplementedError: 'not_ready_yet' is not implemented for: 'Thing'.
	...
	>>> isinstance(Thing.not_ready_yet, not_implemented)
	True
	[`__set_name__`]: https://docs.python.org/3/reference/datamodel.html#object.__set_name__
	[descriptor]: https://docs.python.org/3/howto/descriptor.html
	"""
	def __init__(self, alias: str | None = None, /) -> None:
	# NOTE: Don't like this
	# Trying to workaround `mypy` requiring `@property` everywhere
	self._alias: str | None = alias
	def __repr__(self) -> str:
	return f"<{type(self).__name__}>: {self._name_owner}.{self._name}"
	def __set_name__(self, owner: type[_T], name: str) -> None:
	# https://docs.python.org/3/howto/descriptor.html#customized-names
	self._name_owner: str = owner.__name__
	self._name: str = self._alias or name
	def __get__(
	self, instance: _T | Literal["raise"] | None, owner: type[_T] | None = None, /
	) -> Any:
	if instance is None:
	# NOTE: Branch for `cls._name`
	# We can check that to see if an instance of `type(self)` for
	# https://narwhals-dev.github.io/narwhals/api-completeness/expr/
	return self
	# NOTE: Prefer not exposing the actual class we're defining in
	# `_implementation` may not be available everywhere
	who = getattr(instance, "_implementation", self._name_owner)
	raise _not_implemented_error(self._name, who)

[dangotbanned]

is that we might actually need to decide if the method is a classmethod or just a method.

@FBruzzesi I've thought about this some more and think I have two relatively simple options we could choose from:

1. Just use instance methods for now and potentially add a more fancy version in the future
2. Define to_<backend> as a @classmethod on DTypes that take no arguments, and as an instance method on the others

Either option would be easy to relax any restrictions later if we see a chance to 🙂

[dangotbanned]

Spotted another internal use-case for this while reviewing (#2757)

narwhals/narwhals/_pandas_like/series.py

Lines 1075 to 1089 in cd60c42

	if dtype_backend == "pyarrow":
	import pyarrow.compute as pc
	from narwhals._arrow.utils import native_to_narwhals_dtype
	ca = native.array._pa_array
	result_arr = cast("ChunkedArrayAny", pc.logb(ca, base))
	nw_dtype = native_to_narwhals_dtype(result_arr.type, self._version)
	out_dtype = narwhals_to_native_dtype(
	nw_dtype,
	"pyarrow",
	self._implementation,
	self._backend_version,
	self._version,
	)

Would be much nicer to at least have a .to_arrow in here