chore: Add Compliant*.from_native

[dangotbanned]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Part of API: io functions for v2 #2116

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

Following roughly the same pattern as from_numpy

• FromNative protocol
• CompliantSeries
• CompliantDataFrame
• CompliantNamespace (instance method)
  • EagerNamespace
  • PolarsNamespace
  • LazyNamespace

[dangotbanned]

Reference

Did some experimenting and arrived at this being the pattern I like the most:

Code block

FrameT = TypeVar("FrameT")
SeriesT = TypeVar("SeriesT")


class CompliantNamespace(Protocol[FrameT, SeriesT]):
    @property
    def _dataframe(self) -> type[FromNative[FrameT]]: ...
    @property
    def _series(self) -> type[FromNative[SeriesT]]: ...

    @overload
    def from_native(self, obj: FrameT) -> FromNative[FrameT]: ...

    @overload
    def from_native(self, obj: SeriesT) -> FromNative[SeriesT]: ...

    def from_native(
        self, obj: FrameT | SeriesT | Any
    ) -> FromNative[FrameT] | FromNative[SeriesT]:
        if self._dataframe._is_native(obj):
            return self._dataframe.from_native(obj)
        elif self._series._is_native(obj):
            return self._series.from_native(obj)
        msg = f"Unsupported type: {type(obj).__name__!r}"
        raise TypeError(msg)

Moving up a level, the narrowing looks like:

Screenshot

So rather than the current nw.from_native, which is roughly:

if is_polars_dataframe(native_object): ...
elif is_polars_lazyframe(native_object): ...
elif is_polars_series(native_object): ...
elif is_pandas_dataframe(native_object): ...
elif is_pandas_series(native_object): ...
elif is_modin_dataframe(native_object): ...
elif is_modin_series(native_object): ...
elif is_cudf_dataframe(native_object): ...
elif is_cudf_series(native_object): ...
# ... more

We'd aim to narrow to a Namespace first, then let that dispatch to the right from_native call 😎

if is_native_polars(native_object): ...
elif is_native_pandas_like(native_object): ...

Note

I had to partially implement that in translate.py to get PolarsNamespace.from_native coverage (9396bbc)
(#2324) takes that idea and runs with it

[dangotbanned]

Been thinking about combining these as one protocol.

_StoreNative might still be useful on its own, but something like this could be nice:

from typing import Protocol, TypeVar

from narwhals._translate import FromNative
from narwhals.utils import _StoresNative

NativeT = TypeVar("NativeT")

class Compliant(_StoresNative[NativeT], FromNative[NativeT], Protocol[NativeT]): ...

Not super important though 😅

[dangotbanned]

Throwaway Idea

We could get some more (from_|to_) methods from what is currently nested in (lazy|collect).

Collected some notes while trying to get coverage for CompliantLazyFrame

class DaskLazyFrame(CompliantLazyFrame["DaskExpr", "dd.DataFrame"]):...
# NOTE: Supports:
# Eager.lazy()
# - from_arrow (via Table.to_pandas, dd.from_pandas)
# - from_pandas
# DaskLazyFrame.collect()
# - to_arrow (via self.native.compute() -> pl.from_pandas())
# - to_pandas (via self.native.compute())
# - to_polars (via self.native.compute() ->  pa.Table.from_pandas())

class DuckDBLazyFrame(CompliantLazyFrame["DuckDBExpr", "duckdb.DuckDBPyRelation"]):...
# NOTE: Supports:
# Eager.lazy()
# - from_arrow
# - from_pandas
# DuckDBLazyFrame.collect()
# - to_arrow (via self.native.arrow())
# - to_pandas (via self.native.df())
# - to_polars (via self.native.pl())

class PolarsLazyFrame:
# NOTE: Supports:
# Eager.lazy()  # noqa: ERA001
# - from_arrow (via pl.from_arrow().lazy())
# - from_pandas (via pl.from_pandas().lazy())

class SparkLikeLazyFrame(CompliantLazyFrame["SparkLikeExpr", "SQLFrameDataFrame"]):
# NOTE: Supports:
# SparkLikeLazyFrame.collect()
# - to_arrow (via self._collect_to_arrow())
# - to_pandas (via self.native.toPandas())
# - to_polars (via self._collect_to_arrow() -> pl.from_arrow())

[dangotbanned]

Really want a better solution than this 😒
Seems like 1.3 is a way off, but the stubs are the only part we need

https://github.com/duckdb/duckdb/blob/6d19e746d302372560397a6f20151cafd817a231/tools/pythonpkg/duckdb-stubs/__init__.pyi#L272

• https://github.com/duckdb/duckdb-web/milestone/5

[MarcoGorelli]

the website says May

would it work for you to install the pre-release (pip install -U --pre duckdb)? that's what runs in CI

[dangotbanned]

@MarcoGorelli is there not a way to specify that in our pyproject.toml?

I'm usually pretty cautious about doing one-off installs, since I inevitably forget what I did the next time around 😄

[MarcoGorelli]

thanks @dangotbanned

[dangotbanned]

thanks @dangotbanned

Thanks for all the reviews @MarcoGorelli 😍