[Bug]: Version V1 got an unexpected keyword 'categories' when collecting duckdb enum dtype

[zfaee]

Describe the bug

Version V1 got an unexpected keyword 'categories' when collecting duckdb enum dtype.

The keyword 'categories' is only supported in Version MAIN.

Steps or code to reproduce the bug

import duckdb
import narwhals as nw
import narwhals.stable.v1 as nwv1

enumdf = duckdb.sql("select 'a'::enum('a', 'b', 'c') as a")

nw.from_native(enumdf).collect_schema() # version main is ok
nwv1.from_native(enumdf).collect_schema() # version v1 error

Expected results

No error

Actual results

File .venv\lib\site-packages\narwhals\_duckdb\utils.py:128, in native_to_narwhals_dtype(duckdb_dtype, version)
    126 if duckdb_dtype_id == "enum":
    127     categories = duckdb_dtype.children[0][1]
--> 128     return dtypes.Enum(categories=categories)
    130 return _non_nested_native_to_narwhals_dtype(duckdb_dtype_id, version)

TypeError: Enum.__init__() got an unexpected keyword argument 'categories'

Please run narwhals.show_version() and enter the output below.

System:
    python: 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:01:37) [MSC v.1935 64 bit (AMD64)]
executable: .venv\Scripts\python.exe
   machine: Windows-10-10.0.19045-SP0

Python dependencies:
     narwhals: 1.39.0
       pandas: 2.2.3
       polars: 1.29.0
         cudf: 
        modin: 
      pyarrow: 20.0.0
        numpy: 2.2.5

Relevant log output

[MarcoGorelli]

Thanks for the report will take a look

[dangotbanned]

It seems like on the reverse (narwhals -> native) side, we don't allow duckdb enums

narwhals/narwhals/_duckdb/utils.py

Lines 203 to 211 in 0264f98

	if isinstance_or_issubclass(dtype, dtypes.Enum):
	if version is Version.V1:
	msg = "Converting to Enum is not supported in narwhals.stable.v1"
	raise NotImplementedError(msg)
	if isinstance(dtype, dtypes.Enum):
	categories = "'" + "', '".join(dtype.categories) + "'"
	return f"ENUM ({categories})"
	msg = "Can not cast / initialize Enum without categories present"
	raise ValueError(msg)

But this issue is for (native -> narwhals) where we aren't doing the same check

narwhals/narwhals/_duckdb/utils.py

Lines 126 to 128 in 0264f98

	if duckdb_dtype_id == "enum":
	categories = duckdb_dtype.children[0][1]
	return dtypes.Enum(categories=categories)

@MarcoGorelli do you recall why duckdb only has support in main?

• feat: Add support for DuckDB Enum #2401

[camriddell]

Great digging @dangotbanned. I think the intention was as follows

• In V1 we allow users to create Enum types in their own native tools and ingest them into Narwhals (this is present in the polars support we have).
• In MAIN we allow users to create Enum types in Narwhals to allow for arbitrary casting (e.g. I have a string dtype in Narwhals and I want to cast to an Enum)
  • Aside: also in MAIN we support Enums in _pandas_like (which has no native concept of Enum, just Categorical which we disambiguate depending on whether or not the Categorical is ordered).

Therefore I believe the solution here would be for DuckDB to return the V1 Enum() like we do in Polars. Then in MAIN we support the storage of the inner categories (as is the current behavior).

Definitely want @MarcoGorelli to verify this though.

polars

native → narwhals

narwhals/narwhals/_polars/utils.py

Lines 102 to 110 in 0264f98

	if isinstance_or_issubclass(dtype, pl.Enum):
	if version is Version.V1:
	return dtypes.Enum() # type: ignore[call-arg]
	categories = _DeferredIterable(
	dtype.categories.to_list
	if backend_version >= (0, 20, 4)
	else lambda: cast("list[str]", dtype.categories)
	)
	return dtypes.Enum(categories)

narwhals → native

narwhals/narwhals/_polars/utils.py

Lines 184 to 191 in 0264f98

	if isinstance_or_issubclass(dtype, dtypes.Enum):
	if version is Version.V1:
	msg = "Converting to Enum is not supported in narwhals.stable.v1"
	raise NotImplementedError(msg)
	if isinstance(dtype, dtypes.Enum):
	return pl.Enum(dtype.categories)
	msg = "Can not cast / initialize Enum without categories present"
	raise ValueError(msg)

pandas_like

native → narwhals

narwhals/narwhals/_pandas_like/utils.py

Lines 257 to 258 in 0264f98

	if dtype == "category":
	return native_categorical_to_narwhals_dtype(native_dtype, version)

narwhals/narwhals/_pandas_like/utils.py

Lines 303 to 318 in 0264f98

	def native_categorical_to_narwhals_dtype(
	native_dtype: pd.CategoricalDtype,
	version: Version,
	implementation: Literal[Implementation.CUDF] | None = None,
	) -> DType:
	dtypes = version.dtypes
	if version is Version.V1:
	return dtypes.Categorical()
	if native_dtype.ordered:
	into_iter = (
	_cudf_categorical_to_list(native_dtype)
	if implementation is Implementation.CUDF
	else native_dtype.categories.to_list
	)
	return dtypes.Enum(_DeferredIterable(into_iter))
	return dtypes.Categorical()

narwhals → native

narwhals/narwhals/_pandas_like/utils.py

Lines 511 to 517 in 0264f98

	if isinstance_or_issubclass(dtype, dtypes.Enum):
	if version is Version.V1:
	msg = "Converting to Enum is not supported in narwhals.stable.v1"
	raise NotImplementedError(msg)
	if isinstance(dtype, dtypes.Enum):
	ns = implementation.to_native_namespace()
	return ns.CategoricalDtype(dtype.categories, ordered=True)

[MarcoGorelli]

Therefore I believe the solution here would be for DuckDB to return the V1 Enum() like we do in Polars. Then in MAIN we support the storage of the inner categories (as is the current behavior).

yup