feat: Allow nested structures in `lit` by FBruzzesi · Pull Request #3424 · narwhals-dev/narwhals

FBruzzesi · 2026-01-27T23:17:47Z

Description

I thought this was going to be more complex for more backends. The only troubled one is, as you might expect, pandas. That said, I am quite ok for how it turned out

Important

It's totally possible that there is a simpler way to achieve it
It uses pd.arrays.ArrowExtensionArray which is marked as experimental

TODO:

probably some typing for pandas case. In my defence, each time I try to add anything to _BasePandasLikeSeries or _BasePandasLikeFrame, a test case fails for modin. I had the same issue in feat: Add concat(..., how="*_relaxed"}) #3398. Resolved in 8cf634b
Deeper nesting? explicitly raise NotImplementedError for multi-nested values
Check CuDF -> ~~I am no longer able to run it in kaggle due to dependency ~~hell~~ issues 😢~~ Check discord thread

What type of PR is this? (check all applicable)

Related issues

Closes Support nested structure in nw.lit #3408

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

FBruzzesi · 2026-01-28T15:59:46Z

narwhals/_pandas_like/utils.py

+@no_type_check
+def broadcast_series_to_index(


@dangotbanned I hope you can forgive me one day!

As commented in the description, I added all the relevant methods and properties, yet, as in #3398, the implementation_test.py fails for Modin

Combining these guys should work 😄

feat: Allow nested structures in lit #3424 (comment)

feat: Allow nested structures in lit #3424 (comment)

feat: Allow nested structures in lit #3424 (comment)

It uses pd.arrays.ArrowExtensionArray which is marked as experimental

I wouldn't worry about pandas warning about experimental.

They've had that since introducing pyarrow stuff in (1.5.*), which was over 3 years ago

Amazing! Thanks a ton @dangotbanned - applied all the suggestions in 8cf634b

I left one TODO regarding whether or not we should pass copy=False to pd.array

narwhals/_pandas_like/utils.py

MarcoGorelli

seems reasonable, thanks @FBruzzesi !

dangotbanned · 2026-01-30T19:08:57Z

narwhals/_pandas_like/namespace.py

+                # Use ArrowExtensionArray to avoid pandas unpacking the nested structure
+                ns = self._implementation.to_native_namespace()
+                pandas_series_native = ns.Series(
+                    pd.arrays.ArrowExtensionArray(pa_array),  # type: ignore[attr-defined]
+                    name="literal",
+                    index=df._native_frame.index[0:1],
+                )


I think the wrapping part of (#3424 (comment)) should be reused here.

Maybe broadcast_series_to_index is too specfic of a function?

The two more useful parts IMO are:

repeat

something related to reconstruction?

(#3424 (comment))

Okay yeah so what I'm thinking is a new constructor (or two?) on PandasLikeSeries might work?

All of these are parts of a constructor for both native & compliant, and they're spread across 4 modules.

narwhals/narwhals/_pandas_like/dataframe.py

Line 308 in dd5c623

def _extract_comparand(self, other: PandasLikeSeries) -> pd.Series[Any]:

narwhals/narwhals/_pandas_like/namespace.py

Lines 86 to 87 in dd5c623

def lit(self, value: PythonLiteral, dtype: IntoDType | None) -> PandasLikeExpr:

def _lit_pandas_series(df: PandasLikeDataFrame) -> PandasLikeSeries:

narwhals/narwhals/_pandas_like/series.py

Lines 204 to 206 in dd5c623

@classmethod

def _align_full_broadcast(cls, *series: Self) -> Sequence[Self]:

Series = series[0].__native_namespace__().Series

narwhals/narwhals/_pandas_like/utils.py

Line 668 in dd5c623

def broadcast_series_to_index(

dangotbanned · 2026-01-30T19:12:12Z

tests/utils.py

+def xfail_if_pyspark_connect(  # pragma: no cover
+    constructor: Constructor, request: pytest.FixtureRequest, reason: str = ""
+) -> None:
+    if is_pyspark_connect(constructor):
+        request.applymarker(pytest.mark.xfail(reason=reason))


Suggested change

def xfail_if_pyspark_connect( # pragma: no cover

constructor: Constructor, request: pytest.FixtureRequest, reason: str = ""

) -> None:

if is_pyspark_connect(constructor):

request.applymarker(pytest.mark.xfail(reason=reason))

request.applymarker(pytest.mark.xfail(is_pyspark_connect(constructor), reason=reason))

dangotbanned · 2026-01-30T19:25:05Z

Note

~~Marking~~ Marked as draft while I ~~fix~~ fixed the typing (#3424 (comment))
(69abb91)

- Fixed typo - Skip using `qualified_type_name` when we know it would be `builtins` (and get stripped anyway)

dangotbanned

Thanks @FBruzzesi!

Only a few suggestions from me, nice work 😎

dangotbanned · 2026-01-30T20:28:34Z

narwhals/_arrow/namespace.py

        )

-    def lit(self, value: NonNestedLiteral, dtype: IntoDType | None) -> ArrowExpr:
+    def lit(self, value: PythonLiteral, dtype: IntoDType | None) -> ArrowExpr:


Well, isn't this a nice diff 😄

narwhals/_duckdb/namespace.py

dangotbanned · 2026-01-30T20:31:27Z

narwhals/_ibis/namespace.py

        return self._expr._from_elementwise_horizontal_op(func, *exprs)

-    def lit(self, value: Any, dtype: IntoDType | None) -> IbisExpr:
+    def lit(self, value: PythonLiteral, dtype: IntoDType | None) -> IbisExpr:


Oh nice, no more Any too!

narwhals/_ibis/namespace.py

dangotbanned · 2026-01-30T20:47:02Z

narwhals/_pandas_like/namespace.py

+                # Use ArrowExtensionArray to avoid pandas unpacking the nested structure
+                ns = self._implementation.to_native_namespace()
+                pandas_series_native = ns.Series(
+                    pd.arrays.ArrowExtensionArray(pa_array),  # type: ignore[attr-defined]
+                    name="literal",
+                    index=df._native_frame.index[0:1],
+                )


(#3424 (comment))

Okay yeah so what I'm thinking is a new constructor (or two?) on PandasLikeSeries might work?

All of these are parts of a constructor for both native & compliant, and they're spread across 4 modules.

narwhals/narwhals/_pandas_like/dataframe.py

Line 308 in dd5c623

def _extract_comparand(self, other: PandasLikeSeries) -> pd.Series[Any]:

narwhals/narwhals/_pandas_like/namespace.py

Lines 86 to 87 in dd5c623

def lit(self, value: PythonLiteral, dtype: IntoDType | None) -> PandasLikeExpr:

def _lit_pandas_series(df: PandasLikeDataFrame) -> PandasLikeSeries:

narwhals/narwhals/_pandas_like/series.py

Lines 204 to 206 in dd5c623

@classmethod

def _align_full_broadcast(cls, *series: Self) -> Sequence[Self]:

Series = series[0].__native_namespace__().Series

narwhals/narwhals/_pandas_like/utils.py

Line 668 in dd5c623

def broadcast_series_to_index(

tests/expr_and_series/lit_test.py

FBruzzesi · 2026-01-31T10:25:30Z

Thanks both for your reviews! I am going to merge so that it can make it into Monday's release (yes, it's time to make a new release!!!)

@dangotbanned regarding #3424 (comment), we can follow up with a refactor. I am not 100% sure of what you have in mind 😂 so feel free to open a PR for it

FBruzzesi added 5 commits January 27, 2026 15:55

WIP: all but pandas and ibis?

ff8e647

add ibis to the mix, xfail pandas-like

db18590

fix doctest, test select

c49ccbb

pandas support!

a897470

factor out broadcasting, ignore typing, try to skip old pandas cases

3697d12

FBruzzesi added the enhancement New feature or request label Jan 27, 2026

FBruzzesi added 3 commits January 28, 2026 00:23

port docstrings

d66d4e6

skip polars pre 1.0

e3cee73

pragma no cover, xfail pyspark connect for empty dict

9b615c5

FBruzzesi commented Jan 28, 2026

View reviewed changes

FBruzzesi added the nested data `list`, `struct`, etc label Jan 29, 2026

Merge branch 'main' into feat/allow-lit-nested

fc8ef14

FBruzzesi marked this pull request as ready for review January 29, 2026 19:08

dangotbanned reviewed Jan 30, 2026

View reviewed changes

narwhals/_pandas_like/utils.py Outdated Show resolved Hide resolved

dangotbanned reviewed Jan 30, 2026

View reviewed changes

narwhals/_pandas_like/utils.py Outdated Show resolved Hide resolved

dangotbanned reviewed Jan 30, 2026

View reviewed changes

narwhals/_pandas_like/utils.py Outdated Show resolved Hide resolved

FBruzzesi added 2 commits January 30, 2026 10:19

apply Dan's suggestions

8cf634b

explicitly raise NotImplementedError for multi-nested values

1eb8451

MarcoGorelli approved these changes Jan 30, 2026

View reviewed changes

dangotbanned reviewed Jan 30, 2026

View reviewed changes

dangotbanned self-requested a review January 30, 2026 19:14

dangotbanned marked this pull request as draft January 30, 2026 19:24

dangotbanned added 2 commits January 30, 2026 20:18

fix(typing): Add is_nested_literal, resolve functions.lit

69abb91

refactor: Reuse is_nested_literal + dask drive-by

dd5c623

- Fixed typo - Skip using `qualified_type_name` when we know it would be `builtins` (and get stripped anyway)

dangotbanned marked this pull request as ready for review January 30, 2026 20:25

dangotbanned approved these changes Jan 30, 2026

View reviewed changes

FBruzzesi added 2 commits January 30, 2026 22:08

merge main

04825bf

apply Dan's suggestions

9f65f3a

FBruzzesi merged commit b9b04d0 into main Jan 31, 2026
36 of 37 checks passed

FBruzzesi deleted the feat/allow-lit-nested branch January 31, 2026 10:25

dangotbanned mentioned this pull request Jan 31, 2026

ci: Unpin polars==1.34.0 in --group typing #3434

Merged

	def lit(self, value: PythonLiteral, dtype: IntoDType \| None) -> PandasLikeExpr:
	def _lit_pandas_series(df: PandasLikeDataFrame) -> PandasLikeSeries:

	@classmethod
	def _align_full_broadcast(cls, *series: Self) -> Sequence[Self]:
	Series = series[0].__native_namespace__().Series

Conversation

FBruzzesi commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What type of PR is this? (check all applicable)

Related issues

Checklist

Uh oh!

FBruzzesi Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FBruzzesi Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

dangotbanned commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dangotbanned left a comment

Choose a reason for hiding this comment

Uh oh!

dangotbanned Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dangotbanned Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dangotbanned Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FBruzzesi commented Jan 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FBruzzesi commented Jan 27, 2026 •

edited

Loading

dangotbanned Jan 30, 2026 •

edited

Loading

dangotbanned Jan 30, 2026 •

edited

Loading

dangotbanned commented Jan 30, 2026 •

edited

Loading

dangotbanned Jan 30, 2026 •

edited

Loading