feat: ArrowDataFrame.explode

[FBruzzesi]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

I will leave this as draft until we decide how to move forward.

To summarize the discussion(s) in #1542 :

• pyarrow native methods ignore nulls and empty list in explode
• the workaround here is to have a "fast_path" for when nulls or empty lists are not present, and a dedicated path for when they are
• the issue is that we enter python world to create the index via one .to_pylist() call
• pandas seems to enter cython anyway to explode a list

[dangotbanned]

@FBruzzesi I feel like this shouldn't have got lost!

ArrowDataFrame.explode is 1 of 3 remaining implementations we need

narwhals/narwhals/_arrow/dataframe.py

Line 350 in 2bcc6bb

explode = not_implemented()

narwhals/narwhals/_arrow/dataframe.py

Line 466 in 2bcc6bb

join_asof = not_implemented()

I might add a PR for ArrowDataFrame.clone - since it can just utilize arrow data being immutable

narwhals/narwhals/_arrow/dataframe.py

Line 669 in 2bcc6bb

clone = not_implemented()

[FBruzzesi]

I feel like this shouldn't have got lost!

Thanks @dangotbanned ♥️ The main concern was a conversion to python object: filled_counts.to_pylist() in:

    if fast_path:
        indices = pc.list_parent_indices(native_frame[to_explode[0]])
        flatten_func = pc.list_flatten

    else:
        filled_counts = pc.max_element_wise(counts, 1, skip_nulls=True)
        indices = pa.array(
            [
                i
                for i, count in enumerate(filled_counts.to_pylist())
                for _ in range(count)
            ]
        )

[dangotbanned]

#1644 (comment)

Maybe we can figure out another path hidden somewhere in the stubs? 🤔

[dangotbanned]

• Series[list].explode() should not return None for empty lists pola-rs/polars#17664

@FBruzzesi @MarcoGorelli

It seems like polars wants to make a breaking change in the next major version - resulting in the same behavior as pyarrow.

If we had that behavior as the goal - I think pc.list_flatten(..., recursive=True) would get us most of the way there.
Just something to keep in mind for the future 🙂

[dangotbanned]

@FBruzzesi I'm coming back to this after my recent fiddling with pyarrow 😄
There may still be a way to match polars - while staying in pyarrow-land - but I'm only just starting to look now 🤞

Updated: Yep, it works! 🥳 (#3347)

---

Regarding: (#1644 (comment))

3 weeks ago on that issue (pola-rs/polars#17664 (comment))

Note that the option got added to resolve this in #25289.

• feat: Add empty_as_null and keep_nulls flags to Expr.explode pola-rs/polars#25289

So now polars.DataFrame.explode has this signature:

    def explode(
        self,
        columns: ColumnNameOrSelector | Iterable[ColumnNameOrSelector],
        *more_columns: ColumnNameOrSelector,
        empty_as_null: bool = True,
        keep_nulls: bool = True,
    ) -> DataFrame: