feat: add rank for Lazy backends

[raisadz]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Related issue #<issue number>
• Closes implement remaining order-dependent lazy operations #2174

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

Only min and dense methods are implemented for DuckDB and PySpark without .over().

[FBruzzesi]

Hey @raisadz thanks for the PR. I didn't have the time to look at this, I just wanted to drop a comment and ping @marvinl803 who also took a look at rank and maybe can share some insights for the remaining methods 🙌🏼

[MarcoGorelli]

thanks for working on this 🙌 !

it's occurred to me that we class rank as ExprKind.WINDOW and that, after narwhals.stable.v1, it won't be supported to use rank if not followed by over(order_by=...)

Here it works because the import is import narwhals.stable.v1 as nw, but would raise for import narwhals as nw

🤔 gonna have to think about a solution here, not super-sure

maybe, just maybe, we may want to introduce nw.rank instead of Expr.rank, because then that would compose more naturally with over. And for eager backends, nw.col('a').rank() does the same thing as nw.rank().over('a')

🤔 i'm not totally sure, will do a bit more thinking on this one, but i think we can come up with something

[MarcoGorelli]

Having slept on this, I'd suggest:

nw.rank().over(order_by='a')  # equivalent to pl.col('a').rank()

An alternative could be nw.rank('a', partition_by='b'), but then what happens if a user writes nw.rank('a', partition_by='b').over('b')? I think in general, any expression that uses over internally is problematic

The only solution I can currently think of is: nw.rank().over(order_by='a'), and deprecate Expr.rank (but keep it around in stable.v1 - this is especially important here because, IIRC, we already have downstream libraries using nw.Expr.rank)

[dangotbanned]

#2310 (comment)

@MarcoGorelli I haven't looked into this more than your two comments, so apologies in advance if this seems too simplified of a view.

IMO, the current narwhals internal implementation of the polars API shouldn't lead the decision on the narwhals public API.
If that is the case here, the general concept (if you're unfamiliar) is called a leaky abstraction.

---

I'm not sure how likely this would be, but another concern might be - what if polars adds a pl.rank in the future that has different behaviour?
The list of sugar functions is still growing

[MarcoGorelli]

agree, i'm also not keen on implementation contraints determining the api, ideally it should be the opposite...

will think about this more, but maybe we could get to:

✅ allowed:

• nw.col('a').rank()
• nw.col('a').rank().over('b')

❌ disallowed:

• nw.col('a').rank().over('b', order_by='c')

Then there's also:

• nw.col('a').rank().abs().over('b', order_by='c')

that requires a decision

This involves more refactors, but...it's probably worth it

[dangotbanned]

#2310 (comment)

@MarcoGorelli that seems consistent with the pl.Expr.rank docs 👍

I don't know if this already plays a factor in the narwhals API.
Do you think about aiming to cover sequences of expressions that are demonstrated in the polars docs?
That seems like a good compass to me - even if we can't always get there

[MarcoGorelli]

Yeah i think it's good to aim for as much expressivity as Polars offers, with the restriction that for LazyFrame we only allow relational expressions - so, lf.select(nw.col('a').drop_nulls(), nw.col('b').drop_nulls()) will never be allowed, as it may result in tuple which were never part of the original relation.
The rest should in theory just be limited by how clear our mental model and implemenation is

[MarcoGorelli]

thanks @raisadz - I think we can ship this, we'll sort out .over behaviour later, just left a minor comment

[MarcoGorelli]

can we flip this round (case when not-null), like you did for the spark-like case?

[MarcoGorelli]

i'll address #2310 (comment) now, gonna start by getting this in though