[Bug]: `group_by` context ignores expr arguments

[FBruzzesi]

Describe the bug

For expressions that accept arguments, these are not passed along in group_by context (e.g. ddof in std).

Steps or code to reproduce the bug

import narwhals as nw
import pandas as pd
import polars as pl

data = {"a": [1, 1, 1, 1, 1, 2, 2, 2, 2, 2], "b": list(range(10))}

nw.from_native(pl.DataFrame(data)).group_by("a").agg(nw.col("b").std(ddof=2)).to_native()
shape: (2, 2)
┌─────┬──────────┐
│ a   ┆ b        │
│ --- ┆ ---      │
│ i64 ┆ f64      │
╞═════╪══════════╡
│ 2   ┆ 1.825742 │
│ 1   ┆ 1.825742 │
└─────┴──────────┘

nw.from_native(pd.DataFrame(data)).group_by("a").agg(nw.col("b").std(ddof=2)).to_native()
   a         b
0  1  1.581139
1  2  1.581139

# which is the same as ddof=1

nw.from_native(pd.DataFrame(data)).group_by("a").agg(nw.col("b").std(ddof=1)).to_native()
   a         b
0  1  1.581139
1  2  1.581139

Expected results

Arguments to be passed along or raise

Actual results

Arguments get ignored

Please run narwhals.show_version() and enter the output below.

System:
    python: 3.12.2 (main, Feb 25 2024, 16:35:05) [GCC 11.4.0]

Python dependencies:
     narwhals: 1.18.4
       pandas: 2.2.3
       polars: 1.17.1
         cudf: 
        modin: 0.32.0
      pyarrow: 18.1.0
        numpy: 2.1.3

Relevant log output

No response

[FBruzzesi]

Possibly related: if the same aggregations is passed with different aliases, pandas will raise:

(
nw.from_native(pd.DataFrame(data))
.group_by("a")
.agg(
    nw.col("b").std().alias("std_ddof_1"),
    nw.col("b").std(ddof=2).alias("std_ddof_2"),
).to_native()
)

Raises:

ColumnNotFoundError: The following columns were not found: ['std_ddof_1']

Hint: Did you mean one of these columns: ['a', 'std_ddof_2', 'std_ddof_2']?

[MarcoGorelli]

thanks for reporting / spotting, this isn't great - do you have a solution in mind, or shall i take a look?

[FBruzzesi]

If you have time go ahead. I have scattered slots to take a look or I can tonight 🙃

[FBruzzesi]

@MarcoGorelli after a bit of investigation, I was not able to find a way to pass args and kwargs to a function in the pandas group by notation we use for the fast path (namely output_name = (input_name, aggregation)).

The native option is native_grouped_frame.std(ddof=2), yet this would require a specific "path" similar to the n_unique one, which might lead to combinatorial explosion of the if-elif-else block?

Another alternative is to increase the depth of std/var at source if ddof!=1, treating it as a complex aggreation.

Disclaimer, I did not check pyarrow yet

[MarcoGorelli]

thanks for checking - i also had a look and came to a similar conclusion 😭

I think a third solution is to record the ddof in the function name. We only really have this issue for pandas unfortunately - i'll make a pr and show you

[FBruzzesi]

Following the comment in #1614

• pyarrow
• pandas
• dask: I opened a conversation/thread in their discourse as I am probably missing something from a custom aggregation
• pyspark: this should be fairly easy to achieve via expressions instead of get_spark_function