chore: remove more kwargs from dask, dont treat ddof as expressifiable arg#2059
chore: remove more kwargs from dask, dont treat ddof as expressifiable arg#2059MarcoGorelli merged 5 commits intonarwhals-dev:mainfrom
kwargs from dask, dont treat ddof as expressifiable arg#2059Conversation
d6e3677 to
6ed04d0
Compare
narwhals/_dask/expr.py
Outdated
| # First argument to `call` should be `dx.Series` | ||
| call: Callable[..., dx.Series], | ||
| expr_name: str, | ||
| kwargs: dict[str, Any] | None = None, |
There was a problem hiding this comment.
What if we call this call_kwargs i.e. kwargs for the call function ?
(Just to clarify these kwargs are not passed to _from_call itself)
There was a problem hiding this comment.
@EdAbati Couldn't you just pass call as a partial in the cases where this is needed?
- https://docs.python.org/3/library/functools.html#partial-objects
- https://docs.python.org/3/library/functools.html#functools.partial
- Avoids needing this parameter entirely
- "binding" of arguments is done once, instead of on each use
- Also supports positional arguments
So instead of call, call_kwargs - you'd have call.func, call.keywords.
But the keywords themselves aren't exposed here
There was a problem hiding this comment.
I did not know that functools.partial stored keywords like that, thanks!
There was a problem hiding this comment.
I did not know that
functools.partialstored keywords like that, thanks!
You might need to use operator.methodcaller instead, if you don't know the class that would be passed.
E.g. partial would work if you could do partial(dx.Series.std, ddof=ddof).
But where it might be dx._groupby.GroupBy.std, using the method name would be more widely accepted
narwhals/narwhals/_dask/group_by.py
Lines 54 to 55 in 4dda548
There was a problem hiding this comment.
if you don't know the class that would be passed.
Is that why you're storing function_name, in addition to the lambda?
| def std(self: Self, ddof: int) -> Self: | ||
| return self._from_call( | ||
| lambda _input, ddof: _input.std(ddof=ddof).to_series(), "std", ddof=ddof | ||
| lambda _input: _input.std(ddof=ddof).to_series(), | ||
| "std", | ||
| kwargs={"ddof": ddof}, | ||
| ) |
There was a problem hiding this comment.
Related to #2059 (comment)
and
# Kwargs with metadata which we may need in group-by agg
# (e.g. `ddof` for `std` and `var`).
kwargs: dict[str, Any] | None = None,I'm not following why kwargs is used at all here:
narwhals/narwhals/_dask/group_by.py
Lines 54 to 55 in 4dda548
narwhals/narwhals/_dask/group_by.py
Lines 148 to 150 in 4dda548
Am I reading this wrong, or are the same kwargs applied twice?
|
thanks both for comments! i'd like to refactor this mechanism more broadly, so i'll hold off |
dangotbanned
left a comment
There was a problem hiding this comment.
Sounds good @MarcoGorelli
|
thanks all! |
Now is always empty #2059
I don't like how currently we're muddying the waters between "expressifiable arguments" and "kwargs", which leads us to having to pass
ddofas an expressifiable argument instdandvarI'd like to start by separating the concepts, and not passing
ddofas an expressifiable argumentWhat type of PR is this? (check all applicable)
Related issues
Checklist
If you have comments or can explain your changes, please do so below