-
Notifications
You must be signed in to change notification settings - Fork 210
feat: Added struct namespace with field method.
#2146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 37 commits
0d18f96
496cbd0
9592277
aaebbbe
273f331
2d95d3d
decba02
f89f7d3
e75da43
ef8b863
fa802a4
5e04a46
47345f5
9517b91
e8623e2
2d4ce6b
825d29e
6f867ef
e54663c
8fe86ae
37dc7fc
104d40d
9ead00a
9e2a24d
20ec4eb
ecfd180
6c5597b
99a78e3
105d3f9
d21bdc2
9db179a
865bedd
3ad7ae9
7e5d813
c5cf993
4d36295
8701e43
4ed080c
195088c
71df89d
ebe4d6b
59fb567
736599e
f71e677
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # `narwhals.Expr.struct` | ||
|
|
||
| ::: narwhals.expr.ExprStructNamespace | ||
| handler: python | ||
| options: | ||
| members: | ||
| - field | ||
| show_source: false | ||
| show_bases: false |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # `narwhals.Series.struct` | ||
|
|
||
| ::: narwhals.series.SeriesStructNamespace | ||
| handler: python | ||
| options: | ||
| members: | ||
| - field | ||
| show_source: false | ||
| show_bases: false |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| from narwhals._expression_parsing import reuse_series_namespace_implementation | ||
|
|
||
| if TYPE_CHECKING: | ||
| from typing_extensions import Self | ||
|
|
||
| from narwhals._arrow.expr import ArrowExpr | ||
|
|
||
|
|
||
| class ArrowExprStructNamespace: | ||
| def __init__(self: Self, expr: ArrowExpr) -> None: | ||
| self._compliant_expr = expr | ||
|
|
||
| def field(self: Self, name: str) -> ArrowExpr: | ||
| return reuse_series_namespace_implementation( | ||
| self._compliant_expr, | ||
| "struct", | ||
| "field", | ||
| name=name, | ||
| evaluate_output_names=lambda _col: [name], | ||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| import pyarrow.compute as pc | ||
|
|
||
| if TYPE_CHECKING: | ||
| from typing_extensions import Self | ||
|
|
||
| from narwhals._arrow.series import ArrowSeries | ||
|
|
||
|
|
||
| class ArrowSeriesStructNamespace: | ||
| def __init__(self: Self, series: ArrowSeries) -> None: | ||
| self._compliant_series: ArrowSeries = series | ||
|
|
||
| def field(self: Self, name: str) -> ArrowSeries: | ||
| self._compliant_series._name = name | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. you can't mutate
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the They weren't available when you started the PR @osoucy
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Some more context #2130 (comment)
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is fine as a follow-up
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. At least the current implementation avoids mutating the compliant series. Maybe I can look into @dangotbanned in a future PR?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All good @osoucy π
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So we are good to merge? :) |
||
| return self._compliant_series._from_native_series( | ||
| pc.struct_field(self._compliant_series._native_series, name) | ||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| from duckdb import FunctionExpression | ||
|
|
||
| from narwhals._duckdb.utils import lit | ||
|
|
||
| if TYPE_CHECKING: | ||
| from typing_extensions import Self | ||
|
|
||
| from narwhals._duckdb.expr import DuckDBExpr | ||
|
|
||
|
|
||
| class DuckDBExprStructNamespace: | ||
| def __init__(self: Self, expr: DuckDBExpr) -> None: | ||
| self._compliant_expr = expr | ||
|
|
||
| def field(self: Self, name: str) -> DuckDBExpr: | ||
| return self._compliant_expr._from_call( | ||
| lambda _input: FunctionExpression("struct_extract", _input, lit(name)).alias( | ||
| name | ||
| ), | ||
| "field", | ||
| evaluate_output_names=lambda _col: [name], | ||
| ) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,7 +25,9 @@ | |
| from typing_extensions import Never | ||
| from typing_extensions import TypeIs | ||
|
|
||
| from narwhals._arrow.dataframe import ArrowDataFrame | ||
| from narwhals._arrow.expr import ArrowExpr | ||
| from narwhals._pandas_like.dataframe import PandasLikeDataFrame | ||
| from narwhals._pandas_like.expr import PandasLikeExpr | ||
| from narwhals.expr import Expr | ||
| from narwhals.typing import CompliantDataFrame | ||
|
|
@@ -206,6 +208,10 @@ def reuse_series_namespace_implementation( | |
| expr: ArrowExprT | PandasLikeExprT, | ||
| series_namespace: str, | ||
| attr: str, | ||
| evaluate_output_names: Callable[[ArrowDataFrame], Sequence[str]] | ||
| | Callable[[PandasLikeDataFrame], Sequence[str]] | ||
| | None = None, | ||
| alias_output_names: Callable[[Sequence[str]], Sequence[str]] | None = None, | ||
| **kwargs: Any, | ||
| ) -> ArrowExprT | PandasLikeExprT: | ||
| """Reuse Series implementation for expression. | ||
|
|
@@ -217,18 +223,27 @@ def reuse_series_namespace_implementation( | |
| expr: expression object. | ||
| series_namespace: The Series namespace (e.g. `dt`, `cat`, `str`, `list`, `name`) | ||
| attr: name of method. | ||
| evaluate_output_names: Output names function. | ||
| alias_output_names: Alias output names function. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. outdated |
||
| kwargs: keyword arguments to pass to function. | ||
| """ | ||
| plx = expr.__narwhals_namespace__() | ||
|
|
||
| if evaluate_output_names is None: | ||
| evaluate_output_names = expr._evaluate_output_names | ||
|
|
||
| if alias_output_names is None: | ||
| alias_output_names = expr._alias_output_names | ||
|
|
||
| return plx._create_expr_from_callable( # type: ignore[return-value] | ||
| lambda df: [ | ||
| getattr(getattr(series, series_namespace), attr)(**kwargs) | ||
| for series in expr(df) # type: ignore[arg-type] | ||
| ], | ||
| depth=expr._depth + 1, | ||
| function_name=f"{expr._function_name}->{series_namespace}.{attr}", | ||
| evaluate_output_names=expr._evaluate_output_names, # type: ignore[arg-type] | ||
| alias_output_names=expr._alias_output_names, | ||
| evaluate_output_names=evaluate_output_names, # type: ignore[arg-type] | ||
| alias_output_names=alias_output_names, | ||
| ) | ||
|
|
||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| from __future__ import annotations | ||
|
|
||
| from typing import TYPE_CHECKING | ||
|
|
||
| from narwhals._expression_parsing import reuse_series_namespace_implementation | ||
|
|
||
| if TYPE_CHECKING: | ||
| from typing_extensions import Self | ||
|
|
||
| from narwhals._pandas_like.expr import PandasLikeExpr | ||
|
|
||
|
|
||
| class PandasLikeExprStructNamespace: | ||
| def __init__(self: Self, expr: PandasLikeExpr) -> None: | ||
| self._compliant_expr = expr | ||
|
|
||
| def field(self, name: str) -> PandasLikeExpr: | ||
| return reuse_series_namespace_implementation( | ||
| self._compliant_expr, | ||
| "struct", | ||
| "field", | ||
| name=name, | ||
| evaluate_output_names=lambda _col: [name], | ||
| ) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,9 @@ | |
|
|
||
| class PandasLikeSeriesListNamespace: | ||
| def __init__(self: Self, series: PandasLikeSeries) -> None: | ||
| if not hasattr(series._native_series, "list"): | ||
| msg = "Series must be of PyArrow List type to support struct namespace." | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. typo |
||
| raise TypeError(msg) | ||
| self._compliant_series = series | ||
|
|
||
| def len(self: Self) -> PandasLikeSeries: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For all implementations, instead of overwriting
evaluate output names, can we just use.alias?A good test would be
nw.col('a').struct.field('b').name.keep(). I think for polars the resulting column name would be 'a' we should check that we do the sameThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's odd. That's what I tried initially, but it caused some errors when evaluating (mismatching expected and actual something). I must have forgotten something. I removed the changes made to the
_from_callandreuse_series_namespace_implementationfunctions.