Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clarify separation between rust code and python wrappers #779

Closed
Tracked by #828
Michael-J-Ward opened this issue Jul 26, 2024 · 0 comments · Fixed by #860
Closed
Tracked by #828

clarify separation between rust code and python wrappers #779

Michael-J-Ward opened this issue Jul 26, 2024 · 0 comments · Fixed by #860

Comments

@Michael-J-Ward
Copy link
Contributor

#750 added python wrappers instead of directly exposing pyo3 generated code.

We should leverage that to clean up and improve the rust codebase by:

  1. Letting python wrappers handle any aliases.
  2. Letting python wrappers handle any argument defaults, removing #[pyo3(signature(...))] annotations from the Rust codebase

Additionally, we should provide guidance so that contributors understand the separation.

@timsaucer recommends the following guidelines (and I agree):

I think it's worth adding some guidance about what to put in python vs what to put in rust for this repo. In my mind the things that should go into the python side are

* Trivial aliases (this is a good example)
* Simple type conversion, like path -> string of the path or number to lit(number)
* More complex type conversion if it makes sense to do in python. For example, in the named_struct where sending in a dictionary on the python side makes a lot more sense and isn't as simple to do via pyo3.

And then I'd lean towards everything else sitting on the rust side.
Michael-J-Ward added a commit to Michael-J-Ward/datafusion-python that referenced this issue Jul 29, 2024
@Michael-J-Ward
chore: remove explicit aliases from functions.rs
0c57e59 
Ref: apache#779
andygrove pushed a commit that referenced this issue Jul 31, 2024
@Michael-J-Ward
Upgrade Datafusion 40 (#771)
f580155 
* chore: update datafusion deps

* feat: impl ExecutionPlan::static_name() for DatasetExec

This required trait method was added upstream [0] and recommends to simply forward to `static_name`.

[0]: apache/datafusion#10266

* feat: update first_value and last_value wrappers.

Upstream signatures were changed for the new new `AggregateBuilder` api [0].

This simply gets the code to work. We should better incorporate that API into `datafusion-python`.

[0] apache/datafusion#10560

* migrate count to UDAF

Builtin Count was removed upstream.

TBD whether we want to re-implement `count_star` with new API.

Ref: apache/datafusion#10893

* migrate approx_percentile_cont, approx_distinct, and approx_median to UDAF

Ref: approx_distinct apache/datafusion#10851
Ref: approx_median apache/datafusion#10840
Ref: approx_percentile_cont and _with_weight apache/datafusion#10917

* migrate avg to UDAF

Ref: apache/datafusion#10964

* migrage corr to UDAF

Ref: apache/datafusion#10884

* migrate grouping to UDAF

Ref: apache/datafusion#10906

* add alias `mean` for UDAF `avg`

* migrate stddev to UDAF

Ref: apache/datafusion#10827

* remove rust alias for stddev

The python wrapper now provides stddev_samp alias.

* migrage var_pop to UDAF

Ref: apache/datafusion#10836

* migrate regr_* functions to UDAF

Ref: apache/datafusion#10898

* migrate bitwise functions to UDAF

The functions now take a single expression instead of a Vec<_>.

Ref: apache/datafusion#10930

* add missing variants for ScalarValue with todo

* fix typo in approx_percentile_cont

* add distinct arg to count

* comment out failing test

`approx_percentile_cont` is now returning a DoubleArray instead of an IntArray.

This may be a bug upstream; it requires further investigation.

* update tests to expect lowercase `sum` in query plans

This was changed upstream.

Ref: apache/datafusion#10831

* update ScalarType data_type map

* add docs dependency pickleshare

* re-implement count_star

* lint: ruff python lint

* lint: rust cargo fmt

* include name of window function in error for find_window_fn

* refactor `find_window_fn` for debug clarity

* search default aggregate functions by both name and aliases

The alias list no longer includes the name of the function.

Ref: apache/datafusion#10658

* fix markdown in find_window_fn docs

* parameterize test_window_functions

`first_value` and `last_value` are currently failing and marked as xfail.

* add test ids to test_simple_select tests marked xfail

* update find_window_fn to search built-ins first

The behavior of `first_value` and `last_value` UDAFs currently does not match the built-in behavior.
This allowed me to remove `marks=pytest.xfail` from the window tests.

* improve first_call and last_call use of the builder API

* remove trailing todos

* fix examples/substrait.py

* chore: remove explicit aliases from functions.rs

Ref: #779

* remove `array_fn!` aliases

* remove alias rules for `expr_fn_vec!`

* remove alias rules from `expr_fn!` macro

* remove unnecessary pyo3 var-arg signatures in functions.rs

* remove pyo3 signatures that provided defaults for first_value and last_value

* parametrize test_string_functions

* test regr_ function wrappers

Closes #778
@Michael-J-Ward Michael-J-Ward mentioned this issue Aug 20, 2024
Closed
9 tasks
@Michael-J-Ward Michael-J-Ward mentioned this issue Sep 6, 2024
Merged
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add guidelines on separating python and rust code Michael-J-Ward/datafusion-python
1 participant
@Michael-J-Ward