[FEAT] Approximate quantile aggregation (pulled into main) #2179
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2179 +/- ##
=======================================
Coverage ? 85.33%
=======================================
Files ? 69
Lines ? 7458
Branches ? 0
=======================================
Hits ? 6364
Misses ? 1094
Partials ? 0
|
There was a problem hiding this comment.
Overall lookin really good! A few high level things:
- Add the expression to the docs
- Do we want to expose top level
approx_percentileexpressions for this? (on df and grouped df) so users can dodf.approx_percentiles("values")ordf.group_by('group').approx_percentiles("values")
daft/expressions/expressions.py
Outdated
| @@ -434,6 +434,21 @@ def sum(self) -> Expression: | |||
| expr = self._expr.sum() | |||
| return Expression._from_pyexpr(expr) | |||
|
|
|||
| def approx_percentiles(self, percentiles: builtins.float | builtins.list[builtins.float]) -> Expression: | |||
| """Calculates the approximate percentile(s) for a float column | |||
There was a problem hiding this comment.
Would be cool to enable support for temporal columns in the future
There was a problem hiding this comment.
We should actually expose this functionality to support all ordered columns (numeric, temporal, strings...)
Then we can leverage this for our sorts. Currently our sorts perform its own bespoke solution for sampling for boundaries before repartitioniong, but we can leverage approx_percentiles to generate those boundaries for us!
github-actions
bot
added
the
documentation
|
FYI @maxime-petitjean ! |
Puts the finishing touches on #2076