Add blogpost on flox heuristics #695
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Co-authored-by: Anderson Banihirwe <[email protected]>
| @@ -0,0 +1,157 @@ | |||
| --- | |||
| title: 'flox: GroupBy, now with smarts!' | |||
| date: '2024-05-31' | |||
There was a problem hiding this comment.
| date: '2024-05-31' | |
| date: '2024-08-05' |
There was a problem hiding this comment.
or whenever we decide to merge.
|
|
||
| Unlike [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index), _containment_ [isn't skewed](http://ekzhu.com/datasketch/lshensemble.html) when one of the sets is much larger than the other. | ||
|
|
||
| The steps are as follows: |
There was a problem hiding this comment.
Is this bit clear and useful? Should I take it out? Or expand it?
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Nice! Really incredible work you have done here @dcherian
I left a few comments: some nits, some minor suggestions on structure/content.
| Notice how much cleaner the graph is in this image: | ||
|  | ||
| See our [previous blog post](https://xarray.dev/blog/flox) or the [docs](https://flox.readthedocs.io/en/latest/implementation.html#method-map-reduce) for more. |
There was a problem hiding this comment.
It's a little confusing that these two images are not for identical problems - the inputs are different.
|
|
||
| _Note that the improvements here are strongly dependent on the details | ||
| of the grouping variable, the chunksize, and even dask's scheduler. In fact, writing this post | ||
| prompted the discovery of a bug in dask's scheduler that should substantially improve the "map-reduce" |
src/posts/flox-smart/index.md
Outdated
|
|
||
| Importantly this inference is fast — [~250ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where approximately 3000 groups are distributed over 2500 chunks; and ~1.25s for grouping by US watersheds ~87000 groups across 640 chunks. | ||
|
|
||
| ## What's next? |
There was a problem hiding this comment.
Your conclusion is a bit rambly, and I think some of the content (especially call-outs to any other groupby nerds who can help) could be in a footnote.
I think you should end with a succinct one-line conclusion that emphasises that most of this complexity can now be safely ignored by users.
| Using the algorithm described below, flox will **automatically** set | ||
| `method="cohorts"` for this dataset unless specified, yielding a 5X decrease in | ||
| memory used and a 2X increase in runtime. Read on to figure out how! | ||
| Read on to figure out how! |
There was a problem hiding this comment.
A significant number of readers will think it's cool that this is now automatic, but not really care about the specifics of how you did it. That section below is also very dense. I think if there is any other information that you want readers to take away from this point onwards (e.g. that Groupers exist) you should telegraph it, and explicitly give them the option to skip forwards to a different subheading guilt-free.
|
Thanks for the review @TomNicholas . I'm waiting for some tests on the new dask release before revising: https://docs.dask.org/en/stable/changelog.html#improve-scheduling-efficiency-for-xarray-groupby-reduce-patterns |
* upstream/main: Fix hydration warning due to nested links (#725) Bump typescript from 5.6.2 to 5.6.3 (#724) [pre-commit.ci] pre-commit autoupdate (#720) Bump webpack from 5.95.0 to 5.96.1 (#723) Bump the dev-dependencies group across 1 directory with 4 updates (#719) Bump webpack from 5.94.0 to 5.95.0 (#718) Bump @fontsource-variable/inter from 5.0.20 to 5.1.0 (#717) Bump typescript from 5.5.4 to 5.6.2 (#716) Bump the dependencies group with 9 updates (#714) Bump next from 14.2.7 to 14.2.10 (#711) Turn off survey banner (#710) Update opengraph thumbnails (#709) GroupBy(multiple variables) (#703) Bump webpack from 5.93.0 to 5.94.0 (#704) Bump micromatch from 4.0.7 to 4.0.8 (#707) Bump the dependencies group with 7 updates (#705) Bump the dev-dependencies group with 4 updates (#706)
|
|
||
| we run | ||
|
|
||
| ```python |
There was a problem hiding this comment.
TODO: this doesn't reproduce any more, will need a different problem
https://xarray-dev-git-flox-smart-xarray.vercel.app/blog/flox-smart