Automatically chunk other in GroupBy binary ops.
#7684
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dcherian
commented
Mar 27, 2023
TomNicholas
reviewed
Mar 27, 2023
| if obj.__dask_graph__() is not None and other.__dask_graph__() is None: | ||
| # a chunk size of 1 seems reasonable since we expect it to be repeated | ||
| # TODO: what about dims other than `name`` | ||
| # TODO: What about datasets with some dask vars, and others not? |
There was a problem hiding this comment.
Do we know which variables will be operated on at this point? (I'm not very familiar with this part of the code). Surely we only need to chunk the variables that will be operated on?
There was a problem hiding this comment.
Hmmm.... I guess technically we could have different data vars in obj and other
…lazy-array * upstream/main: (153 commits) Add HDF5 Section to read/write docs page (pydata#8012) [pre-commit.ci] pre-commit autoupdate (pydata#8014) Update interpolate_na in dataset.py (pydata#7974) improved docstring of to_netcdf (issue pydata#7127) (pydata#7947) Expose "Coordinates" as part of Xarray's public API (pydata#7368) Core team member guide (pydata#7999) join together duplicate entries in the text `repr` (pydata#7225) Update copyright year in README (pydata#8007) Allow opening datasets with nD dimenson coordinate variables. (pydata#7989) Move whats-new entry [pre-commit.ci] pre-commit autoupdate (pydata#7997) Add documentation on custom indexes (pydata#6975) Use variable name in all exceptions raised in `as_variable` (pydata#7995) Bump pypa/gh-action-pypi-publish from 1.8.7 to 1.8.8 (pydata#7994) New whatsnew section Remove future release notes before this release Update whats-new.rst for new release (pydata#7993) Remove hue_style from plot1d docstring (pydata#7925) Add new what's new section (pydata#7986) Release summary for v2023.07.0 (pydata#7979) ...
dcherian
commented
Jul 25, 2023
TomNicholas
approved these changes
Jul 25, 2023
Well this was a big regression for the tiny benchmark problem. I still think it's a good idea given https://discourse.pangeo.io/t/xarray-unable-to-allocate-memory-how-to-size-up-problem/3233/1 |
|
Hmm yeah I agree that scalability seems more important. Also I missed that this PR doesn't have a whatsnew entry - we should probably add one to help flag the regression. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
whats-new.rst