Adapt to non-string task keys in distributed #1225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Now that task keys are not always strings, the file-backed zict mapping needs to explicitly stringify keys. Distributed has such a mapping implemented so we just import it and use it here.
Now that task keys are no longer strings when stored in the worker data dictionary, we should not stringify the set of keys when staging. - Closes rapidsai#1224
|
/merge |
|
Once the next release is out, we will probably want to bump our minimum Dask pinning to 2023.8.2, as this PR now makes it such that we fail to import with the latest stable release (2023.8.1) |
I wouldn't mind this but we generally agreed to only do this at release time, the reason for that is there's a heavy load to do that across all projects. Once we have the metapackage to pin Dask versions across RAPIDS this will become much simpler. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now that keys are no longer strings there are two places we must adapt here.
zict.File-backed slow buffer inDeviceHostFileneeds to translate non-string keys to string keys before writing to disk, to do this, use the same implementation that distributed uses for its own spilling buffer.