Use Dask delayed to export large datasets to NetCDF #5391
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jenshnielsen
commented
Sep 28, 2023
jenshnielsen
commented
Sep 28, 2023
jenshnielsen
commented
Sep 28, 2023
jenshnielsen
commented
Sep 28, 2023
jenshnielsen
commented
Sep 28, 2023
jenshnielsen
commented
Sep 28, 2023
Codecov Report
@@ Coverage Diff @@
## master #5391 +/- ##
==========================================
+ Coverage 68.27% 68.30% +0.03%
==========================================
Files 341 341
Lines 29901 29935 +34
==========================================
+ Hits 20414 20448 +34
Misses 9487 9487 |
astafan8
reviewed
Sep 29, 2023
jenshnielsen
force-pushed
the
export_large_dataset
branch
from
b20e83c to
be08a61
Compare
jenshnielsen
commented
Oct 2, 2023
jenshnielsen
force-pushed
the
export_large_dataset
branch
from
0e2b17a to
161ee76
Compare
jenshnielsen
commented
Oct 4, 2023
Significantly reduces memory footprint of exporting larger files
This cannot work for ds in mem
jenshnielsen
force-pushed
the
export_large_dataset
branch
from
50cd392 to
e66a1c2
Compare
jenshnielsen
changed the title
astafan8
approved these changes
Oct 4, 2023
Co-authored-by: Mikhail Astafev <[email protected]>
Co-authored-by: Mikhail Astafev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Avoid loading a potentially huge dataset into memory just to be able to rewrite it to disk.
This writes each row of a dataset to disk individually and then uses dask/xarray to lazily merge the dataset into one file