-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement a rex
backend for xarray (rexarray)
#192
Conversation
rex
backend for xarray (rexarray)rex
backend for xarray (rexarray)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will take a deeper dive later but here are just a couple random comments.
@bnb32, @grantbuster, @castelao Friendly bump :) Curious to hear if anyone has tried using this and/or had success with it |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ppinchuk , some minor comments.
Digging back in is on the list for next week :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, this looks great to me. I messed around in a notebook a little bit and things worked well, including within calls to the sup3r Loader
objects. A couple random comments added. Main question on the use of time_index
vs time
.
class RexArrayWrapper(BackendArray): | ||
"""rexarray implementation of a `BackendArray`""" | ||
|
||
__slots__ = ("datastore", "dtype", "shape", "variable_name", "meta_index", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you find that using slots here reduces mem use significantly? This is one of those things I'm aware of but rarely use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used __slots__
quite heavily in my grad work, but not so much at NREL. IIRC the guidance is that it can be beneficial if you intend to have thousands of instances of the object. Not sure if that actually applies here, but this is what the xarray developers did for their array wrappers and I figured they had a good reason for doing so (so I copied them😄).
Fine by me to add fsspec as a default req! |
ensure that rexarray can open mf hsds with list input
Implement a `rex` backend for xarray (rexarray)
Implement a `rex` backend for xarray (rexarray)
Add a backend for
xarray
that allows users to read in rex-style NREL data.The implementation itself closely follows the
h5netcdf
backend implementation:https://github.com/pydata/xarray/blob/main/xarray/backends/h5netcdf_.py
Lazy loading is fully supported, which is a big reason why this implementation is so long.
HSDS and S3 (via
fsspec
) access is explicitly supported.