add support for read-only s3 virtual driver #1010
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -44,5 +44,6 @@ CurrentModule = HDF5.Drivers | |
| ```@docs | ||
| Core | ||
| POSIX | ||
| ROS3 | ||
| MPIO | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| """ | ||
| ROS3() | ||
|
|
||
| This is the read-only virtual driver that enables access to HDF5 objects stored in AWS S3 | ||
| """ | ||
| struct ROS3 <: Driver | ||
| fa::API.H5FD_ros3_fapl_t | ||
| end | ||
|
|
||
| function ROS3( | ||
| version::Integer, | ||
| auth::Bool, | ||
| region::AbstractString, | ||
| id::AbstractString, | ||
| key::AbstractString | ||
| ) | ||
| return (ROS3 ∘ API.H5FD_ros3_fapl_t)( | ||
| version, | ||
| auth, | ||
| Base.unsafe_convert(Cstring, region), | ||
|
There was a problem hiding this comment. Can this be a regular convert or does it have to be an There was a problem hiding this comment. It needs to be an There was a problem hiding this comment.
There was a problem hiding this comment. Probably the best option is to add them as extra fields to the ROS3 struct? There was a problem hiding this comment. Couldn't we just do There was a problem hiding this comment. I think I missed the original point of this all. Please correct me if I'm still wrong in my thinking There was a problem hiding this comment. We need to convert the Julia struct to the C struct. Part of the issue is that the Julia struct and its Strings could be garbage collected. Meanwhile, the Cstring is an unmanaged pointer to a null terminated character array. Anyways, it turns out that the C struct definition here is wrong. The correct definition is typedef struct H5FD_ros3_fapl_t {
int32_t version;
hbool_t authenticate;
char aws_region[H5FD_ROS3_MAX_REGION_LEN + 1];
char secret_id[H5FD_ROS3_MAX_SECRET_ID_LEN + 1];
char secret_key[H5FD_ROS3_MAX_SECRET_KEY_LEN + 1];
} H5FD_ros3_fapl_t;In Julia, that would be struct H5FD_ros3_fapl_t
version::Int32
authenticate::Bool
aws_region::NTuple{H5FD_ROS3_MAX_REGION_LEN + 1,Cchar}
secret_id ::NTuple{H5FD_ROS3_MAX_SECRET_ID_LEN + 1, Cchar}
secret_key::NTuple{H5FD_ROS3_MAX_SECRET_KEY_LEN + 1, Cchar};
endI've started to rewrite this. Also, I think we need to check our definition of There was a problem hiding this comment. That's probably not the first time there has been a mismatch in definitions. Do you want to push your changes on this branch? There was a problem hiding this comment. Yes, I will later today. I just started it on a different computer than I am on right now. I didn't quite get the tests to pass yet on the branch I was working on. There was a problem hiding this comment. I quickly saw this comment on the hbool issue aldanor/hdf5-rust#28 |
||
| Base.unsafe_convert(Cstring, id), | ||
| Base.unsafe_convert(Cstring, key) | ||
| ) | ||
| end | ||
|
|
||
| ROS3(region::AbstractString, id::AbstractString, key::AbstractString) = | ||
| ROS3(1, true, region, id, key) | ||
| ROS3() = ROS3(1, false, "", "", "") | ||
|
|
||
| function get_driver(fapl::Properties, ::Type{ROS3}) | ||
| r_fa = Ref{H5FD_ros3_fapl_t}() | ||
| API.h5p_get_fapl_ros3(fapl, r_fa) | ||
| return ROS3(r_fa[]) | ||
| end | ||
|
|
||
| function set_driver!(fapl::Properties, driver::ROS3) | ||
| HDF5.has_ros3() || error( | ||
| "HDF5.jl has no ros3 support." * | ||
| " Make sure that you're using ROS3-enabled HDF5 libraries" | ||
| ) | ||
| HDF5.init!(fapl) | ||
| API.h5p_set_fapl_ros3(fapl, Ref(driver.fa)) | ||
| DRIVERS[API.h5p_get_driver(fapl)] = ROS3 | ||
| return nothing | ||
| end | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| using HDF5 | ||
| using Test | ||
|
|
||
| @testset "ros3" begin | ||
| @test HDF5.has_ros3() | ||
|
|
||
| h5open( | ||
| "http://s3.us-east-2.amazonaws.com/hdf5ros3/GMODO-SVM01.h5"; | ||
| driver=HDF5.Drivers.ROS3() | ||
| ) do f | ||
| @test keys(f) == ["All_Data", "Data_Products"] | ||
| end | ||
| end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update the docstring signature
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added additonal signatures below this line.