titiler.xarray.io.Reader i/o future plans

[hrodmn]

In applications that are reading from cloud-hosted datasets there is a need for a wide variety of authentication-related modifications to the default opener in titiler.xarray.io.Reader. I am opening this discussion thread so we can come up with a strategy to improve the default behavior in a way that maximizes the utility of the default opener while maintaining flexibility for downstream applications to provide their own opener.

Related issues/PRs:
#1246
developmentseed/titiler-cmr#88
#1235
developmentseed/titiler-multidim#101

cc @jbusecke @abarciauskas-bgse @maxrjones @vincentsarago @sharkinsspatial

[vincentsarago]

I think the easiest would be to add an opener_options attribute as in https://github.com/EOPF-Explorer/titiler-eopf/blob/main/titiler/eopf/reader.py#L243

[jbusecke]

Copying some relevant comment from DS slack:

Thanks for tagging me in here @Henry Rodman. @vincentsarago for context I have been hacking away inside titiler.xarray here (relevant discussion also in this PR). This works for icechunk (native and virtual), but is at the moment just a proof of concept.
I would love to understand a bit more the design philosophy behind the current code.

• On a first thought removing netcdf support seems like a regression for many usecases? I think we are eventually aiming to push for a lot more virtualization of netcdf files.
• Would it be possible to refactor fsspec -> obstore without removing the netcdf support? I have not personally used the obstore.fsspec module but that might be a less destructive approach?
• I am also wondering if it would be easier to maintain different openers within titiler.xarray.

Like could we have xarray_open_netcdf_fsspec(src_path, group, decode_times, fsspec_kwargs), xarray_open_zarr_obstore(src_path, group, decode_times, obstore_kwargs ), xarray_open_icechunk(src_path, group, decode_times, icechunk_kwargs).

Or we could modularize this even more by splitting the file/store opening and then passing the output to a pure xarray opener.

Sorry this got a bit convoluted, but I guess my larger question is if we want to give users more control over all opening parameter in xr.open_dataset, or do we aim to push a lot of that logic to xarray upstream (this might help a lot!). And should all of this logic live in titiler.xarray or rather in apps like titiler-multidim?

Happy to chat more about this.

[jbusecke]

Notes from the conversation today.

Action items:

• @jbusecke will refactor WIP: Enable icechunk reading - rough first try #1235 into Support icechunk titiler-multidim#101 and focus on implementing runtime options for virtual chunk containers
• @abarciauskas-bgse will take a shot at creating a diagram for the titiler ecosystem and apps that illustrate how to provide custom openers.
• @vincentsarago is working on refactoring titiler to use obstore instead of fsspec and suggests to eventually move to only support zarr by default.