Listing every format that could be represented as virtual zarr

[TomNicholas]

Let's list all the file formats that could potentially be represented efficiently as "virtual zarr" - i.e. zarr + chunk manifests.

The important criteria here is that the format must store data in a small number of contiguous chunks, such that access using http range requests to object storage is efficient. This rules out some formats, for example I don't think we can efficiently access this format that @kmuehlbauer mentioned over in openradar/xradar#187 (comment):

file formats where variables are written interleaved within one chunk of data (eg: 100 bytes v1, 100 bytes v2, 100 bytes v3, 100 bytes v1, 100 bytes v2, 100 bytes v3, ...)? Is there something like strides available?

If we start thinking of Zarr as a "SuperFormat" (super as in superset, not as in super-duper), then this is the list of existing formats comprising that set of what can be referenced using chunk manifests (see zarr-developers/zarr-specs#287).

---

Definitely can support:

• HDF5
• NetCDF4
• GRIB Non-kerchunk backend for GRIB files. #312
• TIFF
• FITS
• DMR++ Reading from dmrcp index files? #85
• Zarr itself! (a "Zarr of Zarrs") Add Zarr Reader(s) #262

Probably can support:

• HDF4 Support HDF4? #216
• arbitrary collections of .npz files
• HEC-RAS HDF5 data Nested HDF5 Data / HEC-RAS fsspec/kerchunk#490
• Hugging Face safetensors (see Reader for Hugging Face's SafeTensor format #367)

Maybe can support?

• CSV CSV reader #200
• Virtual Rasters GDAL Virtual Rasters #166
• rainbow Kerchunk / VirtualiZarr way to open radar files openradar/xradar#187 (comment)
• MATLAB .mat files (specification documented here)
• Parquet? Virtualize Parquet? #441

Probably can't support:

• furuno Kerchunk / VirtualiZarr way to open radar files openradar/xradar#187 (comment)
• sigmet/nexrad Kerchunk / VirtualiZarr way to open radar files openradar/xradar#187 (comment)

(The checkboxes indicate whether or not a working implementation already exists - going through kerchunks' in-memory format as an intermediate or creating a ManifestArray directly.)

cc @jhamman @d-v-b

[maxrjones]

Unfortunately based on https://gdal.org/user/virtual_file_systems.html#jpeg2000 JPEG2000 is likely in the 'probably can't support' category. I would've liked if these datasets could be virtualized, but they're all JPEG2000 for to optimize for the download to disk model :(

Another way to phrase this question, which may help the search, is which of the formats supported by GDAL's raster drivers can be virtualized?

[martindurant]

I like this issue! It's worth saying that anything kerchunk can chunk can be v-zarrred, right? In that repo, there are suggestions of other worthwhile formats, dicom and nifti (medical imaging) spring to mind. The latter is nice, but often whole-file-gzipped, the former is evil in the way that other 90s standards are evil, but extremely widespread.

[norlandrhagen]

... the former is evil in the way that other 90s standards are evil, but extremely widespread.

❤️

[TomNicholas]

anything kerchunk can chunk can be v-zarrred, right?

Yes, that's the idea. This function does kerchunk refs -> virtual dataset, and this function does virtual dataset -> kerchunk refs. Any additional kerchunk file readers can be called as another if...else... in here.

[TomNicholas]

Hugging Face safetensors is an interesting example - it's uncompressed so basically just like reading netCDF3, having no internal chunking. But it also puts all the metadata at the start of the file, making it a bit like cloud-optimized HDF5. See also huggingface/safetensors#527 (comment)

[martindurant]

If the format is simple and common, I say it should be included immediately, especially when there is a straight-forward way to check correctness.

having no internal chunking

but you can assign internal chunking. Is partial reading available in upstream at all yet?

[TomNicholas]

If the format is simple and common, I say it should be included immediately, especially when there is a straight-forward way to check correctness.

I raised #367 to track adding it.

but you can assign internal chunking. Is partial reading available in upstream at all yet?

This issue seems to suggest it is: zarr-developers/zarr-python#1106. But I think to take advantage of this with virtualizarr would require #199 to be merged.

[martindurant]

No, zarr's PR1106 only implemented it for blosc compression, something I've been arguing about for a very very long time!

If you can dynamically re-imagine the chunking at runtime (which is what I tink #119 does), that that would be good enough for most practical uses - but still annoying. Zarr should just do this! i.e., the chunk IO function shouldn't just be passed "I need chunk X", but "I need section (:, s:t, i:j) of chunk X" and a way to characterise what the decompression pipeline looks like (this is OK for decompressed, some blosc, zstd maybe..., but not zlib). This was my suggestion in passing Contexts around in zarr v2.

[TomNicholas]

I don't disagree, but if we want to discuss this further we should do it on a new issue (on this repo or upstream on zarr).

[itsgifnotjiff]

Environment and Climate Change Canada here 😊. We have something called RPN/FST files which are binary files containing metadata, grid and fortran arrays.

They are incredibly efficient in both disk space and compression (optional).

c/Fortran library to work with them
Python library that opens them into xarray Datasets with attrs and all

We would love to use something like VirtualiZarr to create per model data cubes that can be easily accessed by ML modules like Jax/torch/Dask but also by Web APIs such as pygeoapi for OGC APIs.

Some of the difficulties are the fact that we have multiple grids and vertical dimensions per file and not all files of a model contain the same variables (accumulation variables start after 0 forecast hour).

Indexing our data lake is something doable but up until recent advancements in both Zarr indexing and Grouping it did not really fit our plans.

Tools like SHTools, XPublish, XCube, MetPy, Holoviz, etc. All have an Xarray entry point but some particularities makes the workflow still experimental and hacky.

If there was a way to create a massive Zarr/Icechunk/TileDB/Delta Lake on top of a GPFS/Ceph filesystem mounted on a HPC where perhaps access to the data can be accelerated through extra CPU/GPUs that would allow me and my colleagues to build with confidence tooling for some proper Science 😊.

[martindurant]

@itsgifnotjiff , that sounds like it might be the perfect opportunity for me to build a vzarr reader rather than a kerchunk indexer (unless @TomNicholas , you don't think it's worth making a difference between the two). I am in Toronto, so if you are two, I'd be happy to meet in person to go over the format and requirements.

[itsgifnotjiff]

I am actually in Montreal. 😶 I am however available and motivated. Let me know if you have time to meet virtually if nothing else just for me and my boss to say thank you for your wonderful work.

My email for work

[TomNicholas]

binary files containing metadata, grid and fortran arrays.

Yet another self-describing binary file format! Sounds like something that could be virtualized.

We would love to use something like VirtualiZarr to create per model data cubes that can be easily accessed by ML modules like Jax/torch/Dask

Yes, it should be possible to provide access via the zarr-python API.

also by Web APIs such as pygeoapi for OGC APIs.

I'm less familiar with this, but there might be a way to do that.

Some of the difficulties are the fact that we have multiple grids and vertical dimensions per file and not all files of a model contain the same variables (accumulation variables start after 0 forecast hour).

The (current) set of restrictions for the virtual zarr approach are documented here. As your data apparently can already be mapped to the xarray/netCDF data model, as long as you don't have any of those listed issues then I would expect this to be possible.

data lake

A cloud data lake?

If there was a way to create a massive Zarr/Icechunk/TileDB/Delta Lake on top of a GPFS/Ceph filesystem mounted on a HPC

This is technically possible using VirtualiZarr and Icechunk together, though currently the support within icechunk for Ceph & and HPC are underdeveloped compared to the typical AWS use case.

---

perfect opportunity for me to build a vzarr reader rather than a kerchunk indexer

@martindurant it would be great for you to have a go at building a virtualizarr reader (bearing in mind we're just finishing up a refactor that will simply the definition of what a virtualizarr reader actually is), but @itsgifnotjiff as the owner of a bespoke file format you should also be the owner of any code designed to read that format. VirtualiZarr is designed to be extensible in that respect, so you can write a dedicated virtualizarr reader for RPN/FST files that lives outside of this main virtualizarr repository.

[itsgifnotjiff]

A cloud data lake?

Internal cloud ... ish 🙂 (I can get into details if needed)

VirtualiZarr is designed to be extensible in that respect, so you can write a dedicated virtualizarr reader for RPN/FST files that lives outside of this main virtualizarr repository.

That's great to hear! We have published some tools that allow for advanced workflows such as

1. Open an RPN file by creating a Polars or Pandas dataframe where each row has the data for a single record and columns for all its metadata and a final column with the actual data (numpy array at the moment).
2. Filter, add, remove etc. in memory as well as perform post proccessing tasks (important for atmospheric sciences) and a new library coming soon that should be able to reproject (horizontally and vertically) any RPN but also Xarray Dataset in memory from any grid/vertical to any grid/vertical called GeoRef (think the completion of the xesmf project by ECMWF).
3. Opening any RPN file/files directly into an Xarray Dataset by using engine="fstd" or through fstd2nc.

[TomNicholas]

Cool! Let us know how you get on with a virtualizarr reader, and please raise new issues if you have any problems.