zarr store written by iohub.writer doesn't seem to be compliant with OME-HCS format

[mattersoflight]

Using napari-ome-zarr plugin to view the dataset written by the writer module causes an error that suggests missing metadata:
napari --plugin napari-ome-zarr /hpc/projects/CompMicro/rawdata/hummingbird/Janie/2022_03_15_orgs_nuc_mem_63x_04NA/all_21_3.zarr

causes the following error:

ERROR Failed to load Row_0/Col_271/0/0
Traceback (most recent call last):
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/spec.py", line 811, in cat
    return self.cat_file(paths[0], **kwargs)
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/spec.py", line 710, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/spec.py", line 1094, in open
    f = self._open(
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/implementations/local.py", line 175, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/implementations/local.py", line 273, in __init__
    self._open()
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/implementations/local.py", line 278, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/hpc/projects/CompMicro/rawdata/hummingbird/Janie/2022_03_15_orgs_nuc_mem_63x_04NA/all_21_3.zarr/Row_0/Col_271/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/zarr/storage.py", line 1376, in __getitem__
    return self.map[key]
  File "/hpc/user_apps/comp_micro/conda_envs/recorder/lib/python3.9/site-packages/fsspec/mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: 'Row_0/Col_271/0/0/.zarray'

This .zarray json file is found in the reference data. It can be viewed using:
napari --plugin napari-ome-zarr /local/scratch/groups/cmanalysis.grp/zarr_ref_datasets/2551.zarr/

I copied the reference dataset with the following aws cli command:
aws --no-sign-request --endpoint-url=https://uk1s3.embassy.ebi.ac.uk/ s3 cp --recursive --exclude=".git/*" s3://idr/zarr/v0.1/plates/2551.zarr /local/scratch/groups/cmanalysis.grp/zarr_ref_datasets/2551.zarr

[ziw-liu]

iohub.writer offers optional support to write HCS-compliant stores that needs to be explicitly specified:
https://github.com/czbiohub/iohub/blob/2564737c400d38c0856b63f51680e5c222e083ff/iohub/writer.py#L22-L26

@mattersoflight Do you know how is this dataset generated? If it's a recOrder acquisition then it does not use the HCS option.

[ziw-liu]

In addition to this, it does seem that the HCS writer writes to an old spec which may not be supported by newer versions of napari-ome-zarr. (See also mehta-lab/waveorder#94).

While waveorder may be constrained by backwards-compatibility issues, an upgrade to the newest spec (v0.4 or v0.5) should be implemented in iohub.