`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94

talonchandler · 2022-11-24T02:16:51Z

I first noticed this issue during a recOrder session when I tried to drag and drop a zarr store saved by a recOrder acquisition into napari. The layers and shape populated correctly, but the actual pixel values were set to zero.

Digging further, I've realized that this is a WaveorderWriter issue. Here's a minimal example:

$ python
>>> import numpy as np
>>> from waveorder.io import WaveorderWriter
>>> writer = WaveorderWriter("./test")
>>> writer.create_zarr_root("test")
>>> writer.init_array(position=0, data_shape=(1,1,2,3,4), chunk_size=(1,1,2,3,4), chan_names=["TEST"])
>>> writer.write(np.random.random((1,1,2,3,4)), p=0)
>>> quit()
$ napari --plugin napari-ome-zarr test/test.zarr

results in this traceback:

18:06:35 WARNING version mismatch: detected: FormatV01, requested: FormatV04
18:06:35 ERROR Failed to load Row_0/Col_0/0/0
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 665, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 1034, in open
    f = self._open(
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 162, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 260, in __init__
    self._open()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 265, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'Q:/Talon/2022_11_23/recOrderPluginSnap_0/test/test.zarr/Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1376, in __getitem__
    return self.map[key]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: 'Row_0/Col_0/0/0/.zarray'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 241, in _load_metadata_nosync
    meta_bytes = self._store[mkey]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1378, in __getitem__
    raise KeyError(key) from e
KeyError: 'Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\reader.py", line 548, in get_tile
    data = self.zarr.load(path)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\io.py", line 113, in load
    return da.from_zarr(self.__store, subpath)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\dask\array\core.py", line 3569, in from_zarr
    z = zarr.Array(mapper, read_only=True, path=component, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 215, in __init__
    self._load_metadata()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 232, in _load_metadata
    self._load_metadata_nosync()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 243, in _load_metadata_nosync
    raise ArrayNotFoundError(self._path)
zarr.errors.ArrayNotFoundError: array not found at path %r' 'Row_0/Col_0/0/0'
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\Scripts\napari.exe\__main__.py", line 7, in <module>
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\__main__.py", line 446, in main
    _run()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\__main__.py", line 335, in _run
    run(gui_exceptions=True)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\_qt\qt_event_loop.py", line 402, in run
    app.exec_()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\contextlib.py", line 126, in __exit__
    next(self.gen)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\napari\_qt\utils.py", line 459, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)
KeyboardInterrupt
(recorder-dev) PS Q:\Talon\2022_11_23\recOrderPluginSnap_0> python
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:51:29) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> from waveorder.io import WaveorderWriter
>>> writer = WaveorderWriter("./test")
>>> writer.create_zarr_root("tes")
Creating new zarr store at ./test\tes.zarr
>>> writer.init_array(position=0, data_shape=(1,1,2,3,4), chunk_size=(1,1,2,3,4), chan_names=["TEST"])
>>> writer.write(np.random.random((1,1,2,3,4)), p=0)
>>> quit()
(recorder-dev) PS Q:\Talon\2022_11_23\recOrderPluginSnap_0> napari --plugin napari-ome-zarr test/tes.zarr
18:11:25 WARNING version mismatch: detected: FormatV01, requested: FormatV04
18:11:25 ERROR Failed to load Row_0/Col_0/0/0
Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 137, in __getitem__
    result = self.fs.cat(k)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 755, in cat
    return self.cat_file(paths[0], **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 665, in cat_file
    with self.open(path, "rb", **kwargs) as f:
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\spec.py", line 1034, in open
    f = self._open(
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 162, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 260, in __init__
    self._open()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\implementations\local.py", line 265, in _open
    self.f = open(self.path, mode=self.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'Q:/Talon/2022_11_23/recOrderPluginSnap_0/test/tes.zarr/Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1376, in __getitem__
    return self.map[key]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\fsspec\mapping.py", line 141, in __getitem__
    raise KeyError(key)
KeyError: 'Row_0/Col_0/0/0/.zarray'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 241, in _load_metadata_nosync
    meta_bytes = self._store[mkey]
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\storage.py", line 1378, in __getitem__
    raise KeyError(key) from e
KeyError: 'Row_0/Col_0/0/0/.zarray'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\reader.py", line 548, in get_tile
    data = self.zarr.load(path)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\ome_zarr\io.py", line 113, in load
    return da.from_zarr(self.__store, subpath)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\dask\array\core.py", line 3569, in from_zarr
    z = zarr.Array(mapper, read_only=True, path=component, **kwargs)
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 215, in __init__
    self._load_metadata()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 232, in _load_metadata
    self._load_metadata_nosync()
  File "C:\Users\LabelFree\.conda\envs\recorder-dev\lib\site-packages\zarr\core.py", line 243, in _load_metadata_nosync
    raise ArrayNotFoundError(self._path)
zarr.errors.ArrayNotFoundError: array not found at path %r' 'Row_0/Col_0/0/0'

It seems related to our naming convention of Row_0/Col_0/Pos_000. Do we need to change this convention, or change the metadata so that ome-zarr can read our zarrs?

I think this is reasonably important because there are lots of cases when I'd like to use the ome-zarr reader to split zarr's channels into layers (it's native behaviour) instead of the recOrder-napari reader that opens as a single array (also convenient in some cases). It also causes confusion because the reader seems to work but has zeros---did the reconstruction get saved? (It does)

The text was updated successfully, but these errors were encountered:

ziw-liu · 2022-11-29T01:00:33Z

It's possible that napari-ome-zarr throws KeyError: 'Row_0/Col_0/0/0/.zarray' because it expects a newer version of OME-NGFF format. Taking a look at its source, it does not use the fmt: Format = FormatV04 option of ome-zarr.reader.Reader:
https://github.com/ome/napari-ome-zarr/blob/1d423d21a76b6cda8f5feb2b7292ace96df708ae/napari_ome_zarr/_reader.py#L33-L35

ziw-liu · 2022-11-29T01:48:03Z

Possibly introduced in:
0.2.0 | 2021-03-29 | Change chunk dimension separator to "/"

ziw-liu · 2022-12-01T17:32:58Z

Taking a look at its source, it does not use the fmt: Format = FormatV04 option of ome-zarr.reader.Reader

First line of the traceback indicates the problem:

18:06:35 WARNING version mismatch: detected: FormatV01, requested: FormatV04

While ome-zarr-py should be capable of reading data in earlier OME-NGFF versions, napari-ome-zarr does not utilize this feature.

ziw-liu · 2022-12-01T17:39:30Z

Related: ome/napari-ome-zarr#39

ziw-liu · 2023-02-17T18:34:54Z

Now implemented as czbiohub-sf/iohub#31.

We can keep this issue until iohub is released and used by recOrder, and waveorder.io can be removed from waveorder v1.0.0.

talonchandler · 2023-04-19T16:06:03Z

iohub has been released, so this is fixed. Closing.

talonchandler assigned ziw-liu Nov 24, 2022

This was referenced Nov 29, 2022

zarr store written by iohub.writer doesn't seem to be compliant with OME-HCS format czbiohub-sf/iohub#3

Closed

Use ome-zarr-py to write to zarr czbiohub-sf/iohub#9

Closed

ziw-liu mentioned this issue Feb 8, 2023

Refactor API for NGFF datasets czbiohub-sf/iohub#31

Merged

ziw-liu added the bug Something isn't working label Feb 8, 2023

talonchandler closed this as completed Apr 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94

`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94

talonchandler commented Nov 24, 2022

ziw-liu commented Nov 29, 2022

ziw-liu commented Nov 29, 2022

ziw-liu commented Dec 1, 2022

ziw-liu commented Dec 1, 2022

ziw-liu commented Feb 17, 2023

talonchandler commented Apr 19, 2023

WaveorderWriter is not compatible with napari-ome-zarr reader #94

WaveorderWriter is not compatible with napari-ome-zarr reader #94

Comments

talonchandler commented Nov 24, 2022

ziw-liu commented Nov 29, 2022

ziw-liu commented Nov 29, 2022

ziw-liu commented Dec 1, 2022

ziw-liu commented Dec 1, 2022

ziw-liu commented Feb 17, 2023

talonchandler commented Apr 19, 2023

`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94

`WaveorderWriter` is not compatible with `napari-ome-zarr` reader #94