Writing ome-zarr in a serial fashion, one 2D slice at a time

[ckolluru]

Hi

Is there a good way to setup writing to zarr format using this package, by writing one 2D XY slice at a time?

My current workflow is below:

import zarr

# Create a new zarr folder
ZARR_STORE = zarr.open(directory, mode="w")
ch0 = ZARR_STORE.zeros("ch0", shape=(num_time_points, 1, Y_SHAPE_ZARR, X_SHAPE_ZARR), chunks=(1, 1, Y_SHAPE_ZARR, X_SHAPE_ZARR), dtype="i2")

print('Zarr directory tree')
print(ZARR_STORE.tree())

dataset = ZARR_STORE["ch0"]

for i in range(num_time_points):
     dataset[i, 0, :, :] = image   # image at timepoint i

I can create a zarr directory structure this way, but I'm unable to open the stack in Fiji after installing the MoBIE plugin and trying the Open OME-Zarr from file system command. Am I missing something here? I'm assuming this package should let me write images serially from python and open in Fiji.

I checked the documentation but write_image seems to need the full numpy array beforehand.

Thanks,
Chaitanya

[will-moore]

write_image() will take a dask array.

There's some discussion of using dask for this at:
https://forum.image.sc/t/creating-an-ome-zarr-dynamically-from-tiles-stored-as-a-series-of-images-list-of-centre-positions-using-python/81657/4 although that might be more complex than you need.

A simpler alternative might be to use the approach at

ome-zarr-py/ome_zarr/reader.py

Line 534 in 685eb89

def get_stitched_grid(self, level: int, tile_shape: tuple) -> da.core.Array:

for concatenating Wells into a Plate.
In that example, we're concatenating 5D images (1 from each Well of a Plate) across X and Y axes.
You'd want to concatenate 2D planes across T.

pseudo-code!

def get_dask_image():

  def get_plane(t):
   # return image at t-index

  lazy_plane = delayed(get_plane)

  return da.concatenate([lazy_plane(t) for t in range(num_time_points)], axis=0)

write_image(get_dask_image()....)

But there's nothing wrong with doing what you're doing too.

Can you try to view in vizarr and ome-zarr-validator?

Just put this python script as simple_cors.py in the same (or parent) dir as your image.zarr

# From https://stackoverflow.com/questions/21956683/enable-access-control-on-simple-http-server
from http.server import HTTPServer, SimpleHTTPRequestHandler, test
import sys

class CORSRequestHandler (SimpleHTTPRequestHandler):
    def end_headers (self):
        self.send_header('Access-Control-Allow-Origin', '*')
        SimpleHTTPRequestHandler.end_headers(self)

if __name__ == '__main__':
    test(CORSRequestHandler, HTTPServer, port=int(sys.argv[1]) if len(sys.argv) > 1 else 8000)

Then run python simple_cors.py and open your browser at https://ome.github.io/ome-ngff-validator/?source=http://localhost:8000/image.zarr (renaming image.zarr to the name of your data).
That should check your data is valid, and gives you a link to view in vizarr and other tools (and tips for opening in MoBIE).

Do you have MoBIE working with other sample data?

[ckolluru]

@will-moore : thanks for your reply.

I'm able to create the test image using the script here and open that in BigDataViewer in Fiji via the MoBIE plugin. It looks fine.

I don't think dask would work in my case since I'm collecting images and different timepoints across several hours. I would like to have the zarr dataset built up during acquisition and ready to go.

I tried running the ome-ngff validator on my dataset and it initially gave an error that it cannot find the .zattrs file. In fact, I can only see a .zgroup file in my folder. This is the folder structure:

filename.zarr
-- ch0
------.zarray
------0.0.0.0
------1.0.0.0
------2.0.0.0
....

--.zgroup

I tried creating a .zattrs file with the text below, and the validator now thinks the schema is correct, but raises an Error: Dimension separator must be / for version 0.4. I'm not creating a multiresolution pyramid.

{
    "multiscales": [
        {
            "axes": [
		{
		    "name": "t",
		    "type": "time"
		},
                {
                    "name": "z",
                    "type": "space",
		    "unit": "micrometer"
                },
                {
                    "name": "y",
                    "type": "space",
		    "unit": "micrometer"
                },
                {
                    "name": "x",
                    "type": "space",
		    "unit": "micrometer"
                }
            ],
            "datasets": [
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
				1.0,
                                1.0,
                                0.9,
                                0.9
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "ch0"
                }
            ],
            "version": "0.4"
        }
    ]
}

Can you let me know what I'm missing?

Thanks,
Chaitanya

[will-moore]

So, the v0.2 - v0.4 OME-NGFF spec states that you should use a / for the dimension separator, which means that a chunk will have a path of e.g. 0/0/0/0/0 etc instead of 0.0.0.0.0. This means that each level of the path is a new directory, which was designed to prevent having all the chunk files in a single directory, which can get too many for some filesystems.
But, this is more like a "recommendation" of OME-ZARR (rather than the Zarr format itself), so any Zarr library should still be able to read those chunks.

In your original post, you're not doing anything to create an OME-Zarr.
You need to pass the array you've created to the ome-zarr write_image() as in the script you tried.

write_image(image=dataset, .....)

This will create a pyramid from your data (by downsampling in X and Y dimensions) and also create the .zattrs for you, based on the dimension names you provide.

However, if you've already created the.zattrs manually, and the validator is happy (apart from the dimension separator) then that should open with MoBIE?

[ckolluru]

Got you, using a '/' as the dimension separator and manually creating a .zattrs file seems to work for me.

I see that I could write to ome-zarr programmatically via Dask using the approach you've linked to earlier (here), so I will close this issue for now.

Thanks.