Reaching maximum recursion depth while loading zarr file

[FIrgolitsch]

Hello,

I've created a zarr file where I stored a 4D image (2-channel 3D image) and written a label inside the zarr file using the write_labels function. Afterwards when I try to load the image, and list its nodes, I get an error message saying it has reached maximum recursion depth. When I look at the nodes that are in the reader object I find the final two entries to be this:

<path>/S23.zarr/labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/ [zgroup]

<path>/S23.zarr/labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/ [zgroup]

It seems that when it reaches the mask node, it tries to go back to the original image path using ../../ which I set in the metadata using the source property as per the example in section 2.6 of the NGFF specification. What could be causing this recursion issue? I have had it working before and I am unsure why it would start to recursively load the same nodes.

This is the metadata of the labels file:

{
    "image-label": {
        "colors": [
            {
                "label-value": 0,
                "rgba": [
                    0,
                    0,
                    0,
                    0
                ]
            },
            {
                "label-value": 1,
                "rgba": [
                    0,
                    255,
                    0,
                    64
                ]
            }
        ],
        "source": {
            "image": "../../"
        },
        "version": "0.4"
    },
    "multiscales": [
        {
            "axes": [
                {
                    "name": "z",
                    "type": "space",
                    "unit": "micrometer"
                },
                {
                    "name": "y",
                    "type": "space",
                    "unit": "micrometer"
                },
                {
                    "name": "x",
                    "type": "space",
                    "unit": "micrometer"
                }
            ],
            "datasets": [
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                6.5,
                                6.5,
                                6.5
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "0"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                13.0,
                                13.0,
                                13.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "1"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                26.0,
                                26.0,
                                26.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "2"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                52.0,
                                52.0,
                                52.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "3"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                104.0,
                                104.0,
                                104.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "4"
                }
            ],
            "name": "mask",
            "version": "0.4"
        }
    ]
}

And the metadata of the image itself:

{
    "multiscales": [
        {
            "axes": [
                {
                    "name": "c",
                    "type": "channel"
                },
                {
                    "name": "z",
                    "type": "space",
                    "unit": "micrometer"
                },
                {
                    "name": "y",
                    "type": "space",
                    "unit": "micrometer"
                },
                {
                    "name": "x",
                    "type": "space",
                    "unit": "micrometer"
                }
            ],
            "datasets": [
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                1,
                                6.5,
                                6.5,
                                6.5
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "0"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                1,
                                13.0,
                                13.0,
                                13.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "1"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                1,
                                26.0,
                                26.0,
                                26.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "2"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                1,
                                52.0,
                                52.0,
                                52.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "3"
                },
                {
                    "coordinateTransformations": [
                        {
                            "scale": [
                                1,
                                104.0,
                                104.0,
                                104.0
                            ],
                            "type": "scale"
                        }
                    ],
                    "path": "4"
                }
            ],
            "name": "/",
            "version": "0.4"
        }
    ]
}

[FIrgolitsch]

After some further testing, I've noticed that it seems to be a problem introduced in version 0.8.3. When I downgraded the package to 0.8.2 I just got the three nodes for the root group, the labels group and the mask group and no recursive paths.

[FIrgolitsch]

Hello, is there any update on this?

[will-moore]

I've noticed that builds are failing at https://github.com/ome/ome_zarr_test_suite/actions/workflows/main.yml with the same error.

Presumably if you remove the source attribute from the label, then this avoids the recursion?

        "source": {
            "image": "../../"
        },

I'll try to reproduce...

[joshmoore]

@will-moore : #342 might have been related.

[will-moore]

As described at #339 (comment) it seems that I didn't see the recursion issue with fsspec==2023.6.0 (as was pinned before #342) but I did see it with more recent fsspec==2023.12.1.

[will-moore]

I just tried with fsspec-2024.3.1 and I didn't see any recursion issue testing as at #339 with coinsdata.

[will-moore]

I'm not able to reproduce the recursion error testing with current master branch of ome-zarr:

$ pip freeze | grep fs
fsspec==2023.12.1
s3fs==2023.12.1

>>> url = "coinsdata"
>>> from ome_zarr.io import parse_url
>>> from ome_zarr.reader import Reader
>>> store = parse_url(url, mode="r").store
>>> reader = Reader(parse_url(url))
>>> nodes = list(reader())
>>> print(nodes)
[/Users/wmoore/Desktop/ZARR/data/coinsdata [zgroup], /Users/wmoore/Desktop/ZARR/data/coinsdata/labels [zgroup] (hidden), /Users/wmoore/Desktop/ZARR/data/coinsdata/labels/coins [zgroup] (hidden)]

Other related versions:

$ pip freeze | grep zarr
napari-ome-zarr==0.5.2
-e git+ssh://[email protected]/ome/ome-zarr-py.git@e33f736cb7591b7ab9cb2bdd8568dc0e01a853b8#egg=ome_zarr
-e git+ssh://[email protected]/ome/omero-cli-zarr.git@75ab185a184410aa408772e12315ba9cbf209b36#egg=omero_cli_zarr
-e git+ssh://[email protected]/ome/omero-web-zarr.git@38e91255c80bfb04b814ef37056b355d692c2c96#egg=omero_web_zarr
vizarr==0.0.1
zarr==2.14.2

If I uninstall and reinstall:

pip uninstall ome_zarr
pip install ome-zarr

Now I don't see the recursion error, but I get a very long list of nodes!

>>> print(len(nodes))
154

@FIrgolitsch can you test with the latest master branch of ome-zarr-py... That helps for me:

cd ome-zarr-py/
git checkout origin/master 
pip install -e .

[FIrgolitsch]

Thanks for the response! I finally had some time today to test things. I retested with the current version available on pip (0.8.3) and still got the recursion problem:

OSError: [Errno 63] File name too long: '/Users/<uname>/docker_data/LSFM/S19/S19.zarr/labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/atlas/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/../../labels/mask/0/.zarray'

Since it returns an OS error, the specific path length might be MacOS limiting the maximum path length.

I retried after with the version currently on the master branch (0.8.4.dev0), and everything seems to work correctly there. I'll be waiting for the full release of 0.8.4!

As for fsspec, I have version 2024.3.1 installed.

[joshmoore]

@will-moore and I believe that this is now (more?) fixed with 0.9.0