Add zarr3 streaming v0

[MichaelBuessemeyer]

URL of deployed dev instance (used for testing):

• https://addzarrstreamingv.webknossos.xyz

Steps to test:

• Check the zarr.jsons:
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/1/zarr.json
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/zarr.json
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/segmentation/1/zarr.json
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/segmentation/zarr.json
• Check the listing routes:
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/datasource-properties.json
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/segmentation/
  • https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/2-2-1/
• Check whether an annotation is listed properly:
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/color
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/segmentation
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/datasource-properties.json
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/segmentation/4-4-1/
  • https://addzarrstreamingv.webknossos.xyz/data/annotations/zarr3_experimental/S53FgwKKnbWczqUq/color/4-4-1/
• The following "remote-local" dataset should render for mags 1&2&4 https://addzarrstreamingv.webknossos.xyz/datasets/sample_organization/test-zarr3-viewing/view#3573,3568,1024,0,2.358
• Same for the segmentation layer: https://addzarrstreamingv.webknossos.xyz/datasets/sample_organization/Zarr3-Segmentation-Streaming-Test/view#3584,3584,1024,0,1 (only mag 1 is imported)
• And here I streamed an annotation (all mags): https://addzarrstreamingv.webknossos.xyz/datasets/sample_organization/zarr3-annotation-streaming#3584,3584,1024,0,9.85
• All of them should work :)

Issues:

• fixes Add zarr3 streaming #7933

---

(Please delete unneeded items, merge only when none are left open)

• Updated changelog
• Updated documentation if applicable -> maybe?
• Adapted wk-libs python client if relevant API parts change -> maybe add new issue there?
• Considered common edge cases -> @fm3 do you see any cases I did not include in the tests?
• Needs datastore update after deployment

[joshmoore]

This is great! 🙌🏽 When there's a URL that can be tested, please let me know.

[fm3]

Looking pretty good already :) I added a couple comments. Will do some testing next

[fm3]

I’d say the v2 stuff should be in separate file. If some code is shared between v1 and v2, maybe add a third file for the shared stuff.

Also, is v2 the right version number? I thought they were like at 0.5?

Also, please add some links here to the spec documents.

[MichaelBuessemeyer]

I put the shared classes into SharedNgffMetadataAttributes.scala and named the new NgffMetadata version "NgffMetadataV0_5". Should I rename the old version to NgffMetadataV0_4 accordingly?

[fm3]

@normanrz do you think we should also already add a datasource-properties.json route and directory listing routes? Will some clients need that?

[normanrz]

@normanrz do you think we should also already add a datasource-properties.json route and directory listing routes? Will some clients need that?

Yes, I think that would be useful.

[MichaelBuessemeyer]

This is the how the view of a existing color layer looks like when listing an annotation. It says that the layer belongs to a organization and certain dataset. It this alright or should the title be changed? (Bc it kinda leeks the organization & dataset name (although this is already likely leaked via some properties sent to the client)

[MichaelBuessemeyer]

Yes, I think that would be useful.

It should be done. See the added routes in the testing guide.

[normanrz]

I think the metadata is not correct yet. For example, https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/zarr.json should be similar to this:

{
  "zarr_format": 3,
  "node_type": "group",
  "attributes": {
    "ome": {
      "version": "0.5-dev2",
      "multiscales": [{ ... }]
     }
   }
}

You can validate with https://deploy-preview-36--ome-ngff-validator.netlify.app/?source=https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color

[MichaelBuessemeyer]

Thanks for poiting this out @normanrz
the newest version (not yet deployed) should incorporate the schema fix.

But the validator seems broken to me. It shows some warnings about not being able to find schema definitions.

A auick look showed that the referenced schema json for the version is no longer available under the link linked in the schema definition of https://raw.githubusercontent.com/normanrz/ngff/spec-rfc2/latest/schemas/image.schema

Thus, I just used a different json validator and with the assumption that "0.5-dev2" is the correct version string, the schema of the https://addzarrstreamingv.webknossos.xyz/data/zarr3_experimental/sample_organization/l4_sample/color/zarr.json should now be valid :)

[normanrz]

Now it is green!

[MichaelBuessemeyer]

Now it is green!

Awesome 🎉 🚀

[normanrz]

This looks very nice! Thanks.

I just pinged you again on this one comment by @fm3.

[normanrz]