Read Zarr Datasets

[fm3]

Adds a first implementation of reading Zarr-encoded datasets.

• new jzarr package (based on https://github.com/bcdev/jzarr ) but heavily edited (converted to scala, refactored a bit, removed writing-related code, added uint64, removed some checks that required too many permissions on the underlying storage)
• new classes ZarrDataLayer, ZarrMag, ZarrBucketProvider to handle data loading in a way that is compatible with existing abstractions
• The wkw-specific shard-handle cache is not used, instead there is a chunk cache inside of the ZarrArray class
• Datasets can be local or remote via https (with optional basicauth) or s3
• new HttpsFileSystemProvider (partially) implements the java standrard FileSystemProvider interface for https so that jzarr can use it along s3
• new object FileSystemHolder stores FileSystems as created by the FileSystemProviders

URL of deployed dev instance (used for testing):

• https://zarr.webknossos.xyz

TODO

• open zarr images locally
• open zarr images with s3
• open zarr images with https
• use FileSystemProvider abstraction
• Take uri and credentials from datasource-properties.json
• fix jzarr existance checks problems with s3
• convert data to bytes depending on dtype
• basic auth for https
• automatic axes padding
• mags
• Replace infinite chunk cache by LRU cache
• cache clear for array handles (including chunk data)
• how to get blosc lib to production?
• finalize datasource-properties format, coordinate with Initial Zarr support  webknossos-libs#627
• convert remaining java classes to scala, clean up
• avoid sbt compile loop (get rid of java sources?)
• ensure upload (backend-side) works with zarr data
• uint64
• isotropic mag path should be “1” rather than ”1-1-1”
• add 64 MB chunk size limit
• do not pass exceptions (replace Fulls by tryos in ZarrBucketProvider) (can logging be retained?)
• assertions for too-big chunks etc?
• fix axis order
• write follow-up issues from list below:

Follow-Up

• technical clean-up, perf & co Zarr backend follow-up: tech #6119
• multi-layers/ngff, expose as first-class feature to users Zarr backend followup: OME-NGFF, user-side improvements #6120

Steps to test:

• Example datasource-properties.json for remote sample_l4 in zarr format:

  "id" : {
    "name" : "test.zarr",
    "team" : "sample_organization"
  },
  "dataLayers" : [
    {
      "name" : "0",
      "category" : "color",
      "boundingBox" : {
        "topLeft" : [ 0, 0, 0 ],
        "width" : 1024,
        "height": 1024,
        "depth" : 1024
      },
      "elementClass" : "uint8",
      "dataFormat" : "zarr",
      "mags": [
        {
          "mag": [1, 1, 1],
          "path": "https://…/l4_sample_zarr/color.zarr/0/",
          "credentials": {
            "user": "…",
            "password": "…"
          }
        },
        {
          "mag": [2, 2, 1],
          "path": "https://…/l4_sample_zarr/color.zarr/1/",
          "credentials": {
            "user": "…",
            "password": "…"
          }
        }
      ]
    }
  ],
  "scale" : [
    11.239999771118164,
    11.239999771118164,
    28
  ]
}

Issues:

• fixes Support Reading Zarr Images #5997

---

• Updated (unreleased) changelog
• Needs datastore update after deployment
• Ready for review

[jstriebel]

❤️

[jstriebel]