Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix shape in zarr.json in zarr3 streaming #8116

Merged
merged 9 commits into from
Oct 9, 2024
Merged

Fix shape in zarr.json in zarr3 streaming #8116

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6563e94
fix shape in zarr.json in zarr3 streaming
Oct 7, 2024
592ea8e
remove comment to make code DRYer
Oct 8, 2024
d9f86ed
Merge branch 'master' into fix-zarr3-streaming-shape
MichaelBuessemeyer Oct 8, 2024
ebdfbe5
use default mag when creating zarr3 header file in bucket sink
Oct 8, 2024
ffaf8a4
Merge branch 'master' into fix-zarr3-streaming-shape
fm3 Oct 8, 2024
dd43f3d
add changelog entry
Oct 8, 2024
e4d7fc9
Merge branch 'fix-zarr3-streaming-shape' of github.com:scalableminds/…
Oct 8, 2024
cd32695
Update CHANGELOG.unreleased.md
MichaelBuessemeyer Oct 8, 2024
4c16d65
Update CHANGELOG.unreleased.md
MichaelBuessemeyer Oct 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...tore/app/com/scalableminds/webknossos/datastore/controllers/ZarrStreamingController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,7 +342,7 @@ class ZarrStreamingController @Inject()(
"dataSource.notFound") ~> NOT_FOUND
magParsed <- Vec3Int.fromMagLiteral(mag, allowScalar = true) ?~> Messages("dataLayer.invalidMag", mag) ~> NOT_FOUND
_ <- bool2Fox(dataLayer.containsResolution(magParsed)) ?~> Messages("dataLayer.wrongMag", dataLayerName, mag) ~> NOT_FOUND
zarrHeader = Zarr3ArrayHeader.fromDataLayer(dataLayer)
zarrHeader = Zarr3ArrayHeader.fromDataLayer(dataLayer, magParsed)
} yield Ok(Json.toJson(zarrHeader))

def zArrayPrivateLink(token: Option[String],
Expand Down
12 changes: 10 additions & 2 deletions ...store/app/com/scalableminds/webknossos/datastore/datareaders/zarr3/Zarr3ArrayHeader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package com.scalableminds.webknossos.datastore.datareaders.zarr3

import com.scalableminds.util.geometry.Vec3Int
import com.scalableminds.util.tools.{BoxImplicits, JsonHelper}
import com.scalableminds.webknossos.datastore.datareaders.ArrayDataType.ArrayDataType
import com.scalableminds.webknossos.datastore.datareaders.ArrayOrder.ArrayOrder
Expand Down Expand Up @@ -251,13 +252,20 @@ object Zarr3ArrayHeader extends JsonImplicits {
)

}
def fromDataLayer(dataLayer: DataLayer): Zarr3ArrayHeader = {
def fromDataLayer(dataLayer: DataLayer, mag: Vec3Int): Zarr3ArrayHeader = {
val additionalAxes = reorderAdditionalAxes(dataLayer.additionalAxes.getOrElse(Seq.empty))
// TODOM: Make code together with ZarrHeader.scala DRYer
MichaelBuessemeyer marked this conversation as resolved.
Show resolved Hide resolved
val xyzBBounds = Array(
// Zarr can't handle data sets that don't start at 0, so we extend the shape to include "true" coords
(dataLayer.boundingBox.width + dataLayer.boundingBox.topLeft.x) / mag.x,
(dataLayer.boundingBox.height + dataLayer.boundingBox.topLeft.y) / mag.y,
(dataLayer.boundingBox.depth + dataLayer.boundingBox.topLeft.z) / mag.z
)
Zarr3ArrayHeader(
zarr_format = 3,
node_type = "array",
// channel, additional axes, XYZ
shape = Array(1) ++ additionalAxes.map(_.highestValue).toArray ++ dataLayer.boundingBox.bottomRight.toArray,
shape = Array(1) ++ additionalAxes.map(_.highestValue).toArray ++ xyzBBounds,
data_type = Left(dataLayer.elementClass.toString),
chunk_grid = Left(
ChunkGridSpecification(
Expand Down
2 changes: 1 addition & 1 deletion ...app/com/scalableminds/webknossos/tracingstore/tracings/volume/Zarr3BucketStreamSink.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ class Zarr3BucketStreamSink(val layer: VolumeTracingLayer, tracingHasFallbackLay
def apply(bucketStream: Iterator[(BucketPosition, Array[Byte])], mags: Seq[Vec3Int], voxelSize: Option[VoxelSize])(
implicit ec: ExecutionContext): Iterator[NamedStream] = {

val header = Zarr3ArrayHeader.fromDataLayer(layer)
val header = Zarr3ArrayHeader.fromDataLayer(layer, mags.head)
MichaelBuessemeyer marked this conversation as resolved.
Show resolved Hide resolved
bucketStream.flatMap {
case (bucket, data) =>
val skipBucket = if (tracingHasFallbackLayer) isAllZero(data) else isRevertedBucket(data)
Expand Down