ND Datasets (#7136)

* add UI for 4th dimension and mock 4D data when requesting buckets * adapt bucket address space to incorporate higher dimension * clean up imports * add slider to Q input field; tune mocked 4D data retrieval * change max value of number slider * make skeletons compatible with higher dimensions * fix tests; improve typing; create CubeEntry instances lazily * fix linting * use n-dimensional generalization instead of fourth coordinate everywhere * fix resampling of buckets when annotating volume data in higher dimensions * add additionalCoordinates parameter to several other places * hardcode additionalCoords initialization * tackle more todo comments * adapt additionalCoordinates input UI to arbitrary dimensions; use two additional coordinates as hardcoded ones * dynamically generate shaders for nodes and edges to deal with arbitrary ND data * make edges interpolate from alpha 1 to 0 if edge partner is on other dimension * Add additional coordinates for 4+D requests * begin integration of backend additionalCoordinate interface * change frontend format for additional coordinates from number-array to array of objects (analoguous to backend) * Add additional coords to db * Fix naming * remove hardcoded additional coordinates related to DataCube class * properly initialize flycam with additional coordinates * make that arrow keys in vector input increment/decrement element where cursor is at * fix onChange in additional-coordinate-input * use bounds of additional unified coordinates in slider * don't hardcode additional coordinate length in skeleton.ts; guard against underspecified additional coords when setting them in the flycam * fix rendering of skeletons in higher dimensions * export/import additional coordinates in NML (front-end import/export only) * ensure api.getDataValue receives additionalCoordinates parameter everywhere * also fallback to stores additionalCoordinates when they are not provided to api * remove second-last todo comment * Add additional coords as paramters for volume actions * Add editPositionAdditionalCoordinates to tracings * Implement n-dimensional morton code * Start implementation of volume tracing n-d fossildb access * Add additional coordinates to skeleton nodes * fix runtime exception when no additional coordinates exist; fix initialization of flycam * remove todo comment * rename additionalCoords to additionalCoordinates everywhere * fix typing * fix delayed update * hotfix missing additionalCoordinates for volume layer * pass additionalCoordinates everywhere SegmentItem- and MeshItem-related around * clicking on a segment list should activate its additionalCoordinates if they exist * fix linting * fix cyclic dependencies * rename to someAdditionalCoordinates * also store additional coordinates in json-encoded url state * show dimensions as addonBefore in AdditionalCoordinates input box * clean up * also maintain editPositionAdditionalCoordinates in frontend * Add additional coordinates to tracings * Read additional coordinate bounds from zarr header * Store additional coordinate as a vec2int in proto * store additionalCoordinates for volume and skeleton layers * Increment evolution number * fix cube.spec and volume_annotation_sampling.spec * fix wkstore_adapter spec * fix volumetracing_saga.spec * fix nml.spec.ts * fix url manager spec * fix skeleton spec by ensuring singletons are set up * fix saga_integration.spec * Remove nd morton code, use 3d morton code always * update documentation package to fix parser crash * Apply suggestions from code review Co-authored-by: Daniel <[email protected]> * DRY assertion * fix camel case * undo changes in v2 api * avoid several THREE.BufferAttribute casts * add comment about additionalCoordinates = null in bucket pickers * add emit-rerender comment * use bounds[0] as a fallback value when additional Coordinates are missing in an update action * warn when meshing >3d segmentations * fix initialization of additional coordinates from url state * change additional coordinates format in xml to sth like additionalCoordinate-t=10 * remove unneeded functions in nml_helpers * fix scala compilation on CI * format * format backend * test nml import/export with additional coordinates * add test for setting additional coords in flycam * Some sort of sorting * Increment evolution number * Add comment in datasetarray, remove todos * fix linting * Update NML import/export for additional coordinates * Update snapshots * Sort additional coordinates when building bucket key * Update snapshots again * fix yarn.lock * Fix volume tracing bucket key not including additional coordinates because the datalayer did not have them * fix styling of additional-coordinate UI in light theme * also serialize additional coordinates for editPosition and layer itself into NML in frontend * show additional coords in context menu and status bar * Add test for bucket keys with additional coordinates (and nml) * Rename additionalCorrdinate to definition * Assert additional coordinates are the same when merging * Add additional coordinate tests * Update changelog * Remove TODOs * add warning for ai-quick-select for n-d datasets * enforce valid values for additional coordinates in UI * fix tests * fix nml test and add a new one for additional coords * Lint backend * remove t from axis order, allow mag paths relative to dataset dir * improve and fix snapshot tests * prevent user from exporting/downloading layers with nd data * Fix simple refactorings * Rename additionalCoordinateDefinition to additionalAxis * Rename additionalCoordinateRequest to additionalCoordinate * Continue renaming * rename additionalCoordinateWithBounds to AdditionalAxis and adapt to naming change in backend * Do not require migration of bucket keys * update snapshots * Lint backend * Rename Segment->additionalCoordinates to anchorPositionadd.. * Remove unused method * Remove exception throwing * Fix renaming issues * Fix upload of annotations --------- Co-authored-by: frcroth <[email protected]> Co-authored-by: frcroth <[email protected]> Co-authored-by: Daniel <[email protected]> Co-authored-by: Florian M <[email protected]> Co-authored-by: Florian M <[email protected]>
scalableminds · Sep 5, 2023 · 978dde5 · 978dde5
1 parent 6c05922
commit 978dde5
Show file tree

Hide file tree

Showing 175 changed files with 4,492 additions and 2,792 deletions.
diff --git a/CHANGELOG.unreleased.md b/CHANGELOG.unreleased.md
@@ -11,6 +11,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 [Commits](https://github.com/scalableminds/webknossos/compare/23.09.0...HEAD)
 
 ### Added
+- Datasets and annotations can now be more than 3-dimensional, using additional coordinates. [#7136](https://github.com/scalableminds/webknossos/pull/7136)
 - Added disabled drag handles to volume and skeleton layers for visual consistency. These layer cannot be dragged or reordered. [#7295](https://github.com/scalableminds/webknossos/pull/7295)
 - Dataset thumbnails for grayscale layers can now be colored using the value in the view configuration. [#7255](https://github.com/scalableminds/webknossos/pull/7255)
 - OpenID Connect authorization is now compatible with Providers that send the user information in an id_token. [#7294](https://github.com/scalableminds/webknossos/pull/7294)

diff --git a/MIGRATIONS.unreleased.md b/MIGRATIONS.unreleased.md
@@ -9,3 +9,5 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 [Commits](https://github.com/scalableminds/webknossos/compare/23.09.0...HEAD)
 
 ### Postgres Evolutions:
+- [108-additional-coordinates](conf/evolutions/108-additional-coordinates.sql)
+
diff --git a/app/controllers/AnnotationController.scala b/app/controllers/AnnotationController.scala
@@ -8,6 +8,7 @@ import com.scalableminds.util.time.Instant
 import com.scalableminds.util.tools.{Fox, FoxImplicits}
 import com.scalableminds.webknossos.datastore.models.annotation.AnnotationLayerType.AnnotationLayerType
 import com.scalableminds.webknossos.datastore.models.annotation.{AnnotationLayer, AnnotationLayerType}
+import com.scalableminds.webknossos.datastore.models.datasource.AdditionalAxis
 import com.scalableminds.webknossos.tracingstore.tracings.volume.ResolutionRestrictions
 import com.scalableminds.webknossos.tracingstore.tracings.{TracingIds, TracingType}
 import io.swagger.annotations._
@@ -39,7 +40,8 @@ case class AnnotationLayerParameters(typ: AnnotationLayerType,
                                      autoFallbackLayer: Boolean = false,
                                      mappingName: Option[String] = None,
                                      resolutionRestrictions: Option[ResolutionRestrictions],
-                                     name: Option[String])
+                                     name: Option[String],
+                                     additionalAxes: Option[Seq[AdditionalAxis]])
 object AnnotationLayerParameters {
   implicit val jsonFormat: OFormat[AnnotationLayerParameters] =
     Json.using[WithDefaultValues].format[AnnotationLayerParameters]

diff --git a/app/controllers/LegacyApiController.scala b/app/controllers/LegacyApiController.scala
@@ -431,7 +431,8 @@ class LegacyApiController @Inject()(annotationController: AnnotationController,
             autoFallbackLayer = false,
             None,
             request.body.resolutionRestrictions,
-            name = Some(AnnotationLayer.defaultSkeletonLayerName)
+            name = Some(AnnotationLayer.defaultSkeletonLayerName),
+            None
           ))
     val volumeParameters =
       if (request.body.typ == "skeleton") None
@@ -443,7 +444,8 @@ class LegacyApiController @Inject()(annotationController: AnnotationController,
             autoFallbackLayer = false,
             None,
             request.body.resolutionRestrictions,
-            name = Some(AnnotationLayer.defaultVolumeLayerName)
+            name = Some(AnnotationLayer.defaultVolumeLayerName),
+            None
           ))
     List(skeletonParameters, volumeParameters).flatten
   }

diff --git a/app/models/annotation/AnnotationService.scala b/app/models/annotation/AnnotationService.scala
@@ -11,6 +11,7 @@ import com.scalableminds.util.tools.{BoxImplicits, Fox, FoxImplicits, TextUtils}
 import com.scalableminds.webknossos.datastore.SkeletonTracing._
 import com.scalableminds.webknossos.datastore.VolumeTracing.{VolumeTracing, VolumeTracingOpt, VolumeTracings}
 import com.scalableminds.webknossos.datastore.geometry.{
+  AdditionalCoordinateProto,
   ColorProto,
   NamedBoundingBoxProto,
   Vec3DoubleProto,
@@ -24,6 +25,7 @@ import com.scalableminds.webknossos.datastore.models.annotation.{
   FetchedAnnotationLayer
 }
 import com.scalableminds.webknossos.datastore.models.datasource.{
+  AdditionalAxis,
   ElementClass,
   DataSourceLike => DataSource,
   SegmentationLayerLike => SegmentationLayer
@@ -77,7 +79,8 @@ case class RedundantTracingProperties(
     editPosition: Vec3IntProto,
     editRotation: Vec3DoubleProto,
     zoomLevel: Double,
-    userBoundingBoxes: Seq[NamedBoundingBoxProto]
+    userBoundingBoxes: Seq[NamedBoundingBoxProto],
+    editPositionAdditionalCoordinates: Seq[AdditionalCoordinateProto],
 )
 
 class AnnotationService @Inject()(
@@ -144,6 +147,8 @@ class AnnotationService @Inject()(
   ): Fox[VolumeTracing] = {
     val resolutions = VolumeTracingDownsampling.magsForVolumeTracing(dataSource, fallbackLayer)
     val resolutionsRestricted = resolutionRestrictions.filterAllowed(resolutions)
+    val additionalCoordinates =
+      fallbackLayer.map(_.additionalAxes).getOrElse(dataSource.additionalAxesUnion)
     for {
       _ <- bool2Fox(resolutionsRestricted.nonEmpty) ?~> "annotation.volume.resolutionRestrictionsTooTight"
     } yield
@@ -163,7 +168,8 @@ class AnnotationService @Inject()(
         organizationName = Some(datasetOrganizationName),
         mappingName = mappingName,
         resolutions = resolutionsRestricted.map(vec3IntToProto),
-        hasSegmentIndex = Some(fallbackLayer.isEmpty)
+        hasSegmentIndex = Some(fallbackLayer.isEmpty),
+        additionalAxes = AdditionalAxis.toProto(additionalCoordinates)
       )
   }
 
@@ -240,13 +246,15 @@ class AnnotationService @Inject()(
               dataSetName = dataSet.name,
               editPosition = dataSource.center,
               organizationName = Some(datasetOrganizationName),
+              additionalAxes = AdditionalAxis.toProto(dataSource.additionalAxesUnion)
             )
             val skeletonAdapted = oldPrecedenceLayerProperties.map { p =>
               skeleton.copy(
                 editPosition = p.editPosition,
                 editRotation = p.editRotation,
                 zoomLevel = p.zoomLevel,
-                userBoundingBoxes = p.userBoundingBoxes
+                userBoundingBoxes = p.userBoundingBoxes,
+                editPositionAdditionalCoordinates = p.editPositionAdditionalCoordinates
               )
             }.getOrElse(skeleton)
             for {
@@ -273,7 +281,8 @@ class AnnotationService @Inject()(
                   editPosition = p.editPosition,
                   editRotation = p.editRotation,
                   zoomLevel = p.zoomLevel,
-                  userBoundingBoxes = p.userBoundingBoxes
+                  userBoundingBoxes = p.userBoundingBoxes,
+                  editPositionAdditionalCoordinates = p.editPositionAdditionalCoordinates
                 )
               }.getOrElse(volumeTracing)
               volumeTracingId <- client.saveVolumeTracing(volumeTracingAdapted)
@@ -305,15 +314,17 @@ class AnnotationService @Inject()(
             s.editRotation,
             s.zoomLevel,
             s.userBoundingBoxes ++ s.userBoundingBox.map(
-              com.scalableminds.webknossos.datastore.geometry.NamedBoundingBoxProto(0, None, None, None, _))
+              com.scalableminds.webknossos.datastore.geometry.NamedBoundingBoxProto(0, None, None, None, _)),
+            s.editPositionAdditionalCoordinates
           )
         case Right(v) =>
           RedundantTracingProperties(
             v.editPosition,
             v.editRotation,
             v.zoomLevel,
             v.userBoundingBoxes ++ v.userBoundingBox.map(
-              com.scalableminds.webknossos.datastore.geometry.NamedBoundingBoxProto(0, None, None, None, _))
+              com.scalableminds.webknossos.datastore.geometry.NamedBoundingBoxProto(0, None, None, None, _)),
+            v.editPositionAdditionalCoordinates
           )
       }
 
@@ -384,7 +395,8 @@ class AnnotationService @Inject()(
         autoFallbackLayer = false,
         None,
         Some(ResolutionRestrictions.empty),
-        Some(AnnotationLayer.defaultNameForType(newAnnotationLayerType))
+        Some(AnnotationLayer.defaultNameForType(newAnnotationLayerType)),
+        None
       )
       _ <- addAnnotationLayer(annotation, organizationName, newAnnotationLayerParameters) ?~> "makeHybrid.createTracings.failed"
     } yield ()

diff --git a/app/models/annotation/nml/NmlParser.scala b/app/models/annotation/nml/NmlParser.scala
@@ -4,7 +4,14 @@ import com.scalableminds.util.geometry.{BoundingBox, Vec3Double, Vec3Int}
 import com.scalableminds.util.tools.ExtendedTypes.{ExtendedDouble, ExtendedString}
 import com.scalableminds.webknossos.datastore.SkeletonTracing._
 import com.scalableminds.webknossos.datastore.VolumeTracing.{Segment, SegmentGroup, VolumeTracing}
-import com.scalableminds.webknossos.datastore.geometry.{ColorProto, NamedBoundingBoxProto, Vec3IntProto}
+import com.scalableminds.webknossos.datastore.geometry.{
+  AdditionalAxisProto,
+  AdditionalCoordinateProto,
+  ColorProto,
+  NamedBoundingBoxProto,
+  Vec2IntProto,
+  Vec3IntProto
+}
 import com.scalableminds.webknossos.datastore.helpers.{NodeDefaults, ProtoGeometryImplicits, SkeletonTracingDefaults}
 import com.scalableminds.webknossos.datastore.models.datasource.ElementClass
 import com.scalableminds.webknossos.tracingstore.tracings.ColorGenerator
@@ -14,12 +21,12 @@ import com.scalableminds.webknossos.tracingstore.tracings.volume.VolumeSegmentIn
 import com.typesafe.scalalogging.LazyLogging
 import models.annotation.UploadedVolumeLayer
 import net.liftweb.common.Box._
-import net.liftweb.common.{Box, Empty, Failure}
+import net.liftweb.common.{Box, Empty, Failure, Full}
 import play.api.i18n.{Messages, MessagesProvider}
 
 import java.io.InputStream
 import scala.collection.{immutable, mutable}
-import scala.xml.{NodeSeq, XML, Node => XMLNode}
+import scala.xml.{Attribute, NodeSeq, XML, Node => XMLNode}
 
 object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGenerator {
 
@@ -51,15 +58,16 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
         treesSplit = treesAndGroupsAfterSplitting._1
         treeGroupsAfterSplit = treesAndGroupsAfterSplitting._2
         _ <- TreeValidator.validateTrees(treesSplit, treeGroupsAfterSplit, branchPoints, comments)
+        additionalAxisProtos <- parseAdditionalAxes(parameters \ "additionalAxes")
       } yield {
         val dataSetName = overwritingDataSetName.getOrElse(parseDataSetName(parameters \ "experiment"))
         val description = parseDescription(parameters \ "experiment")
         val wkUrl = parseWkUrl(parameters \ "experiment")
         val organizationName =
           if (overwritingDataSetName.isDefined) None else parseOrganizationName(parameters \ "experiment")
         val activeNodeId = parseActiveNode(parameters \ "activeNode")
-        val editPosition =
-          parseEditPosition(parameters \ "editPosition").getOrElse(SkeletonTracingDefaults.editPosition)
+        val (editPosition, editPositionAdditionalCoordinates) =
+          parseEditPosition(parameters \ "editPosition").getOrElse((SkeletonTracingDefaults.editPosition, Seq()))
         val editRotation =
           parseEditRotation(parameters \ "editRotation").getOrElse(SkeletonTracingDefaults.editRotation)
         val zoomLevel = parseZoomLevel(parameters \ "zoomLevel").getOrElse(SkeletonTracingDefaults.zoomLevel)
@@ -92,7 +100,9 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
                 organizationName,
                 segments = v.segments,
                 segmentGroups = v.segmentGroups,
-                hasSegmentIndex = VolumeSegmentIndexService.canHaveSegmentIndex(v.fallbackLayerName)
+                hasSegmentIndex = VolumeSegmentIndexService.canHaveSegmentIndex(v.fallbackLayerName),
+                editPositionAdditionalCoordinates = editPositionAdditionalCoordinates,
+                additionalAxes = additionalAxisProtos
               ),
               basePath.getOrElse("") + v.dataZipPath,
               v.name,
@@ -116,7 +126,9 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
                 None,
                 treeGroupsAfterSplit,
                 userBoundingBoxes,
-                organizationName
+                organizationName,
+                editPositionAdditionalCoordinates,
+                additionalAxes = additionalAxisProtos
               )
             )
 
@@ -189,13 +201,15 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
         case (Some(x), Some(y), Some(z)) => Some(Vec3IntProto(x.toInt, y.toInt, z.toInt))
         case _                           => None
       }
+      val anchorPositionAdditionalCoordinates = parseAdditionalCoordinateValues(node)
       Segment(
         segmentId = getSingleAttribute(node, "id").toLong,
         anchorPosition = anchorPosition,
         name = getSingleAttributeOpt(node, "name"),
         creationTime = getSingleAttributeOpt(node, "created").flatMap(_.toLongOpt),
         color = parseColorOpt(node),
-        groupId = getSingleAttribute(node, "groupId").toIntOpt
+        groupId = getSingleAttribute(node, "groupId").toIntOpt,
+        anchorPositionAdditionalCoordinates = anchorPositionAdditionalCoordinates
       )
     })
 
@@ -248,6 +262,33 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
       depth <- getSingleAttribute(node, "depth").toIntOpt
     } yield BoundingBox(Vec3Int(topLeftX, topLeftY, topLeftZ), width, height, depth)
 
+  private def parseAdditionalAxes(nodes: NodeSeq)(implicit m: MessagesProvider) = {
+    val additionalAxes = nodes.headOption.map(
+      _.child.flatMap(
+        additionalAxisNode => {
+          for {
+            name <- getSingleAttributeOpt(additionalAxisNode, "name")
+            indexStr <- getSingleAttributeOpt(additionalAxisNode, "index")
+            index <- indexStr.toIntOpt
+            minStr <- getSingleAttributeOpt(additionalAxisNode, "min")
+            min <- minStr.toIntOpt
+            maxStr <- getSingleAttributeOpt(additionalAxisNode, "max")
+            max <- maxStr.toIntOpt
+          } yield new AdditionalAxisProto(name, index, Vec2IntProto(min, max))
+        }
+      )
+    )
+    additionalAxes match {
+      case Some(axes) =>
+        if (axes.map(_.name).distinct.size == axes.size) {
+          Full(axes)
+        } else {
+          Failure(Messages("nml.additionalCoordinates.notUnique"))
+        }
+      case None => Full(Seq())
+    }
+  }
+
   private def parseDataSetName(nodes: NodeSeq): String =
     nodes.headOption.map(node => getSingleAttribute(node, "name")).getOrElse("")
 
@@ -266,8 +307,12 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
   private def parseTime(nodes: NodeSeq): Long =
     nodes.headOption.flatMap(node => getSingleAttribute(node, "ms").toLongOpt).getOrElse(DEFAULT_TIME)
 
-  private def parseEditPosition(nodes: NodeSeq): Option[Vec3Int] =
-    nodes.headOption.flatMap(parseVec3Int)
+  private def parseEditPosition(nodes: NodeSeq): Option[(Vec3Int, Seq[AdditionalCoordinateProto])] =
+    nodes.headOption.flatMap(n => {
+      val xyz = parseVec3Int(n)
+      val additionalCoordinates = parseAdditionalCoordinateValues(n)
+      xyz.map(value => (value, additionalCoordinates))
+    })
 
   private def parseEditRotation(nodes: NodeSeq): Option[Vec3Double] =
     nodes.headOption.flatMap(parseRotationForParams)
@@ -451,6 +496,7 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
     for {
       id <- nodeIdText.toIntOpt ?~ Messages("nml.node.id.invalid", "", nodeIdText)
       radius = getSingleAttribute(node, "radius").toFloatOpt.getOrElse(NodeDefaults.radius)
+      additionalCoordinates = parseAdditionalCoordinateValues(node)
       position <- parseVec3Int(node) ?~ Messages("nml.node.attribute.invalid", "position", id)
     } yield {
       val viewport = parseViewport(node)
@@ -459,8 +505,33 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
       val bitDepth = parseBitDepth(node)
       val interpolation = parseInterpolation(node)
       val rotation = parseRotationForNode(node).getOrElse(NodeDefaults.rotation)
-      Node(id, position, rotation, radius, viewport, resolution, bitDepth, interpolation, timestamp)
+      Node(id,
+           position,
+           rotation,
+           radius,
+           viewport,
+           resolution,
+           bitDepth,
+           interpolation,
+           timestamp,
+           additionalCoordinates)
     }
   }
 
+  private def parseAdditionalCoordinateValues(node: XMLNode): Seq[AdditionalCoordinateProto] = {
+    val regex = "additionalCoordinate-(\\w)".r("name")
+    node.attributes.flatMap {
+      case attribute: Attribute => {
+        if (attribute.key.startsWith("additionalCoordinate")) {
+          Some(
+            new AdditionalCoordinateProto(regex.findAllIn(attribute.key).group("name"),
+                                          attribute.value.toString().toInt))
+        } else {
+          None
+        }
+      }
+      case _ => None
+    }.toSeq
+  }
+
 }
Original file line number	Diff line number	Diff line change
Expand Up		@@ -9,3 +9,5 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
		[Commits](https://github.com/scalableminds/webknossos/compare/23.09.0...HEAD)

		### Postgres Evolutions:
		- [108-additional-coordinates](conf/evolutions/108-additional-coordinates.sql)