Support other units than nm for datasets (#7783)

* WIP: adding unit setting to datasource settings

* WIP: backend: voxel size as object

* WIP: Adding unit to dataset scale [ci skip]
- make keep the base unit internally to nm
- use configured unit for 3d space / voxel space
- WIP: post fix all relevant function with Nm or Vx depending on their unit to make this as explicit as possible

* finish migrating to new datasetscale format [ci skip]

- current state: broken; nothing is rendered

* adjust nml schema to backend changes

* Fix initial dataset loading with scale in mircometers

- add a lot of debugging output for testing [ci skip]

* Fix td viewport camera init ( calulate in datasource unit not always in nm)

- [ci skip]
- Dont ignore far plane calculation

* undo accidental semantic change

* support conversion between larger and smaller units in 1d-3d
- Full support for uncommon units is missing for 2d and 3d

* format 1d length units in datasource unit to avoid precision loss

* WIP: Support formatting for 2d and 3d values without always converting to nm [ci skip]

* unify format function to work with multi dimensional values and not only in 1d && fix lots of tests [ci skip]

* finish writing more tests for format utils

* include all unit format in the datasource json validation in the frontend

* Add support to convert to cm

- WIP: Fix tests now that cm is supported

* Add tests for uncommon units [ci skip]

* rename "datasourceUnit" to "unit" everywhere

* WIP: remove complex tests and add example based tests [ci skip]

* add more format utils tests and fix some tests

* fix / remove introduced todos of this pr

* remove debugging console.logs

* fix datasets e2e test

* redo: use shorter decimals in scalebar

* WIP: voxel unit support backend

* fix vx distance calculation of shortest path between nodes

* improve previous vx length calculation fix

* use voxel size unit in more spots

* schema, evolutions

* snapshots

* rename LengthUnit to Unit in frontend

* further clean up of frontend code

* more code cleanup frontend

* update datasource json frontend validation

* add changelog entry

* log warning in case a mesh is loaded whose scale does not match the dataset's scale

* bump api version to 7, drop 1 and 2

* document that ad-hoc meshing uses voxel size factor only

* unused imports

* WIP: use ngff units during explore

* read ngff voxel size unit during explore

* implement backend pr feedback part 1

* update nml spec snapshot

* report unit in ngff streaming. change composeRequest to expect full voxel size with unit

* format

* unused import

* Fix uploading zipped tiff (should set needsConversion=true)

* changelog

* rephrase code comment

* WIP apply feedback

* Add unit to params of wkw conversion job for dataset upload

* apply pr feedback

* fix adhoc mesh computation

* rename datasetScale to voxelSize in frontend
And remove datasetScale/voxelSize uniform from shaders as it was unused

* power-of-two check on all mag dimensions

* adapt voxelSize to nanometer for worker for now

* use term REST API in changelog after all

* use long lengthUnit names as default format

* snapshots

* migrate frontend to use long unit names except for format utils

* rename UnitShortMap to LongUnitToShortUnitMap

* fix flycam reducer tests

---------

Co-authored-by: Florian M <[email protected]>
Co-authored-by: Florian M <[email protected]>

Author: MichaelBuessemeyer

CHANGELOG.unreleased.md

@@ -16,6 +16,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Added the option to set a default mapping for a dataset in the dataset view configuration. The default mapping is loaded when the dataset is opened and the user / url does not configure something else. [#7858](https://github.com/scalableminds/webknossos/pull/7858)
 - Uploading an annotation into a dataset that it was not created for now also works if the dataset is in a different organization. [#7816](https://github.com/scalableminds/webknossos/pull/7816)
 - When downloading + reuploading an annotation that is based on a segmentation layer with active mapping, that mapping is now still be selected after the reupload. [#7822](https://github.com/scalableminds/webknossos/pull/7822)
+- Added the ability to change the unit of the dataset voxel size to any supported unit of the [ome/ngff standard](https://github.com/ome/ngff/blob/39605eec64ceff481bb3a98f0adeaa330ab1ef26/latest/index.bs#L192). This allows users to upload and work with low-resolution datasets with a different base unit than nanometer. [#7783](https://github.com/scalableminds/webknossos/pull/7783)
 - In the Voxelytics workflow list, the name of the WEBKNOSSOS user who started the job is displayed. [#7794](https://github.com/scalableminds/webknossos/pull/7795)
 - Start an alignment job (aligns the section in a dataset) via the "AI Analysis" button. [#7820](https://github.com/scalableminds/webknossos/pull/7820)
 - Added additional validation for the animation job modal. Bounding boxes must be larger then zero. [#7883](https://github.com/scalableminds/webknossos/pull/7883)
@@ -37,6 +38,8 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Fixed that dataset composition did not work when selecting only one dataset for composition. [#7889](https://github.com/scalableminds/webknossos/pull/7889)
 
 ### Removed
+- REST API versions 1 and 2 are no longer supported. Current is 7.
+data
 - If the datasource-properties.json file for a dataset is missing or contains errors, WEBKNOSSOS no longer attempts to guess its contents from the raw data. Exploring remote datasets will still create the file. [#7697](https://github.com/scalableminds/webknossos/pull/7697)
 
 ### Breaking Changes

MIGRATIONS.unreleased.md

@@ -17,3 +17,4 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 - [114-ai-models.sql](conf/evolutions/114-ai-models.sql)
 - [115-annotation-locked-by-user.sql](conf/evolutions/115-annotation-locked-by-user.sql)
 - [116-drop-overtimemailinglist.sql](conf/evolutions/116-drop-overtimemailinglist.sql)
+- [117-voxel-size-unit.sql](conf/evolutions/117-voxel-size-unit.sql)

app/controllers/AnnotationController.scala

@@ -398,8 +398,8 @@ class AnnotationController @Inject()(
     allItems.grouped(batchSize).toList
   }
 
-  private def percent(done: Int, todo: Int) = {
-    val value = done.toDouble / todo.toDouble * 100
+  private def percent(done: Int, pending: Int) = {
+    val value = done.toDouble / pending.toDouble * 100
     f"$value%1.1f %%"
   }
 

app/controllers/AnnotationIOController.scala

@@ -424,7 +424,7 @@ class AnnotationIOController @Inject()(
           "temp",
           fetchedAnnotationLayers,
           Some(annotation),
-          dataset.scale,
+          dataset.voxelSize,
           None,
           organizationName,
           conf.Http.uri,
@@ -452,7 +452,7 @@ class AnnotationIOController @Inject()(
                                                 volumeVersion,
                                                 skipVolumeData,
                                                 volumeDataZipFormat,
-                                                dataset.scale)
+                                                dataset.voxelSize)
         } ?~> "annotation.download.fetchVolumeLayer.failed"
         fetchedSkeletonLayers: List[FetchedAnnotationLayer] <- Fox.serialCombined(annotation.skeletonAnnotationLayers) {
           skeletonAnnotationLayer =>
@@ -464,7 +464,7 @@ class AnnotationIOController @Inject()(
           name,
           fetchedSkeletonLayers ::: fetchedVolumeLayers,
           Some(annotation),
-          dataset.scale,
+          dataset.voxelSize,
           None,
           organizationName,
           conf.Http.uri,

app/controllers/JobController.scala

@@ -1,7 +1,7 @@
 package controllers
 
 import play.silhouette.api.Silhouette
-import com.scalableminds.util.geometry.Vec3Int
+import com.scalableminds.util.geometry.{BoundingBox, Vec3Double, Vec3Int}
 import com.scalableminds.util.accesscontext.GlobalAccessContext
 import com.scalableminds.util.tools.Fox
 import models.dataset.{DataStoreDAO, DatasetDAO, DatasetService}
@@ -19,7 +19,7 @@ import java.util.Date
 import javax.inject.Inject
 import scala.concurrent.ExecutionContext
 import com.scalableminds.util.enumeration.ExtendedEnumeration
-import com.scalableminds.util.geometry.BoundingBox
+import com.scalableminds.webknossos.datastore.models.{LengthUnit, VoxelSize}
 import models.team.PricingPlan
 
 object MovieResolutionSetting extends ExtendedEnumeration {
@@ -109,12 +109,18 @@ class JobController @Inject()(
   }
 
   // Note that the dataset has to be registered by reserveUpload via the datastore first.
-  def runConvertToWkwJob(organizationName: String, datasetName: String, scale: String): Action[AnyContent] =
+  def runConvertToWkwJob(organizationName: String,
+                         datasetName: String,
+                         scale: String,
+                         unit: Option[String]): Action[AnyContent] =
     sil.SecuredAction.async { implicit request =>
       log(Some(slackNotificationService.noticeFailedJobRequest)) {
         for {
           organization <- organizationDAO.findOneByName(organizationName) ?~> Messages("organization.notFound",
                                                                                        organizationName)
+          voxelSizeFactor <- Vec3Double.fromUriLiteral(scale).toFox
+          voxelSizeUnit <- Fox.runOptional(unit)(u => LengthUnit.fromString(u).toFox)
+          voxelSize = VoxelSize.fromFactorAndUnitWithDefault(voxelSizeFactor, voxelSizeUnit)
           _ <- bool2Fox(request.identity._organization == organization._id) ~> FORBIDDEN
           dataset <- datasetDAO.findOneByNameAndOrganization(datasetName, organization._id) ?~> Messages(
             "dataset.notFound",
@@ -124,7 +130,7 @@ class JobController @Inject()(
             "organization_name" -> organizationName,
             "organization_display_name" -> organization.displayName,
             "dataset_name" -> datasetName,
-            "scale" -> scale
+            "scale" -> voxelSize.toNanometer.toUriLiteral
           )
           job <- jobService.submitJob(command, commandArgs, request.identity, dataset._dataStore) ?~> "job.couldNotRunCubing"
           js <- jobService.publicWrites(job)