Allow custom colors for volume annotation segments (#6372)

* [WIP] allow custom colors for volume annotation segments

* pretty

* include color in segment update actions

* fix compile errors

* backend tests

* set color to none if no color was included in the update action

* Custom segment colors with cuckoo hashing (#6417)

* resurrect cuckoo table implementation

* adapt CuckooTable to entries consisting of (number, vec3) and finish proof-of-concept for writing and reading that hash table for segment colors

* fall back to old color generation if custom color could not be looked up

* add UI to select segment color

* correctly save segment color to back-end

* hook up custom segment colors from store with texture

* improve performance of cuckoo updates

* fix isDisabled prop for interpolation button

* convert usage of vec4 color to vec3

* remove type property for uniforms since it's not needed anymore since three r76

* adapt tests to new cuckootable interface

* clean up a bit

* fix cuckoo tests by using broader value range for tests and using uint32array instead of float

* try to get uint32 texture working (int texture only works when defined as byte, but that will clip values to 8 bit)

* dirty proof of concept for uint32 texture

* restore custom color rendering

* fix tests

* DRY mocks in tests

* more clean up

* implement unset for cuckoo table to allow resetting a segment color

* extend tests a bit

* don't hardcode values for cuckoo shader code

* fix linting

* also show segment patterns for custom segment colors

* show correct colors (even when mapped) in segment list

* also use correct color for mesh and synchronize the mesh with the segment color

* fix linting

* clean up some typing

* make custom segment colors compatible with multiple segmentation layers

* fix linting

* fix segment color icon for non-tracing layers

* don't allow changing colors for segments of non-tracing segmentation layer

* remove unused code for low-level texture setup

* remove logging from cuckoo table implementation

* clean up cuckoo_table.spec.ts

* fix React does not recognize the  prop on a DOM element. warning when opening context menu

* don't rely on random IDs in merger mode

* dont use the term color in the merger mode implementation as it is incorrect and confusing

* remove shuffleColorOfCurrentTree functionality for merger mode (let's wait for user feedback to see whether this is really needed)

* also reset stored representative when deleting last node of tree in merger mode

* clean up some mapping related code

* allow changing segment colors for non-tracing segmentation layers (necessary for json mappings); the colors won't be saved, though

* use custom segment color feature when loading a mapping which defines hue values

* clean up typing for withMappingActivationConfirmation

* remove old custom color implementation for JSON mappings

* remove attempt to integrate ACE editor

* move the cuckoo table maintenance into the LayerRenderingManager to avoid redundant updates and ensure diffing with the correct segments

* fix linting

* fix that shader won't render when all volume layers are invisible (but at least one exists)

* don't fill int texture with 255 by default

* revert temporary changes in test.sh

* skip shader tests which try to parse glsl 3000 syntax (and fail)

* fix some tests by having isosurface-related sagas wait for the scene controller to be initialized (which doesn't happen in some tests); therefore fixing null access to the scene controller

* skip mapping-related texture requirement test

* fix that seeds were not initialized correctly when no segmentation is visible initially

* explain why some glsl tests are skipped

* remove isMappingSupported distinction and incorporate custom colors texture into rendering logic

* fix skipped test in rendering logic spec

* implement compaction for update segment UpdateActions

* explain texture format etc better

* clean up test

* make getSegmentsForLayer non-nullable

* fix shader tests by avoiding hex uint notation

* explain isInt workaround

* remove one window mock; enforce correct type of other window mock; remove lots of @ts errors

* explain disabled logic

* fix linting

* fix linting

* initialize color picker properly even when a custom segment color was not defined yet

* don't share material for chunks of mesh, since the opacity has to be controlled individually for the loading animation

* fix linting

* fix crash when anchor position of segment is null

* try to debug buggy restore

* fix bug with repeated sets that caused a key to be appear multiple times; also add regression test

* fix that mapping.mappingColors were still used in toolbar and segments tab; clean up

* fix linting

* avoid that all segments in segment list rerender when mouse is moved

* make custom colors work when using setMapping via frontend API

* add analytics for custom colors in setMapping

* memoize diffing of segments; refactor memoization of diffing trees

* add docs for segment list

* remove obsolete type parameter for uniform

Co-authored-by: Philipp Otto <[email protected]>
Co-authored-by: Philipp Otto <[email protected]>

CHANGELOG.unreleased.md

@@ -17,6 +17,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Added a context menu option to separate an agglomerate skeleton using Min-Cut. Activate the Proofreading tool, select the source node and open the context menu by right-clicking on the target node which you would like to separate through Min-Cut. [#6361](https://github.com/scalableminds/webknossos/pull/6361)
 - Added a "clear" button to reset skeletons/meshes after successful mergers/split. [#6459](https://github.com/scalableminds/webknossos/pull/6459)
 - The proofreading tool now supports merging and splitting (via min-cut) agglomerates by rightclicking a segment (and not a node). Note that there still has to be an active node so that both partners of the operation are defined. [#6464](https://github.com/scalableminds/webknossos/pull/6464)
+- The color of a segments can now be changed in the segments tab. Rightclick a segment in the list and select "Change Color" to open a color picker. [#6372](https://github.com/scalableminds/webknossos/pull/6372)
 
 ### Changed
 - Selecting a node with the proofreading tool won't have any side effects anymore. Previous versions could load additional agglomerate skeletons in certain scenarios which could be confusing. [#6477](https://github.com/scalableminds/webknossos/pull/6477)

app/models/annotation/nml/NmlParser.scala

@@ -169,10 +169,11 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
         case _                           => None
       }
       Segment(
-        getSingleAttribute(node, "id").toLong,
-        anchorPosition,
-        getSingleAttributeOpt(node, "name"),
-        getSingleAttributeOpt(node, "created").flatMap(_.toLongOpt)
+        segmentId = getSingleAttribute(node, "id").toLong,
+        anchorPosition = anchorPosition,
+        name = getSingleAttributeOpt(node, "name"),
+        creationTime = getSingleAttributeOpt(node, "created").flatMap(_.toLongOpt),
+        color = parseColorOpt(node)
       )
     })
 
@@ -195,7 +196,7 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
           id <- idText.toIntOpt ?~ Messages("nml.boundingbox.id.invalid", idText)
           name = getSingleAttribute(node, "name")
           isVisible = getSingleAttribute(node, "isVisible").toBooleanOpt
-          color = parseColor(node)
+          color = parseColorOpt(node)
           boundingBox <- parseBoundingBox(node)
           nameOpt = if (name.isEmpty) None else Some(name)
         } yield NamedBoundingBoxProto(id, nameOpt, isVisible, color, boundingBox)
@@ -295,9 +296,6 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
       ColorProto(colorRed, colorBlue, colorGreen, colorAlpha)
     }
 
-  private def parseColor(node: XMLNode) =
-    parseColorOpt(node)
-
   private def parseName(node: XMLNode) =
     getSingleAttribute(node, "name")
 
@@ -315,7 +313,7 @@ object NmlParser extends LazyLogging with ProtoGeometryImplicits with ColorGener
     val treeIdText = getSingleAttribute(tree, "id")
     for {
       id <- treeIdText.toIntOpt ?~ Messages("nml.tree.id.invalid", treeIdText)
-      color = parseColor(tree)
+      color = parseColorOpt(tree)
       name = parseName(tree)
       groupId = parseGroupId(tree)
       isVisible = parseVisibility(tree, color)

app/models/annotation/nml/NmlWriter.scala

@@ -234,6 +234,7 @@ class NmlWriter @Inject()(implicit ec: ExecutionContext) extends FoxImplicits {
             writer.writeAttribute("anchorPositionY", a.y.toString)
             writer.writeAttribute("anchorPositionZ", a.z.toString)
           }
+          s.color.foreach(_ => writeColor(s.color))
         }
       }
     }

docs/volume_annotation.md

@@ -36,6 +36,20 @@ In the `Segmentation` tab on the right-hand side panel, you can see the segment
 
 The active segment ID under the cursor can be found in the status bar at the bottom of the screen or through the context-sensitive menu on right-click.
 
+### Segments List
+
+The right-hand side panel offers a `Segments` tab that lists segments and allows to edit these.
+A segment is added to the list as soon as it was clicked in the data viewport.
+The following functionality is available for each segment:
+
+- jumping to the segment (via left-click; this uses the position at which the segment was initially registered)
+- naming the segment
+- loading meshes for the segments (ad-hoc and precomputed if available)
+- changing the color of the segment
+- activating the segment id (so that you can annotate with that id)
+
+![Segments Tab](images/segments_tab2.jpeg)
+
 ### Merging volume annotation with fallback data
 
 After finishing the annotation of a volume layer with a fallback layer, the combined state of these layers can be materialized into a new dataset. For this, go to the layer settings in the left border tab. On the top right of the volume layer is the following button:

frontend/javascripts/components/loop.ts

@@ -9,13 +9,11 @@ class Loop extends Component<LoopProps, {}> {
   intervalId: number | null | undefined = null;
 
   componentDidMount() {
-    // @ts-expect-error ts-migrate(2339) FIXME: Property 'setInterval' does not exist on type '(Wi... Remove this comment to see the full error message
     this.intervalId = window.setInterval(this.props.onTick, this.props.interval);
   }
 
   componentWillUnmount() {
     if (this.intervalId != null) {
-      // @ts-expect-error ts-migrate(2339) FIXME: Property 'clearInterval' does not exist on type '(... Remove this comment to see the full error message
       window.clearInterval(this.intervalId);
       this.intervalId = null;
     }

frontend/javascripts/libs/UpdatableTexture.ts

@@ -1,6 +1,7 @@
 import * as THREE from "three";
 import { document } from "libs/window";
 import _ from "lodash";
+import { TypedArray } from "oxalis/constants";
 
 const lazyGetCanvas = _.memoize(() => {
   const canvas = document.createElement("canvas");
@@ -9,25 +10,42 @@ const lazyGetCanvas = _.memoize(() => {
   return canvas;
 });
 
-const getImageData = _.memoize((width: number, height: number): ImageData => {
-  const canvas = lazyGetCanvas();
-  const ctx = canvas.getContext("2d");
-  if (ctx == null) {
-    throw new Error("Could not get context for texture.");
-  }
-  const imageData = ctx.createImageData(width, height);
+const getImageData = _.memoize(
+  (
+    width: number,
+    height: number,
+    isInt: boolean,
+  ): { width: number; height: number; data: TypedArray } => {
+    const canvas = lazyGetCanvas();
+    const ctx = canvas.getContext("2d");
+    if (ctx == null) {
+      throw new Error("Could not get context for texture.");
+    }
 
-  // Explicitly "release" canvas. Necessary for iOS.
-  // See https://pqina.nl/blog/total-canvas-memory-use-exceeds-the-maximum-limit/
-  canvas.width = 1;
-  canvas.height = 1;
-  ctx.clearRect(0, 0, 1, 1);
+    // Integer textures cannot be used with four channels, which is why
+    // we are creating an image-like object here which can be used
+    // with ThreeJS if isDataTexture = true is given.
+    if (isInt) {
+      return { width, height, data: new Uint32Array(4 * width * height) };
+    }
 
-  return imageData;
-});
+    const imageData = ctx.createImageData(width, height);
+
+    // Explicitly "release" canvas. Necessary for iOS.
+    // See https://pqina.nl/blog/total-canvas-memory-use-exceeds-the-maximum-limit/
+    canvas.width = 1;
+    canvas.height = 1;
+    ctx.clearRect(0, 0, 1, 1);
+
+    return imageData;
+  },
+  (width: number, height: number, isInt: boolean) => `${width}_${height}_${isInt}`,
+);
 
 class UpdatableTexture extends THREE.Texture {
-  isUpdatableTexture: boolean;
+  isUpdatableTexture: boolean = true;
+  // Needs to be set to true for integer textures:
+  isDataTexture: boolean = false;
   renderer!: THREE.WebGLRenderer;
   gl: any;
   utils!: THREE.WebGLUtils;
@@ -47,7 +65,7 @@ class UpdatableTexture extends THREE.Texture {
     anisotropy?: number,
     encoding?: THREE.TextureEncoding,
   ) {
-    const imageData = getImageData(width, height);
+    const imageData = getImageData(width, height, type === THREE.UnsignedIntType);
 
     super(
       // @ts-ignore
@@ -69,7 +87,6 @@ class UpdatableTexture extends THREE.Texture {
     this.flipY = false;
     this.unpackAlignment = 1;
     this.needsUpdate = true;
-    this.isUpdatableTexture = true;
   }
 
   setRenderer(renderer: THREE.WebGLRenderer) {
@@ -110,7 +127,13 @@ class UpdatableTexture extends THREE.Texture {
     return this.renderer.properties.get(this).__webglTexture != null;
   }
 
-  update(src: Float32Array | Uint8Array, x: number, y: number, width: number, height: number) {
+  update(
+    src: Float32Array | Uint8Array | Uint32Array,
+    x: number,
+    y: number,
+    width: number,
+    height: number,
+  ) {
     if (!this.isInitialized()) {
       this.renderer.initTexture(this);
     }

frontend/javascripts/libs/error_handling.ts

@@ -201,12 +201,12 @@ class ErrorHandling {
     });
   }
 
-  assert = (
+  assert(
     bool: boolean,
     message: string,
     assertionContext?: Record<string, any>,
     dontThrowError: boolean = false,
-  ) => {
+  ): asserts bool is true {
     if (bool) {
       return;
     }
@@ -224,9 +224,13 @@ class ErrorHandling {
       console.error(error);
       this.airbrake.notify(error);
     }
-  };
+  }
 
-  assertExists(variable: any, message: string, assertionContext?: Record<string, any>) {
+  assertExists<T>(
+    variable: T | null,
+    message: string,
+    assertionContext?: Record<string, any>,
+  ): asserts variable is NonNullable<T> {
     if (variable != null) {
       return;
     }
@@ -264,4 +268,6 @@ class ErrorHandling {
   }
 }
 
-export default new ErrorHandling();
+const errorHandling: ErrorHandling = new ErrorHandling();
+
+export default errorHandling;

frontend/javascripts/libs/input.ts

@@ -614,9 +614,7 @@ export class InputMouse {
     // @ts-expect-error ts-migrate(2531) FIXME: Object is possibly 'null'.
     const boundingRect = document.getElementById(this.targetId).getBoundingClientRect();
     return _.extend({}, boundingRect, {
-      // @ts-expect-error ts-migrate(2339) FIXME: Property 'scrollX' does not exist on type '(Window... Remove this comment to see the full error message
       left: boundingRect.left + window.scrollX,
-      // @ts-expect-error ts-migrate(2339) FIXME: Property 'scrollY' does not exist on type '(Window... Remove this comment to see the full error message
       top: boundingRect.top + window.scrollY,
     });
   }

frontend/javascripts/libs/mjs.ts

@@ -3,7 +3,7 @@
 //   https://github.com/imbcmdth/mjs/blob/master/index.js
 // for all functions in M4x4, V2 and V3.
 import _ from "lodash";
-import type { Vector2, Vector3 } from "oxalis/constants";
+import type { Vector2, Vector3, Vector4 } from "oxalis/constants";
 import { chunk3 } from "oxalis/model/helpers/chunk";
 
 import mjs from "mjs";
@@ -188,6 +188,9 @@ const V2 = {
   equals(vec1: Vector2, vec2: Vector2): boolean {
     return vec1[0] === vec2[0] && vec1[1] === vec2[1];
   },
+  isEqual(a: Vector2, b: Vector2) {
+    return a[0] === b[0] && a[1] === b[1];
+  },
 };
 
 const _tmpVec: Vector3 = [0, 0, 0];
@@ -291,6 +294,15 @@ const V3 = {
     res[index] = Math.floor(res[index] / resolution[index]) * resolution[index];
     return res;
   },
+  isEqual(a: Vector3, b: Vector3) {
+    return a[0] === b[0] && a[1] === b[1] && a[2] === b[2];
+  },
+};
+
+const V4 = {
+  isEqual(a: Vector4, b: Vector4) {
+    return a[0] === b[0] && a[1] === b[1] && a[2] === b[2] && a[3] === b[3];
+  },
 };
 
-export { M4x4, V2, V3 };
+export { M4x4, V2, V3, V4 };

frontend/javascripts/libs/utils.ts

@@ -37,6 +37,10 @@ export function map3<A, B>(fn: (arg0: A, arg1: 0 | 1 | 2) => B, tuple: [A, A, A]
   const [x, y, z] = tuple;
   return [fn(x, 0), fn(y, 1), fn(z, 2)];
 }
+export function take3<A>(tuple: [A, A, A, A]): [A, A, A] {
+  return [tuple[0], tuple[1], tuple[2]];
+}
+
 export function map4<A, B>(
   fn: (arg0: A, arg1: 0 | 1 | 2 | 3) => B,
   tuple: [A, A, A, A],
@@ -191,6 +195,43 @@ export function hexToRgb(hex: string): Vector3 {
   const b = bigint & 255;
   return [r, g, b];
 }
+/**
+ * Converts an HSL color value to RGB. Conversion formula
+ * adapted from http://en.wikipedia.org/wiki/HSL_color_space.
+ * Assumes h, s, and l are contained in the set [0, 1] and
+ * returns r, g, and b in the set [0, 255].
+ *
+ * Taken from:
+ * https://stackoverflow.com/a/9493060
+ */
+export function hslaToRgba(hsla: Vector4): Vector4 {
+  const [h, s, l, a] = hsla;
+  let r;
+  let g;
+  let b;
+
+  if (s === 0) {
+    // eslint-disable-next-line no-multi-assign
+    r = g = b = l; // achromatic
+  } else {
+    const hue2rgb = function hue2rgb(p: number, q: number, t: number) {
+      if (t < 0) t += 1;
+      if (t > 1) t -= 1;
+      if (t < 1 / 6) return p + (q - p) * 6 * t;
+      if (t < 1 / 2) return q;
+      if (t < 2 / 3) return p + (q - p) * (2 / 3 - t) * 6;
+      return p;
+    };
+
+    const q = l < 0.5 ? l * (1 + s) : l + s - l * s;
+    const p = 2 * l - q;
+    r = hue2rgb(p, q, h + 1 / 3);
+    g = hue2rgb(p, q, h);
+    b = hue2rgb(p, q, h - 1 / 3);
+  }
+
+  return [Math.round(r * 255), Math.round(g * 255), Math.round(b * 255), Math.round(a * 255)];
+}
 export function colorObjectToRGBArray({ r, g, b }: ColorObject): Vector3 {
   return [r, g, b];
 }
@@ -469,7 +510,7 @@ export function busyWaitDevHelper(time: number) {
     }
   }
 }
-export function animationFrame(maxTimeout?: number): Promise<void> {
+export function animationFrame(maxTimeout?: number): Promise<number | void> {
   const rafPromise: Promise<ReturnType<typeof window.requestAnimationFrame>> = new Promise(
     (resolve) => {
       window.requestAnimationFrame(resolve);
@@ -859,7 +900,6 @@ export function convertBufferToImage(
 }
 export function getIsInIframe() {
   try {
-    // @ts-expect-error ts-migrate(2339) FIXME: Property 'self' does not exist on type '(Window & ... Remove this comment to see the full error message
     return window.self !== window.top;
   } catch (e) {
     return true;
@@ -870,12 +910,9 @@ export function getWindowBounds(): [number, number] {
   let width = 0;
   let height = 0;
 
-  // @ts-expect-error ts-migrate(2339) FIXME: Property 'innerWidth' does not exist on type '(Win... Remove this comment to see the full error message
   if (typeof window.innerWidth === "number") {
     // Non-IE
-    // @ts-expect-error ts-migrate(2339) FIXME: Property 'innerWidth' does not exist on type '(Win... Remove this comment to see the full error message
     width = window.innerWidth;
-    // @ts-expect-error ts-migrate(2339) FIXME: Property 'innerHeight' does not exist on type '(Wi... Remove this comment to see the full error message
     height = window.innerHeight;
   } else if (
     // @ts-expect-error ts-migrate(2339) FIXME: Property 'documentElement' does not exist on type ... Remove this comment to see the full error message

frontend/javascripts/libs/window.ts

@@ -59,12 +59,13 @@ let performanceCounterForMocking = 0;
 
 const _window =
   typeof window === "undefined"
-    ? {
+    ? ({
         alert: console.log.bind(console),
         app: null,
         location: dummyLocation,
-        // @ts-expect-error ts-migrate(7006) FIXME: Parameter 'resolve' implicitly has an 'any' type.
-        requestAnimationFrame: (resolve) => resolve(),
+        requestAnimationFrame: (resolver: Function) => {
+          setTimeout(resolver, 16);
+        },
         document,
         navigator: {
           onLine: true,
@@ -75,7 +76,7 @@ const _window =
         removeEventListener,
         open: (_url: string) => {},
         performance: { now: () => ++performanceCounterForMocking },
-      }
+      } as typeof window)
     : window;
 
 export default _window;

frontend/javascripts/messages.tsx

@@ -421,8 +421,6 @@ instead. Only enable this option if you understand its effect. All layers will n
   "users.change_email_confirmation": "The email has been changed",
   "mapping.too_big":
     "The mapping contains too many values, currently only up to 2^24 values are supported.",
-  "mapping.too_few_textures":
-    "Not enough textures available to support mappings. Mappings are disabled.",
   "mapping.unsupported_layer": "Mappings can only be enabled for segmentation layers.",
   "project.report.failed_to_refresh":
     "The project report page could not be refreshed. Please try to reload the page.",