Custom segment colors with cuckoo hashing

[philippotto]

URL of deployed dev instance (used for testing):

• https://segmentcolorcuckoo.webknossos.xyz/

Todo:

• implement and adapt cuckoo table for storing colors
• build UI to edit colors
• use integer textures to store colors for 32 bit segment ids
• refactor initial proof-of-concept
• use custom colors for meshes, too
• make compatible with multiple segmentation layers
• make compatible with JSON mappings
  • unify with custom colors in JSON mappings
• fix conceptual bug in merger mode (random colors vs unique ids)
• fix bugs
  • diffing segment map should work against correct map
  • hiding volume layer produces invalid shader
  • fix that segment colors won't work when the tracing is loaded with no segmentation visible initially
• fix CI
• doublecheck compatibility with texture count constraints
• debounce update_segments actions or compact the update actions
• test thoroughly

Steps to test:

• open an annotation with a segmentation layer
  • click a segment
  • go to the segments tab and rightclick a segment to change its color
  • also load an ad-hoc mesh for that segment
  • change the color again --> the color in the viewport, in the segments tab and in the 3D viewport for the mesh should change
  • save & reload
  • the set color should still be there
• add a second volume layer
  • brush and change the segments color
  • switch between the volume layers and ensure that the colors are not mixed up somehow
• test the merger mode a bit
• open a dataset with a segmentation JSON mapping that has custom colors (e.g., ROI2017_wkw with mini-mapping-with-colors.json)
  • activate that mapping and see that the custom colors were used

Issues:

• Allow setting custom colors for segments #6306

---

(Please delete unneeded items, merge only when none are left open)

• Ready for review

[daniel-wer]

First part of my review, I'm missing frontend/javascripts/oxalis/shaders/segmentation.glsl.ts and the actual frontend/javascripts/oxalis/model/bucket_data_handling/cuckoo_table.ts, also testing. Will continue tomorrow :)

[daniel-wer]

What about the window "polyfill" in frontend/javascripts/libs/window.ts - do we need this mock?

[philippotto]

good catch 👍 after I changed the requestAnimationFrame mock (from an immediate resolve to a setTimeout with 10ms), this worked. without a delay, there was an infinite recursion. with 100 ms some tests failed. so, I hope that the 10ms are not flakey (I did several runs and they all succeeded with 10). let's watch out in the next weeks for this :)

[daniel-wer]

Very cool! Another longstanding issue, finally resolved 🎉

My observations from testing:

• The mesh loading is flickering weirdly. Maybe this is related to your changes in the mesh creation code.
• If I activate mini-mapping-with-colors in the ROI2017_wkw dataset, the mapped segments have the correct color. However, using the "Reset Segment Color" functionality from the segment list, seems to behave weirdly - the segment and icon colors no longer match. Also when setting another color, afterwards (which works) and then resetting the color again, the segment color is not reset (only the icon color).
• There's another method to specify mapping colors, using the frontend API. For example, on the test-agglomerate-file dataset, use webknossos.apiReady(3).then(async (api) => {api.data.setMapping(api.data.getVolumeTracingLayerName(), {1: 2}, {colors: [0.25, 0.5]})}). It seems to behave differently between the current master and this branch, the mapped colors (and the unmapped ones) are different. I'm not sure whether we still want to support this feature 🤷
• The color picker should be initialized with the current color (as is the case when changing the tree color)

[daniel-wer]

The changes up until a38fdeb look good to me. I think if you take care of the remaining ToDos and do a final test round, this should be ready to go, although I cannot finally approve it while I'm out of office.

[philippotto]

Very cool! Another longstanding issue, finally resolved 🎉

My observations from testing:

Thanks for your thorough testing 🙏

• The mesh loading is flickering weirdly. Maybe this is related to your changes in the mesh creation code.

This is fixed now. I had optimized the mesh chunks to share one material, however, the chunks are animated individually when loading which is why I had to remove the optimization again.

• If I activate mini-mapping-with-colors in the ROI2017_wkw dataset, the mapped segments have the correct color. However, using the "Reset Segment Color" functionality from the segment list, seems to behave weirdly - the segment and icon colors no longer match. Also when setting another color, afterwards (which works) and then resetting the color again, the segment color is not reset (only the icon color).

This turned out to be two errors: (a) the cuckoo table didn't overwrite keys correctly (I added a regression test for this) and (b) there were some spots which used the old customColor object. Should work now.

• There's another method to specify mapping colors, using the frontend API. For example, on the test-agglomerate-file dataset, use webknossos.apiReady(3).then(async (api) => {api.data.setMapping(api.data.getVolumeTracingLayerName(), {1: 2}, {colors: [0.25, 0.5]})}). It seems to behave differently between the current master and this branch, the mapped colors (and the unmapped ones) are different. I'm not sure whether we still want to support this feature shrug

I found this method to be quite buggy on master, too (the color hue didn't really have an impact). However, on this branch, the custom colors weren't properly used, either. I added code for this, but also added some analytics code so that we can check later whether this functionality is really used. The different types of how a mapping can be defined (mapping object vs equivalence classes) was once again a bit annoying, which is why I'd hope that we can remove it soon.