Add "merge" blend mode #6936

MichaelBuessemeyer · 2023-03-21T13:14:26Z

URL of deployed dev instance (used for testing):

https://coverblendmode.webknossos.xyz/

TODOs

check all layer types
- uint24 doesn't work :/
  - patched it, but need to talk with @philippotto about this "fix". (why the other packing degrees have a default alpah value of 10.0 and but the uint24 case does not.
- Layer with packingDegree = 2 / byteCount = 2 is still untested. This is not included the data layers dataset.
build blend mode switch in frontend
- discuss name of this blend mode. GIMP calls it "merge" as it merges images together where layer 1 has one part of the image and the other layer has the rest of the image. But I think we agreed on calling this blend mode "cover" as the lower layers cover the layers above in the layer list.

Steps to test:

Open a dataset with multiple layers (e.g. the data layers dataset) in view mode
Make all layers visible -> the data should be pretty bright
Got to the settings tab to the data rendering settings section and change the render mode to cover.
Now only the last layer in the list should be rendered.
Turn all layers off besides the uint24 layer. This layer should also be rendered correctly.
Then play a little around with what layers are set to visible and what rendering mode you select.
Please notice, that both float layers are slightly darker than the other layers. This is likely due to the layers / data of the layers themselves (as this behaviour is already present on the master).

Issues:

fixes #4194

(Please delete unneeded items, merge only when none are left open)

Updated changelog
Updated documentation if applicable

…er-blend-mode

- and add ui to switch rendering modes

frontend/javascripts/messages.tsx

frontend/javascripts/oxalis/shaders/texture_access.glsl.ts

frontend/javascripts/oxalis/shaders/filtering.glsl.ts

…er-blend-mode

- Fix layer precedence order.

MichaelBuessemeyer · 2023-04-20T14:35:33Z

As discussed, the code now decides whether a color is valid by alpha < 0.0 and not alpha <= 0.0.

…er-blend-mode

- fix uint24 rendering - fix rendering layers below if the the upper layer has no data yet - fix color and opacity setting in cover blend mode

…s into cover-blend-mode

MichaelBuessemeyer · 2023-04-28T09:47:17Z

As discussed, the code now decides whether a color is valid by alpha < 0.0 and not alpha <= 0.0.

This is now also outdated. The cover blend mode now applies Alpha_blending. This automatically discards colors with an invalid alpha value.

MichaelBuessemeyer · 2023-04-28T09:50:58Z

@philippotto Do you want to take over the pr review? Or should I ask daniel?

MichaelBuessemeyer · 2023-04-28T09:56:39Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

+        float mixed_alpha_factor = (1.0 - data_color.a) * layer_color.a;
+        float mixed_alpha = mixed_alpha_factor + data_color.a;
+        vec3 cover_color_rgb = data_color.a * data_color.rgb + mixed_alpha_factor * layer_color.rgb;
+        // Catching edge case where mixed_alpha is 0.0 and therefore the cover_color would have nan values.
+        float is_mixed_alpha_zero = float(mixed_alpha == 0.0);
+        vec4 cover_color = vec4(cover_color_rgb / (mixed_alpha + is_mixed_alpha_zero), mixed_alpha);
+        cover_color = mix(cover_color, vec4(0.0), is_mixed_alpha_zero);


@daniel-wer I diverged from the original approach with the if statement to guard against the division by zero. Instead I used a neat tick that was already used a few code lines above where a simple 1.0 is added to the potentially dangerous divisor in case the divisor (mixed_alpha) is equal to zero.

I simply saw how it was handled in the code lines above and think that it is even easier to ready than the if expression.

Oh, sadly I cannot comment the lines I mean in the comment above. The trick is already applied in the lines 218 - 228.

…er-blend-mode

docs/tracing_ui.md

philippotto

I left some code feedback :) My biggest question would be about segmentation layers: Did you discuss that these should be handled separately with the cover-mode?

CHANGELOG.unreleased.md

philippotto · 2023-05-03T13:06:48Z

docs/tracing_ui.md

@@ -116,6 +116,12 @@ Note, not all control/viewport settings are available in every annotation mode.
 #### Data Rendering
 - `Hardware Utilization`: Adjusts the quality level used for rendering data. Changing this setting influences how many data is downloaded from the server as well as how much pressure is put on the user's graphics card. Tune this value to your network connection and hardware power. After changing the setting, the page has to be refreshed.
 - `Loading Strategy`: You can choose between two different loading strategies. When using "best quality first" it will take a bit longer until you see data, because the highest quality is loaded. Alternatively, "Progressive quality" can be chosen which will improve the quality progressively while loading. As a result, initial data will be visible faster, but it will take more time until the best quality is shown.
+- `Blend Mode`: You can switch between two modes of blending the color layer. The default (Additive) simply sums up all color values of all visible layers. The cover mode renders all color layers on top of each other. Thus the top most layer covers the layers below. This blend mode is especially useful for datasets using multi modality layers. Here is an example for such a dataset by Bosch et al. [1]:


Is it true that this only affects color layers? Segmentation layers are handled differently?

Segmentation layers are handled differently?

Right now, that's exactly the case. The segmentation layer color pattern is added on top of the color value computed for the color layers.

Imo including the segmentation layer in that blending calculation might make things more complicated because in case a person has a color layer with opacity 100%, this would mean that this person could not see the segmentation. As this is currently the case (one can always see the segmentation if there is one), this might be confusing. Additionally, including the segmentation layer in the blending might require more configuration changes from a user for a typical use case for the "cover" blend mode: First change the blend mode to cover, then activate / deactivate the layer the user wants to see and finally tweak the opacities for the color layers so that the segmentation is still visible in the desired way the user wants.
Leaving the segmentation layer out of the blend mode calculation potentially removes the last step of the steps described above.

But I am open for discussion on this. I'll try to schedule a small meeting for this :)

Update: We already discussed a solution:

Keep the segmentation layer special right now.

Add another pr that enabled editing the layer order

Add another pr that enables setting the blend mode for every single layer including the segmentation layer. The default for segmentation layer will be cover, the default for color layers will be additive.

frontend/javascripts/messages.tsx

frontend/javascripts/oxalis/geometries/materials/plane_material_factory.ts

philippotto · 2023-05-03T13:14:17Z

frontend/javascripts/oxalis/shaders/filtering.glsl.ts

      vec4 color = getMaybeFilteredColor(layerIndex, d_texture_width, packingDegree, worldPositionUVW, suppressBilinearFiltering, supportsPrecomputedBucketAddress);

+      vec4 usedFallbackColor = vec4(0.0);


the semantics of this var is a simple boolean, right? in that case, maybe define a struct with a vec4 and a bool?

Oh wow, I never knew that glsl has structs 🤯. Thanks for the input 🙏

philippotto · 2023-05-03T13:14:46Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

-
+          );
+          float used_fallback = maybe_filtered_color_and_used_fallback[1].r;
+          // For uint24 the alpha channel is always 0.0 :/


is this still true?

oh sorry, forgot to remove this comment. This is no longer true thanks to the changes in texture_bucket_manager.ts.

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

philippotto · 2023-05-03T13:20:12Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

+        color_value = color_value * <%= name %>_alpha * <%= name %>_color;
+        // Marking the color as invalid by setting alpha to 0.0 if the fallback color has been used
+        // so that it does not cover other colors.
+        vec4 layer_color = vec4(color_value, used_fallback == 1.0 ? 0.0 : maybe_filtered_color.a * <%= name %>_alpha);


I think, I'd simply rename color_value to layer_value and then don't define a new variable (as it's not necessary to have both at the same time, I think).

The problem is that the calculates done with color_value are only for the rgb channels and not for the alpha channel. But the blending needs a color with an alpha channel. Thus the calculations on the color are done with vec3 color_value and from that a full vec4 layer_color is calculated and then used for the blending calculation.

If I would remove the layer_color and rename color_value to layer_value, I would still have to construct a vec4 combining the color_value with the alpha channel in the blending methods (now extracted into own methods). So the construction of an additional vec4 cannot be avoided imo.

philippotto · 2023-05-03T13:24:42Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

@@ -222,14 +226,33 @@ void main() {
        color_value = abs(color_value - <%= name %>_is_inverted);
        // Catch the case where max == min would causes a NaN value and use black as a fallback color.
        color_value = mix(color_value, vec3(0.0), is_max_and_min_equal);
-        // Multiply with color and alpha for <%= name %>
-        data_color += color_value * <%= name %>_alpha * <%= name %>_color;
+        color_value = color_value * <%= name %>_alpha * <%= name %>_color;


@MichaelBuessemeyer can you give this entire code block a bit of love? variable naming could be more consistent and I'd also hope that the code for the different blend mode could be extracted into helper functions?

the different blend mode could be extracted into helper functions?

That is a very good suggestion: I added a new glsl module called blending and extracted the methods there. I also renamed all variable to snail case and all methods & struct names to camelCase. Sadly, the name is very inconsistent in the shaders and thus I couldn't find a clear pattern what naming scheme / style to use. I also discussed this shortly with daniel (the naming style)

philippotto · 2023-05-03T13:26:41Z

@philippotto Do you want to take over the pr review? Or should I ask daniel?

I reviewed it now, but I'm not sure whether the two of you already discussed some details I'm missing now.

Independent from that, I think, it would be good to show a new dev instance to the team to keep these in the loop who gave feedback last time.

…er-blend-mode

Co-authored-by: Philipp Otto <[email protected]>

…s into cover-blend-mode

- make clear that the blend mode only applies to color layers excluding segment layers

…er-blend-mode

MichaelBuessemeyer

Thanks for the feedback philipp. I think I covered everything. Let's wait for the CI and then ask for some further testing feedback. I hope everything should work now as expected.

MichaelBuessemeyer · 2023-05-04T09:19:38Z

docs/tracing_ui.md

@@ -116,6 +116,12 @@ Note, not all control/viewport settings are available in every annotation mode.
 #### Data Rendering
 - `Hardware Utilization`: Adjusts the quality level used for rendering data. Changing this setting influences how many data is downloaded from the server as well as how much pressure is put on the user's graphics card. Tune this value to your network connection and hardware power. After changing the setting, the page has to be refreshed.
 - `Loading Strategy`: You can choose between two different loading strategies. When using "best quality first" it will take a bit longer until you see data, because the highest quality is loaded. Alternatively, "Progressive quality" can be chosen which will improve the quality progressively while loading. As a result, initial data will be visible faster, but it will take more time until the best quality is shown.
+- `Blend Mode`: You can switch between two modes of blending the color layer. The default (Additive) simply sums up all color values of all visible layers. The cover mode renders all color layers on top of each other. Thus the top most layer covers the layers below. This blend mode is especially useful for datasets using multi modality layers. Here is an example for such a dataset by Bosch et al. [1]:


Segmentation layers are handled differently?

Right now, that's exactly the case. The segmentation layer color pattern is added on top of the color value computed for the color layers.

Imo including the segmentation layer in that blending calculation might make things more complicated because in case a person has a color layer with opacity 100%, this would mean that this person could not see the segmentation. As this is currently the case (one can always see the segmentation if there is one), this might be confusing. Additionally, including the segmentation layer in the blending might require more configuration changes from a user for a typical use case for the "cover" blend mode: First change the blend mode to cover, then activate / deactivate the layer the user wants to see and finally tweak the opacities for the color layers so that the segmentation is still visible in the desired way the user wants.
Leaving the segmentation layer out of the blend mode calculation potentially removes the last step of the steps described above.

But I am open for discussion on this. I'll try to schedule a small meeting for this :)

frontend/javascripts/messages.tsx

MichaelBuessemeyer · 2023-05-04T11:25:12Z

frontend/javascripts/oxalis/shaders/filtering.glsl.ts

      vec4 color = getMaybeFilteredColor(layerIndex, d_texture_width, packingDegree, worldPositionUVW, suppressBilinearFiltering, supportsPrecomputedBucketAddress);

+      vec4 usedFallbackColor = vec4(0.0);


Oh wow, I never knew that glsl has structs 🤯. Thanks for the input 🙏

MichaelBuessemeyer · 2023-05-04T11:25:25Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

-
+          );
+          float used_fallback = maybe_filtered_color_and_used_fallback[1].r;
+          // For uint24 the alpha channel is always 0.0 :/


oh sorry, forgot to remove this comment. This is no longer true thanks to the changes in texture_bucket_manager.ts.

MichaelBuessemeyer · 2023-05-04T12:04:45Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

@@ -222,14 +226,33 @@ void main() {
        color_value = abs(color_value - <%= name %>_is_inverted);
        // Catch the case where max == min would causes a NaN value and use black as a fallback color.
        color_value = mix(color_value, vec3(0.0), is_max_and_min_equal);
-        // Multiply with color and alpha for <%= name %>
-        data_color += color_value * <%= name %>_alpha * <%= name %>_color;
+        color_value = color_value * <%= name %>_alpha * <%= name %>_color;


the different blend mode could be extracted into helper functions?

That is a very good suggestion: I added a new glsl module called blending and extracted the methods there. I also renamed all variable to snail case and all methods & struct names to camelCase. Sadly, the name is very inconsistent in the shaders and thus I couldn't find a clear pattern what naming scheme / style to use. I also discussed this shortly with daniel (the naming style)

MichaelBuessemeyer · 2023-05-04T12:45:25Z

frontend/javascripts/oxalis/shaders/main_data_shaders.glsl.ts

+        color_value = color_value * <%= name %>_alpha * <%= name %>_color;
+        // Marking the color as invalid by setting alpha to 0.0 if the fallback color has been used
+        // so that it does not cover other colors.
+        vec4 layer_color = vec4(color_value, used_fallback == 1.0 ? 0.0 : maybe_filtered_color.a * <%= name %>_alpha);


The problem is that the calculates done with color_value are only for the rgb channels and not for the alpha channel. But the blending needs a color with an alpha channel. Thus the calculations on the color are done with vec3 color_value and from that a full vec4 layer_color is calculated and then used for the blending calculation.

If I would remove the layer_color and rename color_value to layer_value, I would still have to construct a vec4 combining the color_value with the alpha channel in the blending methods (now extracted into own methods). So the construction of an additional vec4 cannot be avoided imo.

philippotto

Looks great to me :) Looking forward to the future iterations with layer ordering and layer-specific blend modes.

…ty-list-drawings * 'master' of github.com:scalableminds/webknossos: [Docs] Update embedded YouTube URLs (#7102) Show organization in dataset info tab (#7087) Added Tutorials to Docs (#7095) Combine both download modals into one (#7068) Add "merge" blend mode (#6936) Changed the spacing/width of VX reports runs selection (#7094) Allow dataset managers to see all workflow reports of their orga (#7081) Fix rectangle at 0,0,0 (#7088) Allow to disable automatic reloading of meshes during proofreading (#7076)

MichaelBuessemeyer added 2 commits March 15, 2023 18:08

WIP add new blend mode

71e511e

WIP add cover blend mode

3490baf

MichaelBuessemeyer added enhancement frontend rendering labels Mar 21, 2023

MichaelBuessemeyer self-assigned this Mar 21, 2023

MichaelBuessemeyer added 5 commits March 21, 2023 14:38

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

9710973

…er-blend-mode

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

3f871db

…er-blend-mode

fix uint24 rendering in merge mode

ce1fff2

- and add ui to switch rendering modes

remove unused imports

c22b91c

rename to render mode to blend mode

4821ce5

MichaelBuessemeyer commented Apr 20, 2023

View reviewed changes

frontend/javascripts/messages.tsx Outdated Show resolved Hide resolved

frontend/javascripts/oxalis/shaders/texture_access.glsl.ts Outdated Show resolved Hide resolved

frontend/javascripts/oxalis/shaders/filtering.glsl.ts Outdated Show resolved Hide resolved

MichaelBuessemeyer added 2 commits April 20, 2023 12:17

improve in line comments

11020f5

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

1f5b217

…er-blend-mode

MichaelBuessemeyer changed the title ~~WIP: Add "merge" blend mode~~ Add "merge" blend mode Apr 20, 2023

decide whether a color is valid by alpha < 0.

0dbf947

- Fix layer precedence order.

MichaelBuessemeyer marked this pull request as ready for review April 20, 2023 14:35

MichaelBuessemeyer and others added 6 commits April 27, 2023 11:08

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

6b4da6a

…er-blend-mode

multiple fixes in cover blend mode

820bd9d

- fix uint24 rendering - fix rendering layers below if the the upper layer has no data yet - fix color and opacity setting in cover blend mode

fix reading blendmode from backend on inital load

2ca4a81

Merge branch 'master' into cover-blend-mode

f9cdd4d

refactoring & make changes ready for review

ebadcb8

Merge branch 'cover-blend-mode' of github.com:scalableminds/webknosso…

134acf8

…s into cover-blend-mode

MichaelBuessemeyer commented Apr 28, 2023

View reviewed changes

MichaelBuessemeyer added 3 commits April 28, 2023 12:31

add changelog entry

30b268f

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

f9fe4b9

…er-blend-mode

add blend mode setting to docs

777c923

MichaelBuessemeyer commented Apr 28, 2023

View reviewed changes

docs/tracing_ui.md Show resolved Hide resolved

philippotto reviewed May 3, 2023

View reviewed changes

MichaelBuessemeyer and others added 5 commits May 4, 2023 11:35

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

fd9a454

…er-blend-mode

Apply suggestions from code review

a23718e

Co-authored-by: Philipp Otto <[email protected]>

Merge branch 'cover-blend-mode' of github.com:scalableminds/webknosso…

9ff0aa4

…s into cover-blend-mode

refactor shader code for blending

56e971a

- make clear that the blend mode only applies to color layers excluding segment layers

Merge branch 'master' of github.com:scalableminds/webknossos into cov…

96cb1b2

…er-blend-mode

MichaelBuessemeyer commented May 4, 2023

View reviewed changes

MichaelBuessemeyer requested a review from philippotto May 10, 2023 08:15

philippotto approved these changes May 12, 2023

View reviewed changes

Merge branch 'master' into cover-blend-mode

c4df6c2

MichaelBuessemeyer enabled auto-merge (squash) May 25, 2023 08:49

MichaelBuessemeyer merged commit d338889 into master May 25, 2023

MichaelBuessemeyer deleted the cover-blend-mode branch May 25, 2023 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add "merge" blend mode #6936

Add "merge" blend mode #6936

MichaelBuessemeyer commented Mar 21, 2023 •

edited

Loading

MichaelBuessemeyer commented Apr 20, 2023

MichaelBuessemeyer commented Apr 28, 2023

MichaelBuessemeyer commented Apr 28, 2023

MichaelBuessemeyer Apr 28, 2023

MichaelBuessemeyer Apr 28, 2023

philippotto left a comment

philippotto May 3, 2023

MichaelBuessemeyer May 4, 2023

MichaelBuessemeyer May 4, 2023

philippotto May 3, 2023

MichaelBuessemeyer May 4, 2023

philippotto May 3, 2023

MichaelBuessemeyer May 4, 2023

philippotto May 3, 2023

MichaelBuessemeyer May 4, 2023

philippotto May 3, 2023

MichaelBuessemeyer May 4, 2023

philippotto commented May 3, 2023

MichaelBuessemeyer left a comment

MichaelBuessemeyer May 4, 2023

MichaelBuessemeyer May 4, 2023

MichaelBuessemeyer May 4, 2023

MichaelBuessemeyer May 4, 2023

MichaelBuessemeyer May 4, 2023

philippotto left a comment

		vec4 color = getMaybeFilteredColor(layerIndex, d_texture_width, packingDegree, worldPositionUVW, suppressBilinearFiltering, supportsPrecomputedBucketAddress);

		vec4 usedFallbackColor = vec4(0.0);

Add "merge" blend mode #6936

Add "merge" blend mode #6936

Conversation

MichaelBuessemeyer commented Mar 21, 2023 • edited Loading

URL of deployed dev instance (used for testing):

TODOs

Steps to test:

Issues:

MichaelBuessemeyer commented Apr 20, 2023

MichaelBuessemeyer commented Apr 28, 2023

MichaelBuessemeyer commented Apr 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philippotto left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philippotto commented May 3, 2023

MichaelBuessemeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philippotto left a comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Mar 21, 2023 •

edited

Loading