Add a 32 bpc frame buffer option to Viewport

[Lauson1ex]

Describe the project you are working on

Third-person, high-fidelity visuals 3d game.

Describe the problem or limitation you are having in your project

Godot doesn't have, and won't have for a long time, if ever, MRTs (Multiple Render Targets). Therefore, enable us to use the second best thing: 32-bit per component frame buffers.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

For starters, it is more efficient to use a single, 32-bit per component frame buffer than using two MRTs. With clever use of custom shaders, it is possible to pack and store more information than just colors inside buffers. 32-bit per component frame buffers are actually 128-bit data buffers. With them, it is be possible to implement advanced features in Godot, such as Motion Vectors for exemple, and thus implement advanced post process effects like TAA and #2933 without having to completely overhaul the underlying rendering backend, and distribute those as user-created add-ons.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Currently, Godot 3.x only supports up to 16-bit per component frame buffers (GL_RGBA16F, when the option use_hdr in a Viewport is enabled). This proposal aims to add optional 32-bit per component frame buffers (VK_FORMAT_R32G32B32A32_SFLOAT for Vulkan, GL_RGBA32F for GLES3) in order to circumvent the lack of MRTs (since enabling this rendermode is far less troublesome than fully implementing MRTs). Due to the nature of what I just said, this is a feature intended for advanced user only, and therefore must be an option in the project settings that must be opt-ed in. Then, the option to use 32-bit per component frame buffers will be available to be enabled on Viewports, just like the use_hdr option.

Using reinterpret casts such as floatBitsToUint/uintBitsToFloat and bit manipulation in shaders, it is possible to pack information during the regular geometry rendering in a single pass. For example, Godot 4's DOF post process shader uses the alpha component to store the blur radius. With clever packing and bit manipulation, it is possible to store albedo, depth, normals and lights in a single 128-bit buffer, essentially creating a tiny deferred buffer in a forward renderer. Or, storing colors and motion vectors as the geometry gets rendered. The applications are quite endless.

For reference, I experimented packing colors and depth in a single buffer in Godot 3.x, by packing depth as a 24-bit value and 8-bit per component RGB (by transforming linear RGB to sRGB first), and then reading everything back with a single texture sample, unpacking, and finally enjoying depth in canvas_item shaders. It works well, but then I lose HDR. My project needs HDR, so this isn't usable. Higher bit-depth frame buffers must be available for that. 16-bit per component (48-bit in total) is enough for color with HDR. 16-bit per component (32-bit in total) are enough for motion vectors (since usual implementations use GL_RG16F). As you can see, you can pack a lot of data in these buffers. Then, it is the user's responsibility to implement whatever they may need, without having to change the underlying rendering backend. Literature on how to efficiently perform GPU data packing is plenty on the Internet.

A new shader render_mode will probably be needed for spatial shaders: disable_alpha, to be able to write to the alpha component of the buffer without making the geometry go to the transparent pipeline, and therefore enjoy the full 128-bit buffer, but this is a very low-priority requirement; having 96-bits to work with is a good start.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

This is core.

[Calinou]

Due to the nature of what I just said, this is a feature intended for advanced user only, and therefore must be an option in the project settings that must be opt-ed in. Then, the option to use 32-bit per component frame buffers will be available to be enabled on Viewports, just like the use_hdr option.

I wouldn't lock the ability to use a property behind a project setting. As long as it's not the default, I think it's fine. The only thing we need to be careful about is to emit a node configuration warning and print a warning message if an incompatible mode is used in GLES2.

[Lauson1ex]

I wouldn't lock the ability to use a property behind a project setting. As long as it's not the default, I think it's fine. The only thing we need to be careful about is to emit a node configuration warning and print a warning message if an incompatible mode is used in GLES2.

The justification for suggesting this feature to be locked behind a project settings is because I fear misuse. A user who is not careful may think: "32 is better than 16, right? So my game will automatically look better if I use this mode", when in reality they'll be using twice the video memory requirement for that Viewport. Plus the fact that frame buffers with 32-bit components use precision highp by default, which can impact performance compared to mediump.

Godot has settings that are locked behind project settings or outright hidden unless you read the API and enable them via scripting, so I thought it would only be appropriate, but I could be wrong.

[Calinou]

A user who is not careful may think: "32 is better than 16, right? So my game will automatically look better if I use this mode", when in reality they'll be using twice the video memory requirement for that Viewport. Plus the fact that frame buffers with 32-bit components use precision highp by default, which can impact performance compared to mediump.

In master, we can replace the hdr boolean property with a depth_per_component enum property with the following values: 8-bit (LDR), 16-bit (HDR), 32-bit (Advanced Shaders Only).

In 3.x, we have to keep the hdr boolean for compatibility with existing projects. I guess we have to add a second boolean property like depth_per_component_32. This property would be ignored if hdr is disabled (with a node configuration warning displayed).

[Lauson1ex]

In master, we can replace the hdr boolean property with a depth_per_component enum property with the following values: 8-bit (LDR), 16-bit (HDR), 32-bit (Advanced Shaders Only).

In 3.x, we have to keep the hdr boolean for compatibility with existing projects, so I guess we have to add a second boolean property like depth_per_component_32. This property would be ignored if hdr is disabled (with a node configuration warning displayed).

This is actually more usable than what I suggested. 🙂

[Calinou]

@Lauson1ex I've implemented this for both the master and 3.x branches, could you give it a try to confirm it works as intended in both versions? See the references above for the pull request links. Thanks in advance 🙂

[Lauson1ex]

@Lauson1ex I've implemented this for both the master and 3.x branches, could you give it a try to confirm it works as intended in both versions? See the references above for the pull request links. Thanks in advance 🙂

@Calinou Absolutely. Is there a precompiled version where I can test this on real quick? Or should I fork, merge and compile it myself?

[Calinou]

Absolutely. Is there a precompiled version where I can test this on real quick? Or should I fork, merge and compile it myself?

You may be able to find precompiled builds under the pull requests' respective Checks tab. See Testing pull requests in the documentation for more information.

[Lauson1ex]

You may be able to find precompiled builds under the pull requests' respective Checks tab. See Testing pull request in the documentation for more information.

Gotcha. Thank you.

[19PHOBOSS98]

@Lauson1ex I've implemented this for both the master and 3.x branches, could you give it a try to confirm it works as intended in both versions? See the references above for the pull request links. Thanks in advance 🙂

Awesome, when can we expect to get it in 3.4?

[Calinou]

Awesome, when can we expect to get it in 3.4?

The 3.4 branch will only get bug fixes now that 3.4.stable has been released. New features will land in the 3.x branch, which will become 3.5 in the future (likely in 4-5 months from now).

[Calinou]

For reference, this feature is now implemented in 3.5beta by godotengine/godot#51708. You can try it in 3.5beta3.

I'll leave this proposal open, as it should still be implemented in 4.x eventually.

[danilw]

I'll leave this proposal open, as it should still be implemented in 4.x eventually.

if I understand correctly, in current Godot 4 beta 5 - SubViewport is only 8 bits?
get_node("SubViewport").get_viewport().get_texture().get_image().get_format() return FORMAT_RGB8
and I dont see any HDR-related options

[clayjohn]

I'll leave this proposal open, as it should still be implemented in 4.x eventually.

if I understand correctly, in current Godot 4 beta 5 - SubViewport is only 8 bits? get_node("SubViewport").get_viewport().get_texture().get_image().get_format() return FORMAT_RGB8 and I dont see any HDR-related options

Kind of. 3D rendering still happens in HDR (16bpc) but all 2D rendering uses 8bpc. We have a plan to expose a way to get the HDR buffer directly from the viewport though to alleviate this issue. Ultimately though, for custom effects we expose the ability to create your own framebuffers and textures through the RenderingDevice directly so you don't have to go through the Godot pipeline.

[danilw]

for custom effects we expose the ability to create your own framebuffers and textures through the RenderingDevice directly so you don't have to go through the Godot pipeline.

thank you, il check it

[Platinguin]

We have a plan to expose a way to get the HDR buffer directly from the viewport though to alleviate this issue

Thanks, I'm really looking forward to this! I was using the Viewport / Subviewport to create a height map and a gradient map for coloring the terrain, with a couple of pre-rendered mountains. (they were rendered in 16bit in Godot 3.5) The Viewport is actually a very powerful feature and can be utilized to render something quickly within one frame in the background. (and baked to a texture) It would be a shame to not be able to use it's full potential.

[danilw]

Small update on Why people need 16-32 bit in framebuffer-visual in Godot4:

1. You can not sync compute to graphic in Godot4, if you write anything in Compute shader - you need to copy memory from VRAM to RAM and then copy back to VRAM for graphic-pipeline to use in visual-shaders.
2. Work with HDR images 16-32 bit exr - you can load them in Godot4, but you can not save from framebuffer/viewport because frambuffer is only 8-bit.
3. Custom complex postprocessing - where you need to read/write/modify depth or normal or some other data - you need it in visual-shaders and save to 32-bit format, 8bit is not enough. (currently you can not even copy depth or normal to compute, even with huge vram-ram-vram overhead)

Compute-shaders-buffers is not solution for all of this.

[clayjohn]

@danilw Have you seen the TextureRD textures? They allow writing to a texture in a compute shader and then using it directly in the engine through a GDShader.

I use them in my volumetric cloud demo https://github.com/clayjohn/godot-volumetric-cloud-demo-v2. They allow a much better workflow for custom texture work than Viewports and you can easily avoid the entire overhead of the Viewport API.

To me, adding a 32 bpc option to the Viewport is a huge hack that is only done to support workflows which are better solved without Viewports. I'd prefer to see an expansion of what users can do with the RD API and things like DrawableTexture

[danilw]

Have you seen the TextureRD textures? They allow writing to a texture in a compute shader and then using it directly in the engine through a GDShader.

il try it, thanks

[danilw]

To me, adding a 32 bpc option to the Viewport is a huge hack that is only done to support workflows which are better solved without Viewports

There still many use cases for Viewport-32 bit that is much more comfortable to do without compute:

To do "fade-feeedback" - where you need 32bit to "slow fade" like COLOR.rgb = clamp(texture(previous_frame, UV)*0.95+COLOR.rgb, 0., 1.); - like you draw in Viewport many particles and then "fade" them in next history+viewport - to have "trails like" effect.

Work with 16-32-bit images, like I said - it much more comfortable to have everything in "viewport-fragment-shaders-logic" as custom-visual shaders.

Depth/normal to compute - you need 32bit fbo to save depth/normal.

[danilw]

Update - if someone need example of using 32-bit buffer to do stuff with accessing depth:

JFA_driven_motion_blur_addon

https://github.com/sphynx-owner/JFA_driven_motion_blur_addon/blob/master/addons/MyJumpFloodIteration/jump_flood_blur.gd - this script use RenderingDevice to send depth from current frame to compute shader, process compute shader and send compute result to fragment shader-render pass, compute shader also read previous state of self - it is complex example and exact use case I needed.

Compute-shader buffer obviously can be 32-bit.