Real-time lights on lightmapped surfaces with bake mode "All" are only disabled in GLES2, not GLES3

[Calinou]

Continued from #41629.

Godot version: 3.2 Git 4f891b7

OS/device including version: Fedora 33, GeForce GTX 1080 (NVIDIA 460.39)

Issue description:

Real-time lights on lightmapped surfaces with bake mode "All" are only disabled in GLES2, not GLES3. This happens regardless of which renderer was used at the time of lightmap baking. Texture atlases are only used when baking in GLES3, but this bug is also visible when using lightmaps baked without texture arrays in GLES3.

We should fix this so that the behavior in GLES3 is identical to GLES2.

Steps to reproduce:

Edit the minimal reproduction project and switch it to GLES3 using the dropdown in the top-right corner.

GLES3

DirectionalLight visible

DirectionalLight hidden

GLES2

DirectionalLight visible

DirectionalLight hidden

Minimal reproduction project: test_lightmap_shadow.zip

[Janders1800]

I don't know if this goes out of scope for this report but I've found out a couple more problems.

• Dynamic cannot cast shadows onto baked (Notice the sphere shadow gets cut off wen casting onto baked floor, this happens on GLES2, on GLES3 can't be tested due to original bug report)
• BakedLight cells don't store environment lighting (Notice dynamic objects aren't affected by sky light and look way darker)

GLES2:

GLES3:

Array textures seems to work fine for me in GLES3, since if I bake on GLES3 I get this error if I switch to GLES2.

Here is a modified test project reflecting this problems.
test_lightmap_shadow.zip

[Calinou]

Dynamic cannot cast shadows onto baked (Notice the sphere shadow gets cut off wen casting onto baked floor, this happens on GLES2, on GLES3 can't be tested due to original bug report)

By definition, this is impossible to do since the DirectionalLight isn't emitting any light on the baked surface. Mixing dynamic and fully-baked lighting is a difficult-to-solve problem with no perfect solutions.

We'd need to implement a subtractive shadowmapping option for DirectionalLight where no light is emitted and only a shadow is cast. Unity provides a subtractive lighting mode (here's a video comparing all its lighting modes). Subtractive shadowmapping isn't very realistic, but it's cheap enough to be usable on mobile platforms nowadays.

I started looking at implementing subtractive lighting in DirectionalLight but haven't succeeded yet: https://github.com/Calinou/godot/tree/add-directionallight-subtractive-lighting

In practice, subtractive shadowmapping could look something like this: https://www.youtube.com/watch?v=lQMCy328avE
Notice how the character's real-time shadow is configured to roughly match the color of the baked shadows, yet it never darkens surfaces that are already darkened by baked shadows.

Edit: Proposal opened: godotengine/godot-proposals#2354

BakedLight cells don't store environment lighting (Notice dynamic objects aren't affected by sky light and look way darker)

I can reproduce that as well on my end. I can see code in the shader to handle environment lighting for materials within a lightmap capture area, but I'm not sure how to enable it from the BakedLightmap or BakedLightmapData.

Edit: Bug report opened: #46591

[Janders1800]

We'd need to implement a subtractive shadowmapping option for DirectionalLight

Why only for DirectionalLight?

[Calinou]

Why only for DirectionalLight?

It can be implemented for other light types too, but shadows will look bad if they cross with each other. You should generally only use subtractive shadowmapping with a single light.

Also, subtractive shadowmapping is often used with omni/spot lights being baked in All mode, and therefore being unable to cast real-time shadows.

[akien-mga]

Fixed by #46694.