fix(lighting): create Masks2 RT and divide SSGI AO by vertexAO

[doodlum]

Extracted from the effect11sh2 branch as a standalone change.

Summary

Adds a dedicated render target for vertex AO so the deferred composite can correct SSGI overdarkening: vanilla SSAO does not know about vertex-color AO that has already shadowed mesh/grass pixels, and SSGI then darkens the already-darkened pixels.

Plumbing (C++ + RT)

• src/Deferred.cpp wires up MASKS2 (R16G16B16A16_FLOAT, blendable) as the 8th forward render target and binds its SRV at t9 for DeferredCompositeCS. The MASKS2 macro already exists in Deferred.h.
• Cleared every frame via the existing SRTM_CLEAR loop in StartDeferred — MASKS2 lives at index 7 of forwardRenderTargets, which the loop for (uint i = 2; i < 8; i++) already marks as SRTM_CLEAR. No extra clear calls are added.

Shader-side writes

• Lighting.hlsl and RunGrass.hlsl write the mask — these are the paths that bake vertex AO into the deferred color, so they're the paths SSGI needs to compensate for. Distant trees, effects, and decals don't apply vertex AO and are left alone.
• Stored as 1 - vertexAO. The cleared default of 0 then correctly maps to "no occlusion" (vertexAO = 1) for any pixel that doesn't write (sky, water, distant trees, effects, decals). This avoids per-feature explicit writes for paths that have no vertex AO.
• Lighting's pixel shader now defines vertexAO in all branches (HAIR=1, SKYLIGHTING=existing, else=new) so the psout.Masks2 write is well-defined.
• The stale per-SNOW psout.Parameters output, which sat at SV_Target7 with no consumer, is replaced by Masks2 — its writes are removed.

Shader-side read

• DeferredCompositeCS.hlsl samples Masks2.x, recovers vertexAO = 1 - x, and divides ssgiAo by it before the multi-bounce AO step, so SSGI darkens only the unshadowed remainder.

vertexAO correctness fixes

• RunGrass.hlsl — Both pixel shader paths previously normalised vertexColor by its own max before computing vertexAO, so vertexAO always evaluated to ~1 and never modulated skylightingDiffuse. Capture vertexAO before the normalisation divide, then divide vertexColor by it.
• Lighting.hlsl SKYLIGHTING + else paths — wrap the max-of-channels with Color::ColorToLinear so vertexAO lives in linear space, matching the linear skylightingDiffuse / ssgiAo it scales.

Test plan

• Build a Skyrim DLL and confirm grass under heavy foliage shows visible skylight attenuation (was previously flat) and is not overdarkened with SSGI on.
• With SSGI + Skylighting on, confirm tinted/dark-vertex-colored meshes are not overdarkened in exteriors at midday.
• In RenderDoc, confirm Masks2 is bound as the 8th deferred RT and at t9 SRV in DeferredCompositeCS. Confirm the RT is cleared to 0,0,0,0 at the start of each frame (via SRTM_CLEAR).
• Confirm distant trees, effect-shader geometry, decals, and sky still render correctly — they should leave Masks2 at the cleared 0 value, which the composite reads as no AO modulation.

Summary by CodeRabbit

• Graphics & Rendering
  • Added a new mask render target to extend mask outputs for sky, water, decals and effects
  • Improved ambient occlusion compensation in the screen-space GI path for more stable AO
  • Normalized vertex-based AO before skylighting for more consistent diffuse lighting
  • Updated grass rendering to output the new mask target for improved ground shading

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9b2d7401-bfe6-4760-abaf-ee34ed5f726c

📥 Commits

Reviewing files that changed from the base of the PR and between b212d7f and 457f0c6.

📒 Files selected for processing (2)

• package/Shaders/DeferredCompositeCS.hlsl
• src/Deferred.cpp

🚧 Files skipped from review as they are similar to previous changes (2)

• package/Shaders/DeferredCompositeCS.hlsl
• src/Deferred.cpp

---

📝 Walkthrough

Walkthrough

Adds MASKS2 (RT7) carrying per-pixel vertex AO (1.0 - vertexAO). C++ allocates and exposes MASKS2 SRV; Lighting and RunGrass shaders write Masks2; DeferredComposite binds Masks2 at t9 and uses it to normalize SSGI/skylighting AO samples.

Changes

Vertex AO Render Target and Deferred Lighting Integration

Layer / File(s)	Summary
C++ Runtime: MASKS2 Render Target Setup src/Deferred.cpp	Allocates new MASKS2 render target (R16_UNORM), includes it in deferred RT selection, and supplies its SRV to the deferred composite shader at t9.
Deferred Composite Shader: SSGI AO Compensation package/Shaders/DeferredCompositeCS.hlsl	Binds Masks2Texture : register(t9) and samples per-pixel vertexAO from it to rescale ssgiAo (divide by max(vertexAO, EPSILON_DIVISION)) before saturate.
Lighting Shader: Masks2 Output and Vertex AO Linearization package/Shaders/Lighting.hlsl	Replaces SNOW-only Parameters RT with Masks2: SV_Target7, removes SNOW Parameters writes, linearizes vertex color to derive vertexAO for skylighting normalization, and writes psout.Masks2 = float4(1.0 - vertexAO,0,0,0).
RunGrass Shader: Masks2 Output and Vertex AO Handling package/Shaders/RunGrass.hlsl	Adds Masks2: SV_Target7 to grass and non-grass PS outputs, computes vertexAO from max vertexColor channel, normalizes vertexColor by vertexAO, and writes Masks2 = float4(1.0 - vertexAO,0,0,0) instead of Parameters.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

• community-shaders/skyrim-community-shaders#2183: Modifies RunGrass vertex-AO behavior and overlaps with this PR's RunGrass vertexAO changes.
• community-shaders/skyrim-community-shaders#2463: Changes Lighting.hlsl vertex-color linearization and AO derivation related to skylighting.
• community-shaders/skyrim-community-shaders#2185: Adjusts deferred composite SRV/register wiring around t9, overlapping with this PR's composite binding changes.

Suggested reviewers

• alandtse
• SkrubbySkrubInAShrub

Poem

🐰 In masks of render, soft and new,
I tuck small shadows into view,
AO stitched into RT seven,
So grass and sky glow true at heaven,
Hop, blend, and brighten — carrot-coded hue.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the primary changes: creating a new Masks2 render target and implementing vertex-AO normalization for SSGI, which are the main objectives of the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/lighting-vertexao-linearise

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 OpenGrep (1.22.0)

OpenGrep fatal error (exit code 2): [00.12][ERROR]: Error: exception Unix_error: No such file or directory stat src/Deferred.cpp
Raised by primitive operation at UTmp.replace_named_pipe_by_regular_file_if_needed in file "libs/commons/UTmp.ml", line 145, characters 8-27
Called from Scan_CLI.replace_target_roots_by_regular_files_where_needed.(fun) in file "src/osemgrep/cli_scan/Scan_CLI.ml", lines 1086-1087, characters 19-65
Called from List_.fast_map in file "libs/commons/List_.ml", line 81, characters 17-20
Called from Scan_CLI.re

🔧 Infer (1.2.0)

src/Deferred.cpp

Usage Error: Failed to execute compilation command:
'/opt/infer-linux-x86_64-v1.2.0/lib/infer/infer/bin/../../facebook-clang-plugins/clang/install/bin/clang++'
-c src/Deferred.cpp -o /tmp/coderabbit-infer/9aac6da5ffddefdc/file.o

Error message:
clang++: error: no such file or directory: 'src/Deferred.cpp'

*** Infer needs a working compilation command to run.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

No actionable suggestions for changed features.

[SkrubbySkrubInAShrub]

looks fine to me

[coderabbitai]

🧹 Nitpick comments (1)

package/Shaders/DeferredCompositeCS.hlsl (1)

143-146: 💤 Low value

Minor: Comment is incomplete.

The comment says "(Lighting.hlsl only)" but RunGrass.hlsl also writes to Masks2. Consider updating to "(Lighting.hlsl and RunGrass.hlsl)" for accuracy.

Otherwise, the vertexAO recovery and SSGI normalization logic is correct.

📝 Suggested comment update

-	// Masks2.x stores 1 - vertexAO (Lighting.hlsl only); cleared to 0 for
+	// Masks2.x stores 1 - vertexAO (Lighting.hlsl, RunGrass.hlsl); cleared to 0 for
 	// pixels with no vertex AO contribution, so vertexAO defaults to 1.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@package/Shaders/DeferredCompositeCS.hlsl` around lines 143 - 146, Update the
inline comment above the vertexAO recovery to correctly list both writers of
Masks2: change "(Lighting.hlsl only)" to "(Lighting.hlsl and RunGrass.hlsl)" so
the comment accurately reflects that Masks2Texture is written by both
Lighting.hlsl and RunGrass.hlsl; leave the existing vertexAO calculation (float
vertexAO = 1.0 - Masks2Texture[dispatchID.xy].x;) and the ssgi normalization
(ssgiAo = saturate(ssgiAo / max(vertexAO, EPSILON_DIVISION));) unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@package/Shaders/DeferredCompositeCS.hlsl`:
- Around line 143-146: Update the inline comment above the vertexAO recovery to
correctly list both writers of Masks2: change "(Lighting.hlsl only)" to
"(Lighting.hlsl and RunGrass.hlsl)" so the comment accurately reflects that
Masks2Texture is written by both Lighting.hlsl and RunGrass.hlsl; leave the
existing vertexAO calculation (float vertexAO = 1.0 -
Masks2Texture[dispatchID.xy].x;) and the ssgi normalization (ssgiAo =
saturate(ssgiAo / max(vertexAO, EPSILON_DIVISION));) unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 681293f0-806b-4d57-8c60-95b79a3419b9

📥 Commits

Reviewing files that changed from the base of the PR and between 26c10c4 and 88b8872.

📒 Files selected for processing (4)

• package/Shaders/DeferredCompositeCS.hlsl
• package/Shaders/Lighting.hlsl
• package/Shaders/RunGrass.hlsl
• src/Deferred.cpp

[github-actions]

✅ A pre-release build is available for this PR:
Download

[jiayev]

you are creating a r16g16b16a16 for one channel, pls make use of one alpha