fix(emat): allow solid-black height-only masks

[FIocker]

• Adds an explicit exception for perfectly solid-black RGB masks. This allows height-only alpha masks generated by tools like PG Patcher to successfully trigger Complex Parallax, while keeping the safeguard intact for vanilla grayscale environment masks to protect against Elven Shield and Shield of Ysgramor texture issue #1969.

• Fixes 1.5 RC2 - Missing parallax on decal flagged TriShapes #2201 and CS 1.4.11 -> 1.5.1 causes some complex material not to show parallax #2343

Summary by CodeRabbit

• Bug Fixes
  • Improved material detection — refined env-mask sampling to more reliably identify complex materials, reducing misclassification and improving rendering consistency.
  • Texture compatibility — adjusted texture sampling logic to maintain height-only/parallax effects for specially patterned masks (near-black height masks) while preventing false positives from grayscale textures.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1ffa2b7e-1b6d-4cb2-a6c4-a63827e05694

📥 Commits

Reviewing files that changed from the base of the PR and between 0427e19 and 6467601.

📒 Files selected for processing (1)

• package/Shaders/Lighting.hlsl

🚧 Files skipped from review as they are similar to previous changes (1)

• package/Shaders/Lighting.hlsl

---

📝 Walkthrough

Walkthrough

This PR changes envmask complex-material detection to sample a fixed mip (SampleLevel(..., uv, 15)), set complexMaterial from the sampled alpha vs epsilon, and disable complexMaterial when the sampled RGB is grayscale except for near-black RGB with in-range alpha (height-only masks).

Changes

Complex Material Detection Fix

Layer / File(s)	Summary
Mip-alpha based complex material detection and grayscale-RGB disabling package/Shaders/Lighting.hlsl	Complex material is gated by mipSample.w sampled at mip level 15 against kMaskEpsilon. Adds grayscale-RGB detection that disables complexMaterial unless RGB is near-black while alpha remains in-range (preserving height-only masks).

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• community-shaders/skyrim-community-shaders#2422: Modifies the same envmask complex-material detection to use .w mip-level sampling and related gating.
• community-shaders/skyrim-community-shaders#2032: Related changes addressing grayscale/"wrong format" envmask detection and complexMaterial gating.

Suggested reviewers

• doodlum
• jiayev
• alandtse

Poem

🐰
I sampled deep at mip fifteen,
Alpha told me what the channels mean,
When RGB matches, complex fades away,
Unless black height masks quietly stay,
A rabbit nods — small tweaks brighten play.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(emat): allow solid-black height-only masks' is specific and directly related to the main change—enabling solid-black height-only masks in environment mask complex-material detection.
Linked Issues check	✅ Passed	The code change addresses the core objective from issue #2201 by allowing height-only masks (solid-black RGB with valid alpha) to trigger Complex Parallax while preserving the safeguard from issue #1969 against vanilla grayscale masks.
Out of Scope Changes check	✅ Passed	All changes in Lighting.hlsl are scoped to environment mask complex-material detection logic and directly support the linked objectives; no extraneous modifications are present.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

No actionable suggestions for changed features.

[doodlum]

Does #1969 still get fixed?

[FIocker]

Does #1969 still get fixed?

Yes of course always has I didn't edit this after the fact.