fix(TAA): banding and oversharpening

[davo0411]

fixes bugs in TAA. Rewritten from raw .hlsl again, similar to alandtse's pr but I could not address the bugs in that pr so I started over.

Dev has above lines etc.
this pr fixes it
closes #2371 & #2375

Summary by CodeRabbit

• Enhancement
  • Rewritten temporal anti-aliasing using packed-neighborhood sampling for more stable reconstruction.
  • Dynamic-resolution UV clamping and improved neighbor/center sampling for consistent quality across resolutions.
  • Depth-guided UV selection and stronger disocclusion/mask rejection to reduce ghosting and better handle occlusions.
  • Refined temporal blending, flicker scoring, clamping and sharpening to reduce temporal instability.
  • HDR-aware color conversion and VR-specific output handling; feedback output consistently enabled.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Rewrites ISTemporalAA.hlsl from a phase-based 9-tap TAA to a packed 3x3 neighbor pipeline with dynamic-resolution UV clamping, depth-guided reprojection, luminance bracket/flicker scoring, temporal blend/clamp/sharpen, and motion/disocclusion/mask rejection; PS entry/output remain unchanged.

Changes

Temporal Anti-Aliasing Rewrite

Layer / File(s)	Summary
Resources, HDR helpers, and macro package/Shaders/ISTemporalAA.hlsl	Defines PS outputs, bindings, introduces cmp macro and HDR conversion/feedback encode-decode helpers.
Luma weights and dynamic-resolution UV clamping package/Shaders/ISTemporalAA.hlsl	Adds kLumaWeights, ClampScreenUV/ClampScreenUV4/ClampHistoryUV, and GetDynamicResolutionMax.
Packed neighbor data model and sampling helpers package/Shaders/ISTemporalAA.hlsl	Introduces ISTAA_NeighborTap, LoadNeighborGRB/SampleNeighborGRB/PackNeighborTap/AssignPackedNeighbor, and SampleCenterRGB.
Alpha/coverage, flicker scoring, shallowest-depth helper package/Shaders/ISTemporalAA.hlsl	Adds AlphaCoverageMask, FlickerLumaContribution, and PickIfShallowestUV.
Depth-guided UV selection package/Shaders/ISTemporalAA.hlsl	Implements SelectDepthGuidedUV to clamp 3x3 candidate UVs, sample depth, select shallowest reprojection UV, and return clamped neighbor sets plus corner GRB.
Main pipeline orchestration package/Shaders/ISTemporalAA.hlsl	Main body computes motion/reprojection, decodes history (HDR path), assembles packed neighbor taps and center RGB, and seeds luminance min/max brackets.
Temporal bracket, flicker-informed blend and sharpen package/Shaders/ISTemporalAA.hlsl	Builds luminance AABB/brackets from packed data, applies flicker scoring, and computes temporal blend/clamp/sharpen intermediate color.
Disocclusion/mask rejection and output composition package/Shaders/ISTemporalAA.hlsl	Applies velocity/mask/alpha coverage gates across neighborhood UVs, performs rejection-controlled selection, forces feedbackOut.w = 1, and sets colorOut.w per VR/motion gate.
Final HDR conversion and target writes package/Shaders/ISTemporalAA.hlsl	Applies ConvertRenderOutput when HDR_OUTPUT is enabled and writes psout.Color and psout.Feedback.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• community-shaders/skyrim-community-shaders#2086: Refactors ISTemporalAA.hlsl into a decompile-style pipeline and overlaps on HDR/TAA helper changes.
• community-shaders/skyrim-community-shaders#2122: Modifies luma-based neighborhood history clamping/bracket logic related to flicker/bracket selection.

Suggested reviewers

• alandtse
• doodlum
• soda3000

Poem

🐰 I hop through neighbor taps and depth so deep,
I clamp the UVs where histories sleep,
Brackets hush flicker, motion finds its gate,
Feedback locked true, the frame looks great,
A rabbit nods—this shader’s neat.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(TAA): banding and oversharpening' directly addresses the primary issues fixed in this changeset—banding artifacts and unwanted sharpening in the TAA shader.
Linked Issues check	✅ Passed	The rewritten TAA shader addresses both linked issue #2371 requirements: eliminating banding artifacts and oversharpening through a new packed-neighborhood pipeline with improved depth-guided reprojection and temporal blend logic.
Out of Scope Changes check	✅ Passed	All changes in the shader are scoped to fixing TAA banding and sharpening issues; no unrelated modifications or scope creep are present.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

No actionable suggestions for changed features.

[Copilot]

Pull request overview

This PR rewrites the BSImagespaceShaderISTemporalAA pixel shader to more closely mirror the vanilla/decompiled TAA implementation, aiming to fix banding artifacts and unintended oversharpening when Imagespace is toggled (issues #2371 / #2375 context).

Changes:

• Replaces the prior higher-level/annotated TAA implementation with “verbatim” decompile-style math and register packing/swizzle behavior.
• Reworks neighborhood sampling, history reprojection, and rejection logic (depth-guided selection, mask/alpha gating, flicker score, sharpening stages).
• Refactors HDR conversion handling by adding local PQ conversion helpers and applying them around sampling/output.

Comments suppressed due to low confidence (1)

package/Shaders/ISTemporalAA.hlsl:88

• ClampHistoryUV mirrors FrameBuffer::GetPreviousDynamicResolutionAdjustedScreenPosition but does not apply VR eye-half clamping. In VR this can allow history sampling from the opposite eye, which typically shows up as temporal smear/ghosting at the eye boundary. Suggest using the FrameBuffer helper or incorporating the same VR clamp logic.

float2 ClampHistoryUV(float2 reprojectedUV)
{
	float2 uv = max(FrameBuffer::DynamicResolutionParams1.zw * reprojectedUV, float2(0, 0));
	uv.x = min(FrameBuffer::DynamicResolutionParams2.w, uv.x);
	uv.y = min(FrameBuffer::DynamicResolutionParams1.w, uv.y);
	return uv;
}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@package/Shaders/ISTemporalAA.hlsl`:
- Around line 455-467: The alpha-coverage gating logic for motionReject misses
the drUVMax/corner sample from SelectDepthGuidedUV (loaded into corner/drUVMax),
so a fully-covered path can be taken when only that corner is uncovered; update
the AlphaCoverageMask checks (the sequence using AlphaCoverageMask on
drNeighborsA/B/C and sampleUV.z) to also call AlphaCoverageMask on the
drUVMax/corner UVs and AND that result into motionReject.x (and/or
motionReject.w as appropriate) so the corner tap is required before allowing the
covered path. Ensure you reference motionReject, AlphaCoverageMask,
drUVMax/corner, and SelectDepthGuidedUV when making the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: eb2472cb-6743-4cca-9302-997e6c36c5d3

📥 Commits

Reviewing files that changed from the base of the PR and between 377da5f and b98fe57.

📒 Files selected for processing (1)

• package/Shaders/ISTemporalAA.hlsl

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@package/Shaders/ISTemporalAA.hlsl`:
- Around line 439-449: The HDR luma-difference scale is wrong: in the HDR branch
where motionReject.z is computed and lumaDiffScaled is set, replace the
multiplier 0.05 with 2.5 so the downstream history.xw computation (which uses
lumaDiffScaled.xx * float2(20,100)) produces the intended ~50 and ~250
coefficients; update the scalar in the block that computes lumaDiffScaled (refer
to motionReject.z, history.xw, centerMeta.x) accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9c579a1d-e061-49bf-955a-778f78f881fb

📥 Commits

Reviewing files that changed from the base of the PR and between 0bed5d2 and 7bff639.

📒 Files selected for processing (1)

• package/Shaders/ISTemporalAA.hlsl

[SkrubbySkrubInAShrub]

lgtm