Skip to content

fix: heap corruption#1895

Closed
SimplisticMind wants to merge 4 commits into
community-shaders:devfrom
SimplisticMind:dev
Closed

fix: heap corruption#1895
SimplisticMind wants to merge 4 commits into
community-shaders:devfrom
SimplisticMind:dev

Conversation

@SimplisticMind
Copy link
Copy Markdown

@SimplisticMind SimplisticMind commented Feb 17, 2026

Partially fixes heap corruption; I don't have ability to test PBR materials. Tested by using AppVerifier and going ingame.
Fixes #1654

Summary by CodeRabbit

  • Refactor
    • Changed how materials are allocated for non-menu screens by switching to a different allocation mechanism; allocation for menu screens remains unchanged. Overall material linking and runtime behavior are preserved.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Feb 17, 2026

No actionable comments were generated in the recent review. 🎉


📝 Walkthrough

Walkthrough

Replaced direct material allocation in the non-menu branch of BSLightingShaderProperty_LoadBinary with a relocation-resolved CreateOnScrapHeap call that allocates the material on Skyrim's Thread ScrapHeap (8-byte alignment); menu-screen path and subsequent linking/state updates remain unchanged.

Changes

Cohort / File(s) Summary
Material allocation change
src/TruePBR.cpp
Replaced RE::BSLightingShaderMaterialBase::CreateMaterial(feature) with a relocation-resolved CreateOnScrapHeap invocation for non-menu-screen material allocation (uses Thread ScrapHeap, align=8). Control flow, linking, and state updates unchanged; menu branch unchanged.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested reviewers

  • alandtse
  • doodlum

Poem

🐇 I dug a tiny heap below,
Where eight-byte dreams can safely grow.
Materials snug in scrapbed light,
I hopped away — the pointers right.
A happy hop, the code says so.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix: heap corruption' directly matches the primary objective in the linked issue #1654, clearly summarizing the main change in addressing heap corruption.
Linked Issues check ✅ Passed The code change replaces direct material allocation with a relocation-based CreateOnScrapHeap function pointer call, which addresses the heap corruption issue identified in #1654 by changing how memory is allocated for shader materials.
Out of Scope Changes check ✅ Passed The changes are focused solely on modifying the allocation mechanism in BSLightingShaderProperty_LoadBinary, which is directly scoped to fixing the heap corruption issue identified in #1654.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 17, 2026

Using provided base ref: a056641
Using base ref: a056641
Base commit date: 2026-02-19T14:36:45Z (Thursday, February 19, 2026 02:36 PM)
No actionable suggestions for changed features.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 17, 2026

✅ A pre-release build is available for this PR:
Download

@alandtse
Copy link
Copy Markdown
Collaborator

Thanks. Perhaps this needs to be fixed on the comonlib side actually. I'll investigate since it is using some of our internal RE (that should be upstream instead).

@alandtse
Copy link
Copy Markdown
Collaborator

@doodlum seems plausible, but thoughts?

@doodlum
Copy link
Copy Markdown
Collaborator

doodlum commented Feb 17, 2026

i thought clib fixed this. this method looks fine, just the relid should be in globals

@alandtse
Copy link
Copy Markdown
Collaborator

alandtse commented Feb 17, 2026

i thought clib fixed this. this method looks fine, just the relid should be in globals

@doodlum Was this your PR? We should be on it now.

@doodlum
Copy link
Copy Markdown
Collaborator

doodlum commented Feb 17, 2026

yeah

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/TruePBR.cpp (1)

569-572: Commit title: add a scope for clarity

The current title fix: heap corruption follows conventional commits but the scope is absent. A scoped title more precisely conveys the change:

fix(pbr): use scrapheap for material allocation in LoadBinary

As per coding guidelines on Conventional Commit Titles: type(scope): description, max 50 characters.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/TruePBR.cpp` around lines 569 - 572, Update the commit title to include a
scope and concise description following Conventional Commits (type(scope):
description) so it reads like "fix(pbr): use scrapheap for material allocation
in LoadBinary" (keep under ~50 chars); this change is purely the commit message
for the change around
CreateOnScrapHeap/RE::BSLightingShaderMaterialBase::CreateMaterial allocation in
LoadBinary where material is assigned, so amend the existing commit message (git
commit --amend or interactive rebase) to the scoped title and push/update the
PR.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/TruePBR.cpp`:
- Around line 562-567: The PBR branch is allocating BSLightingShaderMaterialPBR
via BSLightingShaderMaterialPBR::Make() which currently uses RE::malloc with
align=0 instead of allocating from the Thread ScrapHeap with align=8, leaving
LinkObject/Deallocate vulnerable to crashes; update the allocation path for
BSLightingShaderMaterialPBR (the Make/constructor path used in the
kMenuScreen/PBR branch) to allocate from the Thread ScrapHeap with align=8
(matching the fix applied to the other material branch), ensure any
corresponding deallocation uses the ScrapHeap-aware Deallocate/LinkObject logic,
and add a short comment referencing Thread ScrapHeap align=8 for future
reviewers.

---

Nitpick comments:
In `@src/TruePBR.cpp`:
- Around line 569-572: Update the commit title to include a scope and concise
description following Conventional Commits (type(scope): description) so it
reads like "fix(pbr): use scrapheap for material allocation in LoadBinary" (keep
under ~50 chars); this change is purely the commit message for the change around
CreateOnScrapHeap/RE::BSLightingShaderMaterialBase::CreateMaterial allocation in
LoadBinary where material is assigned, so amend the existing commit message (git
commit --amend or interactive rebase) to the scoped title and push/update the
PR.

Comment thread src/TruePBR.cpp
Comment thread src/TruePBR.cpp
{
RE::BSLightingShaderMaterialBase* material = nullptr;
if (property->flags.any(kMenuScreen)) {
// Note: Same bug as below, should be using Thread ScrapHeap with align=8 instead of RE::malloc with align=0
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@claude can you please add the implementation as needed? Please also determine if this fix should actually be PR upstream to commonlib which we will sycn with via our commonlib. https://github.com/powerof3/CommonLibSSE

@alandtse
Copy link
Copy Markdown
Collaborator

Ok, confirmed this is fixed from commonlib already. Will do another PR to handle the PBR path.

@SimplisticMind thanks for filing the issue and the pr.

@alandtse alandtse closed this Feb 22, 2026
alandtse added a commit to alandtse/open-shaders that referenced this pull request Feb 22, 2026
PBR material Make() was using operator new (via TES_HEAP_REDEFINE_NEW),
which allocates from the global heap with RE::malloc. The game expects
materials to be allocated from the Thread ScrapHeap with align=8.
This mismatch causes heap corruption when materials are later freed
via scrap heap deallocation paths.

Switch to scrapHeap->Allocate + std::construct_at, matching the
pattern used by CommonLib's BSLightingShaderMaterialBase::CreateMaterial

Ref: community-shaders#1895, community-shaders#1654
alandtse added a commit to alandtse/open-shaders that referenced this pull request Feb 22, 2026
PBR material Make() was using operator new (via TES_HEAP_REDEFINE_NEW),
which allocates from the global heap with RE::malloc. The game expects
materials to be allocated from the Thread ScrapHeap with align=8.
This mismatch causes heap corruption when materials are later freed
via scrap heap deallocation paths.

Switch to scrapHeap->Allocate + std::construct_at, matching the
pattern used by CommonLib's BSLightingShaderMaterialBase::CreateMaterial

Ref: community-shaders#1895, community-shaders#1654
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Heap corruption

3 participants