fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools by prihosur-amd · Pull Request #11355 · ROCm/rocm-systems

prihosur-amd · 2026-09-09T08:11:24Z

Motivation

Device-SVM/AIE memory pools advertised the wrong maximum allocation
size, causing concurrent allocate/free stress to fail on AIE pools
(e.g. aie2p Pool 8) on gfx1151 (Strix Halo APU).

Technical Details

MemoryRegion::GetInfo() returned max_sysmem_alloc_size_ for
HSA_HEAPTYPE_DEVICE_SVM regions instead of the pool's own
max_single_alloc_size_. MemoryRegion::AllocateImpl() had the same
issue in its IsSystem() size-cap check. As a result, device-SVM/AIE
pools advertised the system-memory allocation limit instead of their
real per-pool limit, letting allocators request sizes the pool cannot
satisfy and fail with HSA_STATUS_ERROR_OUT_OF_RESOURCES.

Fix: use max_single_alloc_size_ for HSA_HEAPTYPE_DEVICE_SVM in both
GetInfo()'s HSA_REGION_INFO_ALLOC_MAX_SIZE case and
AllocateImpl()'s size-cap check, matching the value already used for
local-memory allocations.

Issue Tracking

JIRA ID: ROCM-21660

Test Plan

Built hsa-runtime64 with the fix and ran the rocrtst reproducer
before/after on gfx1151 (Strix Halo APU) hardware.

Test Result

Test	Before	After
`rocrtstStress.Memory_Concurrent_Allocate_Test`	FAIL (`HSA_STATUS_ERROR_OUT_OF_RESOURCES`, Pool 8)	PASS
`rocrtstStress.Memory_Concurrent_Free_Test`	FAIL (`HSA_STATUS_ERROR_OUT_OF_RESOURCES`, Pool 8)	PASS

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

therock-pr-bot · 2026-09-09T08:12:24Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like `test_<name>.py` / `test_<name>.cpp` (or `<name>_test.`). Current:* code file(s) changed: `projects/rocr-runtime/runtime/hsa-runtime/core/driver/xdna/amd_xdna_driver.cpp`, `projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_aie_agent.cpp`, `projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_memory_region.cpp`; no test file found
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-09-09T08:12:26Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

…SVM pools MemoryRegion::GetInfo() and AllocateImpl() used max_sysmem_alloc_size_ for HSA_HEAPTYPE_DEVICE_SVM regions, so device-SVM/AIE pools advertised the system-memory allocation cap instead of their own per-pool limit. This let allocators request sizes the pool cannot actually satisfy, which fails during concurrent alloc/free on AIE pools (e.g. aie2p Pool 8) with HSA_STATUS_ERROR_OUT_OF_RESOURCES. Report max_single_alloc_size_ for HSA_HEAPTYPE_DEVICE_SVM instead, matching the same value already used in AllocateImpl()'s own local-memory size check. Verified on gfx1151 (Strix Halo APU) with rocrtst: - rocrtstStress.Memory_Concurrent_Allocate_Test: FAIL -> PASS - rocrtstStress.Memory_Concurrent_Free_Test: FAIL -> PASS

shwetagkhatri

LGTM.

shwetagkhatri

These are non blocking comments - but while you are owning this change - i would like to fix the following corresponding files as well -

2 comments in other files below are - now saying the opposite of what the code is doing and whoever reads them next - is going to think this bug is still open.

amd_aie_agent.cpp -
is currently saying:
// The system will report the max allocatable size as MemoryRegion::max_sysmem_alloc_size_, but
// this is incorrect. The pool is exactly XdnaDriver::GetDevHeapByteSize() bytes.

this can be fixed now:

// The pool reports its max allocatable size as MemoryRegion::max_single_alloc_size_, which is exactly
// XdnaDriver::GetDevHeapByteSize() bytes.

amd_xdna_driver.cpp -
is currently saying:

// While this is already checked in MemoryRegion::AllocateImpl, the max size is
// MemoryRegion::max_sysmem_alloc_size_ for HSA_HEAPTYPE_DEVICE_SVM which is incorrect
// for dev heap.

can be fixed now:

  // MemoryRegion::AllocateImpl already caps this at MemoryRegion::max_single_alloc_size_, which
  // is dev_heap_size for HSA_HEAPTYPE_DEVICE_SVM. Kept as a defensive duplicate returning the
  // same status, so an oversized CREATE_BO cannot reach the driver if that cap ever regresses.

shwetagkhatri · 2026-10-01T18:07:50Z

        case HSA_HEAPTYPE_FRAME_BUFFER_PRIVATE:
        case HSA_HEAPTYPE_FRAME_BUFFER_PUBLIC:
        case HSA_HEAPTYPE_GPU_SCRATCH:
          *((size_t*)value) = max_single_alloc_size_;
          break;


Actually looking into the full source code - the new lines 238-240 have the code body same as 241-245. Please fold the label into the existing group here as below:

case HSA_HEAPTYPE_DEVICE_SVM: case HSA_HEAPTYPE_FRAME_BUFFER_PRIVATE: case HSA_HEAPTYPE_FRAME_BUFFER_PUBLIC: case HSA_HEAPTYPE_GPU_SCRATCH: *((size_t*)value) = max_single_alloc_size_; break;

cfreeamd

🤖 AI-assisted pre-review (for @cfreeamd)

Verdict: approve-with-nits — the fix is correct. One Required item on the PR title before merge.

What this PR does

Fixes two related bugs where HSA_HEAPTYPE_DEVICE_SVM pools were using the system-memory allocation cap (max_sysmem_alloc_size_) instead of the device allocation cap (max_single_alloc_size_):

GetInfo(HSA_REGION_INFO_ALLOC_MAX_SIZE) — HSA_HEAPTYPE_DEVICE_SVM was grouped with HSA_HEAPTYPE_SYSTEM and returned max_sysmem_alloc_size_. Now split out and returns max_single_alloc_size_.
AllocateImpl() — The IsSystem() allocation-size guard was using max_sysmem_alloc_size_ even when IsDeviceSVM() is true. Fixed to use max_single_alloc_size_ when IsDeviceSVM().

Both sites are now consistent: HSA_HEAPTYPE_DEVICE_SVM is VRAM-backed (device memory), so reporting and enforcing max_single_alloc_size_ is correct.

[Required] PR title missing `type(component):` format

Current title: rocr: fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools

Suggested: fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools

ROCm squash-merges, so the PR title becomes the permanent commit title. The type(component): prefix is required by the commit convention.

What looks correct

Both GetInfo() and AllocateImpl() are updated consistently — they now agree on which cap applies to HSA_HEAPTYPE_DEVICE_SVM.
The IsDeviceSVM() check in AllocateImpl() is correctly placed before the max_sysmem_alloc_size_ guard, so the ternary selects the right cap.
No performance impact — both changed paths are in the cold allocation path.

cfreeamd · 2026-10-05T21:09:36Z

Please address Shweta's comment and change the PR title to the github friendly format; e.g.,

fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools

Otherwise, LGTM.

…ax-size

…loc-max-size

ypapadop-amd

LGTM, only one comment.

For some context, I was hesitant to change what SVM was reporting for size (thus the comments in XdnaDriver) because I did not have any insight on if someone was expecting the system memory as a result.

…loc-max-size

lamikr · 2026-10-29T03:05:21Z

Merging, only failed test is the system-pr bot which failed for existing issue: rocm-systems #11156

(open, created 2026-09-02): "therock-pr-bot fails on every develop PR:
required 'pre-commit' check is never published (pre-formatting.yml is gated to testbranch)".****

systems-assistant · 2026-10-30T04:03:05Z

TheRock Submodule Bump Activity

Newest first

2026-10-30 — Included in TheRock via ROCm/TheRock#9592.

prihosur-amd requested review from atgutier, cfreeamd, dayatsin-amd, kentrussell and shwetagkhatri as code owners September 9, 2026 08:11

github-actions Bot added the project: rocr-runtime label Sep 9, 2026

therock-pr-bot Bot added the Not ready to Review label Sep 9, 2026

systems-assistant Bot added the organization: ROCm label Sep 9, 2026

prihosur-amd force-pushed the users/prihosur-amd/fix-device-svm-alloc-max-size branch from 169aa7f to e4f386c Compare September 9, 2026 10:12

shwetagkhatri approved these changes Oct 1, 2026

View reviewed changes

shwetagkhatri self-requested a review October 1, 2026 18:05

shwetagkhatri reviewed Oct 1, 2026

View reviewed changes

shwetagkhatri self-requested a review October 1, 2026 18:32

cfreeamd reviewed Oct 5, 2026

View reviewed changes

prihosur-amd added 2 commits October 8, 2026 18:53

fix(rocr): correct Device-SVM allocation limit comments

a2d4d35

Merge upstream/develop into users/prihosur-amd/fix-device-svm-alloc-m…

5c243ed

…ax-size

prihosur-amd requested a review from ypapadop-amd as a code owner October 8, 2026 13:36

Merge branch 'ROCm:develop' into users/prihosur-amd/fix-device-svm-al…

689c6c9

…loc-max-size

ypapadop-amd approved these changes Oct 8, 2026

View reviewed changes

Comment thread projects/rocr-runtime/runtime/hsa-runtime/core/driver/xdna/amd_xdna_driver.cpp Resolved

prihosur-amd changed the title ~~rocr: fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools~~ fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools Oct 8, 2026

prihosur-amd added 2 commits October 8, 2026 20:31

fix(rocr): remove redundant Device-SVM size check

624adb2

Merge branch 'ROCm:develop' into users/prihosur-amd/fix-device-svm-al…

fa3fe4f

…loc-max-size

cfreeamd approved these changes Oct 9, 2026

View reviewed changes

prihosur-amd removed the Not ready to Review label Oct 12, 2026

Merge branch 'ROCm:develop' into users/prihosur-amd/fix-device-svm-al…

336134d

…loc-max-size

dayatsin-amd approved these changes Oct 26, 2026

View reviewed changes

lamikr merged commit 415459e into ROCm:develop Oct 29, 2026
1086 of 1184 checks passed

Navigation Menu

Repository navigation

fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools - #11355

fix(rocr): fix incorrect ALLOC_MAX_SIZE reporting for HSA_HEAPTYPE_DEVICE_SVM pools#11355
lamikr merged 7 commits into
ROCm:developfrom
prihosur-amd:users/prihosur-amd/fix-device-svm-alloc-max-size

prihosur-amd commented Sep 9, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026

Uh oh!

shwetagkhatri left a comment

Uh oh!

shwetagkhatri left a comment

Uh oh!

shwetagkhatri Oct 1, 2026

Uh oh!

prihosur-amd Oct 8, 2026

Uh oh!

cfreeamd left a comment

Uh oh!

cfreeamd commented Oct 5, 2026

Uh oh!

ypapadop-amd left a comment

Uh oh!

Uh oh!

lamikr commented Oct 29, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

prihosur-amd commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Sep 9, 2026

Uh oh!

shwetagkhatri left a comment

Choose a reason for hiding this comment

Uh oh!

shwetagkhatri left a comment

Choose a reason for hiding this comment

Uh oh!

shwetagkhatri Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

prihosur-amd Oct 8, 2026

Choose a reason for hiding this comment

Uh oh!

cfreeamd left a comment

Choose a reason for hiding this comment

What this PR does

[Required] PR title missing type(component): format

What looks correct

Uh oh!

cfreeamd commented Oct 5, 2026

Uh oh!

ypapadop-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lamikr commented Oct 29, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 30, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

prihosur-amd commented Sep 9, 2026 •

edited

Loading

therock-pr-bot Bot commented Sep 9, 2026 •

edited

Loading

[Required] PR title missing `type(component):` format