Repository navigation
fix(rocr): use runtime XNACK state for SVM_ACCESSIBLE_BY_DEFAULT query - #12558
Merged
Merged
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
dayatsin-amd
approved these changes
Sep 30, 2026
kjayapra-amd
force-pushed
the
users/kjayapra/rocrxnackbug
branch
from
October 9, 2026 18:25
99d1725 to
2fcbd6f
Compare
kjayapra-amd
marked this pull request as ready for review
October 9, 2026 18:27
kjayapra-amd
requested review from
a team,
atgutier,
kentrussell and
shwetagkhatri
as code owners
October 9, 2026 18:27
gandryey
approved these changes
Oct 9, 2026
On gfx1250, hipMemPrefetchAsync on pageable host memory always returns hipErrorNotSupported even when XNACK is enabled at runtime via HSA_XNACK=1 and the KFD kernel driver accepts the mode change. HSA_AMD_SYSTEM_INFO_SVM_ACCESSIBLE_BY_DEFAULT was checking the static ISA registry feature flag (GetXnack()) which is hardcoded as IsaFeature::Unsupported for gfx1250, instead of querying the actual runtime XNACK state negotiated with KFD during initialization. This caused a disconnect where: - KFD accepts XNACK=1 (hsaKmtSetXNACKMode succeeds) - rocminfo reports XNACK enabled: YES - But SVM_ACCESSIBLE_BY_DEFAULT returns false - CLR sets hmmCpuMemoryAccessible_=0 - hipMemPrefetchAsync on pageable memory returns hipErrorNotSupported Replace the static ISA feature loop with a query to Runtime::XnackEnabled(), which already holds the actual XNACK state set by BindXnackMode() at init time. This is consistent with how HSA_AMD_SYSTEM_INFO_XNACK_ENABLED already uses XnackEnabled(). Verified on gfx1250 (MI450): - HSA_XNACK=1: SVM_ACCESSIBLE_BY_DEFAULT now returns true - No HSA_XNACK: SVM_ACCESSIBLE_BY_DEFAULT correctly returns false - hipMemPrefetchAsync on pageable std::vector memory succeeds with HSA_XNACK=1 - hipMemPrefetchAsync correctly returns hipErrorNotSupported without HSA_XNACK Co-Authored-By: Claude <noreply@anthropic.com>
kjayapra-amd
force-pushed
the
users/kjayapra/rocrxnackbug
branch
from
October 10, 2026 11:10
2fcbd6f to
a09a0ec
Compare
systems-assistant Bot
pushed a commit
to ROCm/clr
that referenced
this pull request
Oct 13, 2026
SVM_ACCESSIBLE_BY_DEFAULT query (#12558) ## Motivation On gfx1250, hipMemPrefetchAsync on pageable host memory always returns hipErrorNotSupported even when XNACK is enabled at runtime via HSA_XNACK=1 and the KFD kernel driver accepts the mode change. ## Technical Details HSA_AMD_SYSTEM_INFO_SVM_ACCESSIBLE_BY_DEFAULT was checking the static ISA registry feature flag (GetXnack()) which is hardcoded as IsaFeature::Unsupported for gfx1250, instead of querying the actual runtime XNACK state negotiated with KFD during initialization. This caused a disconnect where: - KFD accepts XNACK=1 (hsaKmtSetXNACKMode succeeds) - rocminfo reports XNACK enabled: YES - But SVM_ACCESSIBLE_BY_DEFAULT returns false - CLR sets hmmCpuMemoryAccessible_=0 - hipMemPrefetchAsync on pageable memory returns hipErrorNotSupported ## Issue Tracking ROCM-30999 ## Test Plan Verified on gfx1250 (MI450): - HSA_XNACK=1: SVM_ACCESSIBLE_BY_DEFAULT now returns true - No HSA_XNACK: SVM_ACCESSIBLE_BY_DEFAULT correctly returns false - hipMemPrefetchAsync on pageable std::vector memory succeeds with HSA_XNACK=1 - hipMemPrefetchAsync correctly returns hipErrorNotSupported without HSA_XNACK ## Test Result Verified on gfx1250 ## Submission Checklist - [X ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. Co-authored-by: Karthik Jayaprakash <kjayapra@amd.com> Co-authored-by: Claude <noreply@anthropic.com> [rocm-systems] ROCm/rocm-systems#12558 (commit ae6a9a2)
systems-assistant Bot
pushed a commit
to ROCm/hip-tests
that referenced
this pull request
Oct 13, 2026
SVM_ACCESSIBLE_BY_DEFAULT query (#12558) ## Motivation On gfx1250, hipMemPrefetchAsync on pageable host memory always returns hipErrorNotSupported even when XNACK is enabled at runtime via HSA_XNACK=1 and the KFD kernel driver accepts the mode change. ## Technical Details HSA_AMD_SYSTEM_INFO_SVM_ACCESSIBLE_BY_DEFAULT was checking the static ISA registry feature flag (GetXnack()) which is hardcoded as IsaFeature::Unsupported for gfx1250, instead of querying the actual runtime XNACK state negotiated with KFD during initialization. This caused a disconnect where: - KFD accepts XNACK=1 (hsaKmtSetXNACKMode succeeds) - rocminfo reports XNACK enabled: YES - But SVM_ACCESSIBLE_BY_DEFAULT returns false - CLR sets hmmCpuMemoryAccessible_=0 - hipMemPrefetchAsync on pageable memory returns hipErrorNotSupported ## Issue Tracking ROCM-30999 ## Test Plan Verified on gfx1250 (MI450): - HSA_XNACK=1: SVM_ACCESSIBLE_BY_DEFAULT now returns true - No HSA_XNACK: SVM_ACCESSIBLE_BY_DEFAULT correctly returns false - hipMemPrefetchAsync on pageable std::vector memory succeeds with HSA_XNACK=1 - hipMemPrefetchAsync correctly returns hipErrorNotSupported without HSA_XNACK ## Test Result Verified on gfx1250 ## Submission Checklist - [X ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. Co-authored-by: Karthik Jayaprakash <kjayapra@amd.com> Co-authored-by: Claude <noreply@anthropic.com> [rocm-systems] ROCm/rocm-systems#12558 (commit ae6a9a2)
1 task done
Contributor
TheRock Submodule Bump ActivityNewest first
|
litvaOo
pushed a commit
that referenced
this pull request
Oct 21, 2026
#13430) ## Motivation `HipTest::isXnackOn()` and five tests detect XNACK by searching `gcnArchName` for `xnack+`. GPUs whose ISA has no XNACK on/off target feature (e.g. gfx1250) never report `xnack+`, so these tests always skip there, even with `HSA_XNACK=1`. ## Technical Details - `isXnackOn()` now returns `hipDeviceAttributePageableMemoryAccess`. Since #12558 this reflects the process XNACK mode on every GPU, and it is the capability these tests actually need (GPU access to `malloc`/`mmap` memory). - The five tests that duplicated the `gcnArchName` check now call `isXnackOn()`: - `hipMemCoherencyTstMProc.cc` - `hipHmmOvrSubscriptionTst.cc` - `hipHostRegisterStress.cc` - `hipMemAdvise_old.cc` - `hipGetProcAddressVmmApis.cc` - On NVIDIA `isXnackOn()` returns false; the NVIDIA backend never fills `gcnArchName`, so the old check could not match there either. ## Issue Tracking JIRA ID: AIRUNTIME-2926 ## Test Plan MI350X (gfx950), ROCr/CLR built from develop `ae6a9a29dd`. Ran all `isXnackOn()`-gated tests, unpatched develop vs this change, with `HSA_XNACK` unset and `HSA_XNACK=1`. ## Test Result Identical results with and without the change: | Test | `HSA_XNACK` unset | `HSA_XNACK=1` | |---|---|---| | `Unit_malloc_CoherentTst`, `Unit_malloc_CoherentTstWthAdvise` | skip | pass | | `Unit_mmap_CoherentTst`, `Unit_mmap_CoherentTstWthAdvise` | skip | pass | | `Unit_hipMemAdvise_TstAlignedAllocMem` | skip | pass | | `Unit_hipMemAdvise_TstAlignedAllocMem_XNACK` | pass | pass | | `Unit_hipGetProcAddress_VMM` | pass | pass | | `Stress_HMM_OverSubscriptionTst`, `Stress_hipHostRegister_Oversubscription` | skip | not run | The two stress tests were not run with `HSA_XNACK=1` because they oversubscribe host memory (~1.2× GPU memory). Their gate is the same `isXnackOn()` call as the tests above. Not tested on gfx1250. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Co-authored-by: Claude <noreply@anthropic.com>
dgaliffiAMD
pushed a commit
that referenced
this pull request
Oct 21, 2026
#13430) ## Motivation `HipTest::isXnackOn()` and five tests detect XNACK by searching `gcnArchName` for `xnack+`. GPUs whose ISA has no XNACK on/off target feature (e.g. gfx1250) never report `xnack+`, so these tests always skip there, even with `HSA_XNACK=1`. ## Technical Details - `isXnackOn()` now returns `hipDeviceAttributePageableMemoryAccess`. Since #12558 this reflects the process XNACK mode on every GPU, and it is the capability these tests actually need (GPU access to `malloc`/`mmap` memory). - The five tests that duplicated the `gcnArchName` check now call `isXnackOn()`: - `hipMemCoherencyTstMProc.cc` - `hipHmmOvrSubscriptionTst.cc` - `hipHostRegisterStress.cc` - `hipMemAdvise_old.cc` - `hipGetProcAddressVmmApis.cc` - On NVIDIA `isXnackOn()` returns false; the NVIDIA backend never fills `gcnArchName`, so the old check could not match there either. ## Issue Tracking JIRA ID: AIRUNTIME-2926 ## Test Plan MI350X (gfx950), ROCr/CLR built from develop `ae6a9a29dd`. Ran all `isXnackOn()`-gated tests, unpatched develop vs this change, with `HSA_XNACK` unset and `HSA_XNACK=1`. ## Test Result Identical results with and without the change: | Test | `HSA_XNACK` unset | `HSA_XNACK=1` | |---|---|---| | `Unit_malloc_CoherentTst`, `Unit_malloc_CoherentTstWthAdvise` | skip | pass | | `Unit_mmap_CoherentTst`, `Unit_mmap_CoherentTstWthAdvise` | skip | pass | | `Unit_hipMemAdvise_TstAlignedAllocMem` | skip | pass | | `Unit_hipMemAdvise_TstAlignedAllocMem_XNACK` | pass | pass | | `Unit_hipGetProcAddress_VMM` | pass | pass | | `Stress_HMM_OverSubscriptionTst`, `Stress_hipHostRegister_Oversubscription` | skip | not run | The two stress tests were not run with `HSA_XNACK=1` because they oversubscribe host memory (~1.2× GPU memory). Their gate is the same `isXnackOn()` call as the tests above. Not tested on gfx1250. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On gfx1250, hipMemPrefetchAsync on pageable host memory always returns hipErrorNotSupported even when XNACK is enabled at runtime via HSA_XNACK=1 and the KFD kernel driver accepts the mode change.
Technical Details
HSA_AMD_SYSTEM_INFO_SVM_ACCESSIBLE_BY_DEFAULT was checking the static ISA registry feature flag (GetXnack()) which is hardcoded as IsaFeature::Unsupported for gfx1250, instead of querying the actual runtime XNACK state negotiated with KFD during initialization.
This caused a disconnect where:
Issue Tracking
ROCM-30999
Test Plan
Verified on gfx1250 (MI450):
Test Result
Verified on gfx1250
Submission Checklist