Repository navigation
feat(rocr/libhsakmt/dxg): map DXG system memory across GPUs - #13442
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
Map system and user memory onto another adapter at the owner's VA. The peer view does not free that reservation.
Record each GPU's HSA node when the topology snapshot is taken. Callers that still need a single device use the first entry.
When the caller does not name a node, map system memory, user pointers, and dmabuf imports onto every default GPU at the owner's VA.
Submit the page-table update to the device named by NodeId, and use that device's peer view when it does not own the allocation.
982e7fd to
fdd7881
Compare
|
ping? |
PR Pundit review(Testing PR Pundit Code Review agent. Not all comments may be relevant. This is a one-time automated pass: please address what is useful, no need to request a re-review. A quick reply per point (useful or not, and why) helps us calibrate the agent. Reach out to @peymanr for suggestions and feedback.)
1. 🔴 Blocking: Multi-GPU DXG: mapping VRAM/dma-buf to default nodes fails, this PR fans default_node to all GPUs plus VRAM-unsupported peer views
On multi-GPU DXG/WSL hosts, How it is reached: Multi-GPU DXG (Windows) system: hipDeviceEnablePeerAccess / hsa_amd_agents_allow_access on a VRAM pointer -> MemoryRegion::AllowAccess (local region, whitelist = owner + peer GPUs) -> KfdDriver::MakeMemoryResident -> hsaKmtMapMemoryToGPUNodes -> FindAllocation hits a non-userptr kLocal allocation -> map_to_nodes (memory.cpp:921) -> GpuMemory::ViewOn(peer) -> MapFromPeer returns UnSupported -> HSAKMT_STATUS_ERROR. Likewise GpuAgent scratch/reserved MakeMemoryResident -> hsaKmtMapMemoryToGPU with default_node = all GPUs -> same branch on kLocal scratch memory. Who is affected: Every user of ROCm on a Windows/DXG host with two or more GPUs: peer access to VRAM, scratch reservation and default-node VRAM dma-buf registration now fail where the base succeeded. Checked against the code by an independent pass. 2. 🟠 Should fix: VaUnmap leaves stale peer view in peers_: this PR does not clear peer bookkeeping on unmap, causing silent no-op remap and double-unmap
Who is affected: Callers that VaMap a kSystem/kUserMemory handle on a non-owner GPU at the owner's VA, then VaUnmap and VaMap it again on that GPU, without hsakmt_debug_sysmem. Checked against the code by an independent pass. 3. 🟠 Should fix: Cross-GPU peer map plus paging-fence wait runs under process-wide allocation_map_lock_, serializing other threads' memory calls
Every Who is affected: Multi-GPU WSL/DXG processes that map system or user memory to several GPUs while other threads allocate, map or free memory. Checked against the code by an independent pass. 4. 🟠 Should fix: DropPeer evicts and deletes peer while its unmap may still be pending: this PR omits the paging-fence wait VaUnmap performs
Who is affected: Windows/WSL DXG users whose system or userptr memory is mapped to more than one GPU, when peer views are torn down. Checked against the code by an independent pass. 5. 🟠 Should fix: Imported system-memory peer view may not be backed by the resource's pages: this PR copies owner desc_ wholesale in MapFromPeer
Who is affected: Multi-GPU WSL systems that map imported Checked against the code by an independent pass. Shared from a pilot of PR Pundit, an agentic code reviewer that builds and tests the change and reads the project's past reviews. Reviewed at |
TheRock Submodule Bump ActivityNewest first
|
## Motivation Follow-up to #13442. After `default_node` started listing every GPU, some multi-GPU DXG paths that used to succeed now fail, and peer views have a few teardown and bookkeeping problems. Single-GPU behavior is unchanged. ## Technical Details - `map_to_nodes` creates peer views only for system memory and user pointers. VRAM and imported resources stay mapped on the GPU that owns them, as before #13442. - `hsaKmtMemoryVaUnmap` drops a peer view when the whole range is unmapped. A later `hsaKmtMemoryVaMap` builds a new view instead of returning early with "bo is mapped already", and freeing the memory no longer unmaps it twice. - `DropPeer` waits on the peer device's paging fence between the unmap and `Evict()`, the same order `hsaKmtMemoryVaUnmap` uses. - `MapFromPeer` refuses imported resources, whose pages are not behind the owner's `cpu_addr`. It also clears `is_shared` and `is_locked` on the peer: the owner keeps the shared handle and the Lock2 pin, and the peer only wraps the same pages. ## Issue Tracking <!-- Include a GitHub Issue and/or JIRA ID. Do not post any full JIRA links here. --> <!-- GitHub issue: #1234 --> <!-- JIRA ID: EXAMPLECOMPONENT-1234 --> ## Test Plan <!-- Explain any relevant testing done to verify this PR. --> ## Test Result <!-- Briefly summarize test outcomes. --> ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. --------- Signed-off-by: Flora Cui <flora.cui@amd.com>
Motivation
Map DXG system memory onto multiple GPUs. hsaKmtMemoryVaMap and hsaKmtMemoryVaUnmap submit the page-table update to the device selected by NodeId and wait only on that device's paging fence.
Only system memory can be mapped across GPUs. Peer VRAM mapping is not supported. GpuMemory::MapFromPeerDev returns UnSupported for local VRAM. Mapping local VRAM onto another GPU fails and is rolled back. Single-GPU behavior is unchanged.
Technical Details
Issue Tracking
Test Plan
import torch
torch.cuda.set_device(0)
x = torch.randn(5000, 5000, device='cuda:0')
y = torch.matmul(x, x)
torch.cuda.set_device(1)
x = torch.randn(100, 100, device='cuda:1')
Test Result
test pass. no segfault or hang
Submission Checklist