Repository navigation
Conversation
…appings hsa_amd_pointer_info on a mapping of a VMM handle imported from another process (hsa_amd_vmem_import_shareable_handle) reports no agentOwner and no memory flags, since the handle has no region. Callers that classify memory by its owner then cannot tell imported device memory from host memory: UCX's ROCm transports, for one, treat it as host memory, so cross-process transfers of VMM memory fail. Recover the import's placement from its dma-buf, as VMemoryGetAllocPropertiesFromHandle does, and report the agent whose memory backs it, like CUDA reports the device an imported allocation resides on. A buffer the driver cannot describe, or memory of an agent outside this process, still reports no owner. A recovered region does not make the import own an allocation handle, so FindDriverMemoryHandle keeps resolving imported handles only through their per-agent imports. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
Address review feedback: "One the driver cannot describe" read awkwardly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
@cfreeamd @dayatsin-amd @manocharahul Routing for ROCr technical review. Part of the VMM-on-ROCm fix set (#13989 to #13992), needed by openucx/ucx#12137 to #12140 and vllm-project/vllm#56492. This PR builds on #11449 and #11606: EnsureImportInfo recovers the owning agent for imported VMM mappings. Rahul, you added the import location and size recovery in #11606, so your review of the reuse there would help. |
|
We have some other discussions in progress. Overall, I this commit in this upcoming PR would fix this: Note that there are use cases where the agent owner cannot be returned by hsa_amd_pointer_info and your code should be able to handle this. The 2 cases are:
|
…dle (#13992) ## Motivation Memory created with `hsa_amd_vmem_handle_create` and mapped with `hsa_amd_vmem_map` is what `hipMemCreate` + `hipMemMap` produce. Frameworks that grow GPU buffers in place map into a reserved range on demand, for example vLLM's extensible KV cache (vllm-project/vllm#56492). Its KV connectors then register that memory with transports such as UCX and RDMA, which start from a pointer. The equivalent works with CUDA VMM memory. This is one of four independent ROCr fixes found while bringing this up on MI350X: - #13989 – `hsa_amd_portable_export_dmabuf` for VMM memory - #13990 – mapping base in `hsa_amd_pointer_info` - #13991 – owning agent in `hsa_amd_pointer_info` for imported mappings - #13992 – interior addresses in `hsa_amd_vmem_retain_alloc_handle` (this PR) ## Technical Details `hsa_amd_vmem_retain_alloc_handle` only found a handle when given the exact start of its mapping. Callers often hold an address within an allocation, such as a buffer at an offset into a VMM-backed pool that a transport wants to share through its handle. They got `HSA_STATUS_ERROR_INVALID_ALLOCATION`. `cuMemRetainAllocationHandle` accepts any address within the mapping. Resolve the address to the mapping that contains it. The header documentation is updated to match. ## Test Plan Call `hsa_amd_vmem_retain_alloc_handle` at the start of a VMM mapping and at +2 MiB, and compare the returned handles. Validated on MI350X (gfx950) with the ROCm 10.0 TheRock wheels from the vLLM CI image, with the patched `libhsa-runtime64` bind-mounted over the SDK copies. Because that runtime is 1.21.0, this fix was tested there on top of backports of the ROCr-internal parts of #11449 and #11606. The four related fixes were also built and tested together on develop. ## Test Result | check | stock | patched | |---|---|---| | retain at mapping start | OK | OK | | retain at +2 MiB | `HSA_STATUS_ERROR_INVALID_ALLOCATION` | OK, same handle | ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: erman-gurses <99776114+erman-gurses@users.noreply.github.com>
Motivation
Memory created with
hsa_amd_vmem_handle_createand mapped withhsa_amd_vmem_mapis whathipMemCreate+hipMemMapproduce. Frameworks that grow GPU buffers in place map into a reserved range on demand, for example vLLM's extensible KV cache (vllm-project/vllm#56492). Its KV connectors then register that memory with transports such as UCX and RDMA, which start from a pointer. The equivalent works with CUDA VMM memory. This is one of four independent ROCr fixes found while bringing this up on MI350X:hsa_amd_portable_export_dmabuffor VMM memoryhsa_amd_pointer_infohsa_amd_pointer_infofor imported mappings (this PR)hsa_amd_vmem_retain_alloc_handleTechnical Details
Since #11449,
hsa_amd_pointer_infoon a mapping of a handle imported from another process (hsa_amd_vmem_import_shareable_handle) no longer crashes. It reports noagentOwnerand no memory flags, though. Callers that classify memory by its owner then can't tell imported device memory from host memory. UCX's ROCm transports, for one, treat it as host memory, so cross-process transfers of VMM memory fail.EnsureImportInfo(from fix(rocr/clr): recover location and size for import vmm handles #11606). This runs under the exclusivememory_lock_thatPtrInfoalready holds.FindDriverMemoryHandle: once an imported handle's region is recovered, this no longer returns the handle'sdriver_handle. That handle carries only a dma-buf fd, so the per-agent imports remain the way to resolve it.Test Plan
An exporter allocates VMM memory with
hipMemCreate(POSIX-fd handle type), fills it, and passes the dma-buf fd over a unix socket (SCM_RIGHTS). An importer then imports it, maps it, and sets access. It querieshsa_amd_pointer_infoand runshsa_amd_memory_async_copyin both directions. Covered: in-process and cross-process, with the importer on the same GPU and on another GPU.Validated on MI350X (gfx950) with the ROCm 10.0 TheRock wheels from the vLLM CI image, with the patched
libhsa-runtime64bind-mounted over the SDK copies. Because that runtime is 1.21.0, this fix was tested there on top of backports of the ROCr-internal parts of #11449 and #11606. The four related fixes were also built and tested together on develop.Test Result
pointer_infoon imported mappingasync_copyimported ↔ local (same and cross GPU, in/cross process)rocm_ipcreading another process's VMM memory (vLLM P/D accuracy test)rocm_ipc, accuracy unchangedWithout this fix, the importer leaves
agentOwnerunset (since #11449; before it, the call crashed), and UCX treats the mapping as host memory.Submission Checklist
🤖 Generated with Claude Code