Skip to content

fix(rocr): Report the owning agent in pointer info for imported VMM mappings - #13991

Draft
njhill wants to merge 2 commits into
ROCm:developfrom
njhill:users/njhill/rocr-vmem-import-agent-owner
Draft

njhill wants to merge 2 commits into
ROCm:developfrom
njhill:users/njhill/rocr-vmem-import-agent-owner

Conversation

@njhill

@njhill njhill commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Memory created with hsa_amd_vmem_handle_create and mapped with hsa_amd_vmem_map is what hipMemCreate + hipMemMap produce. Frameworks that grow GPU buffers in place map into a reserved range on demand, for example vLLM's extensible KV cache (vllm-project/vllm#56492). Its KV connectors then register that memory with transports such as UCX and RDMA, which start from a pointer. The equivalent works with CUDA VMM memory. This is one of four independent ROCr fixes found while bringing this up on MI350X:

Technical Details

Since #11449, hsa_amd_pointer_info on a mapping of a handle imported from another process (hsa_amd_vmem_import_shareable_handle) no longer crashes. It reports no agentOwner and no memory flags, though. Callers that classify memory by its owner then can't tell imported device memory from host memory. UCX's ROCm transports, for one, treat it as host memory, so cross-process transfers of VMM memory fail.

  • Fix: recover the import's placement from its dma-buf with EnsureImportInfo (from fix(rocr/clr): recover location and size for import vmm handles #11606). This runs under the exclusive memory_lock_ that PtrInfo already holds.
  • Result: report the agent whose memory backs the import, as CUDA reports the device that an imported allocation resides on. An import the driver can't describe, or one backed by memory of an agent outside this process, still reports no owner.
  • FindDriverMemoryHandle: once an imported handle's region is recovered, this no longer returns the handle's driver_handle. That handle carries only a dma-buf fd, so the per-agent imports remain the way to resolve it.

Test Plan

An exporter allocates VMM memory with hipMemCreate (POSIX-fd handle type), fills it, and passes the dma-buf fd over a unix socket (SCM_RIGHTS). An importer then imports it, maps it, and sets access. It queries hsa_amd_pointer_info and runs hsa_amd_memory_async_copy in both directions. Covered: in-process and cross-process, with the importer on the same GPU and on another GPU.

Validated on MI350X (gfx950) with the ROCm 10.0 TheRock wheels from the vLLM CI image, with the patched libhsa-runtime64 bind-mounted over the SDK copies. Because that runtime is 1.21.0, this fix was tested there on top of backports of the ROCr-internal parts of #11449 and #11606. The four related fixes were also built and tested together on develop.

Test Result

check patched
pointer_info on imported mapping owner = exporter's GPU agent, base = mapping start
async_copy imported ↔ local (same and cross GPU, in/cross process) OK, data verified by both processes
UCX rocm_ipc reading another process's VMM memory (vLLM P/D accuracy test) zero-copy rocm_ipc, accuracy unchanged

Without this fix, the importer leaves agentOwner unset (since #11449; before it, the call crashed), and UCX treats the mapping as host memory.

Submission Checklist

🤖 Generated with Claude Code

…appings

hsa_amd_pointer_info on a mapping of a VMM handle imported from another
process (hsa_amd_vmem_import_shareable_handle) reports no agentOwner and no
memory flags, since the handle has no region. Callers that classify memory
by its owner then cannot tell imported device memory from host memory: UCX's
ROCm transports, for one, treat it as host memory, so cross-process transfers
of VMM memory fail.

Recover the import's placement from its dma-buf, as
VMemoryGetAllocPropertiesFromHandle does, and report the agent whose memory
backs it, like CUDA reports the device an imported allocation resides on. A
buffer the driver cannot describe, or memory of an agent outside this
process, still reports no owner.

A recovered region does not make the import own an allocation handle, so
FindDriverMemoryHandle keeps resolving imported handles only through their
per-agent imports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
@therock-pr-bot

Copy link
Copy Markdown

✅ All Policy Checks Passed

Check Status Details
📝 PR Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ⚠️ Warning Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocr-runtime/runtime/hsa-runtime/core/runtime/runtime.cpp; no test file found
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@cfreeamd cfreeamd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-assisted pre-review (for @cfreeamd)

runtime.cpp changes look correct: the FindDriverMemoryHandle guard and the EnsureImportInfo path in VMemoryPtrInfo are consistent with the surrounding imported-handle handling.

Address review feedback: "One the driver cannot describe" read awkwardly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
@ChuanLi1101

ChuanLi1101 commented Oct 29, 2026 •

Copy link
Copy Markdown

@cfreeamd @dayatsin-amd @manocharahul Routing for ROCr technical review. Part of the VMM-on-ROCm fix set (#13989 to #13992), needed by openucx/ucx#12137 to #12140 and vllm-project/vllm#56492. This PR builds on #11449 and #11606: EnsureImportInfo recovers the owning agent for imported VMM mappings. Rahul, you added the import location and size recovery in #11606, so your review of the reuse there would help.

@dayatsin-amd

dayatsin-amd commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

We have some other discussions in progress.

Overall, I this commit in this upcoming PR would fix this:
#14279
1a1d0b3

Note that there are use cases where the agent owner cannot be returned by hsa_amd_pointer_info and your code should be able to handle this. The 2 cases are:

  1. If the memory was imported using a Fabric Handle. This is because this memory may belong to a GPU on a different node.
  2. If the memory was imported using dmabuf, but that dmabuf comes from an third-party drivers (e.g external NIC for RDMA).

erman-gurses added a commit that referenced this pull request Nov 6, 2026
…dle (#13992)

## Motivation

Memory created with `hsa_amd_vmem_handle_create` and mapped with
`hsa_amd_vmem_map` is what `hipMemCreate` + `hipMemMap` produce.
Frameworks that grow GPU buffers in place map into a reserved range on
demand, for example vLLM's extensible KV cache
(vllm-project/vllm#56492). Its KV connectors then register that memory
with transports such as UCX and RDMA, which start from a pointer. The
equivalent works with CUDA VMM memory. This is one of four independent
ROCr fixes found while bringing this up on MI350X:

- #13989 – `hsa_amd_portable_export_dmabuf` for VMM memory
- #13990 – mapping base in `hsa_amd_pointer_info`
- #13991 – owning agent in `hsa_amd_pointer_info` for imported mappings
- #13992 – interior addresses in `hsa_amd_vmem_retain_alloc_handle`
(this PR)

## Technical Details

`hsa_amd_vmem_retain_alloc_handle` only found a handle when given the
exact start of its mapping. Callers often hold an address within an
allocation, such as a buffer at an offset into a VMM-backed pool that a
transport wants to share through its handle. They got
`HSA_STATUS_ERROR_INVALID_ALLOCATION`. `cuMemRetainAllocationHandle`
accepts any address within the mapping.

Resolve the address to the mapping that contains it. The header
documentation is updated to match.

## Test Plan

Call `hsa_amd_vmem_retain_alloc_handle` at the start of a VMM mapping
and at +2 MiB, and compare the returned handles.

Validated on MI350X (gfx950) with the ROCm 10.0 TheRock wheels from the
vLLM CI image, with the patched `libhsa-runtime64` bind-mounted over the
SDK copies. Because that runtime is 1.21.0, this fix was tested there on
top of backports of the ROCr-internal parts of #11449 and #11606. The
four related fixes were also built and tested together on develop.

## Test Result

| check | stock | patched |
|---|---|---|
| retain at mapping start | OK | OK |
| retain at +2 MiB | `HSA_STATUS_ERROR_INVALID_ALLOCATION` | OK, same
handle |

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: erman-gurses <99776114+erman-gurses@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants