Repository navigation
Conversation
ROCr reports memory mapped through its virtual memory API (hipMemCreate + hipMemMap, hsa_amd_vmem_map) as HSA_EXT_POINTER_TYPE_HSA_VMEM, with the GPU as its owner. Memory type detection and mem_query only accepted HSA_EXT_POINTER_TYPE_HSA, so VMM device memory was reported as host memory and rocm_copy tried to lock it as host memory, which fails with UCS_ERR_IO_ERROR. This mirrors the CUDA VMM (cuMemMap) support in cuda_copy. Detect the enum value at configure time, since older ROCm headers lack it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
This was referenced Oct 22, 2026
|
🤖 Starting review — findings will be posted here when done. |
| } | ||
|
|
||
| if ((hsa_mem_type == HSA_EXT_POINTER_TYPE_HSA) && | ||
| if (uct_rocm_base_is_hsa_mem_type(hsa_mem_type) && |
There was a problem hiding this comment.
VMM memory is now classified as UCS_MEMORY_TYPE_ROCM, so uct_rocm_base_mem_query() reaches ucs_memtype_cache_update() for it. However, src/ucm/rocm/rocmmem.c only hooks hsa_amd_memory_pool_allocate/free; hipMemUnmap/HSA vmem release emits no UCM_EVENT_MEM_TYPE_FREE. Impact: a cached ROCM entry survives unmap, and if the VA is later reused by another memory type UCX picks the ROCm path for non-ROCm memory. CUDA hooks cuMemMap/cuMemUnmap for exactly this reason. Question, not a blocker: what invalidates the entry when the application calls hipMemUnmap and the VA range is later reused for host memory?
| @@ -0,0 +1,129 @@ | |||
| /** | |||
There was a problem hiding this comment.
The HIP VMM APIs (hipMemAddressReserve, hipMemCreate, hipMemGetAllocationGranularity) are used unconditionally in a file compiled whenever HAVE_HIP is set; configure only requires ROCm >= 3.7 and mem_buffer.cc still carries #if HIP_VERSION >= 50500000 guards. Impact: gtest build breaks on older ROCm. Will this compile against the older ROCm versions we still support? Maybe guard the file with HIP_VERSION like mem_buffer.cc does.
| * map them */ | ||
| class rocm_vmm_mem_buffer { | ||
| public: | ||
| rocm_vmm_mem_buffer(size_t size, size_t num_chunks = 1, |
There was a problem hiding this comment.
handle_type is never passed by any caller (unlike the CUDA helper, which has fabric/posix-fd subclasses) — dead parameter. Maybe drop handle_type until a test needs it? All callers use the default, so prop.requestedHandleTypes is always hipMemHandleTypeNone.
| ucs_status_t uct_rocm_base_query_devices(uct_md_h md, | ||
| uct_tl_device_resource_t **tl_devices_p, | ||
| unsigned *num_tl_devices_p); | ||
| /* Memory allocated from an HSA memory pool (hipMalloc, hipHostMalloc), or |
There was a problem hiding this comment.
The inline helper is inserted into the prototype block with no separating blank line, and the #if-split return with a dangling ; is hard to read. Suggestion:
Suggested change
| /* Memory allocated from an HSA memory pool (hipMalloc, hipHostMalloc), or | |
| /* Memory allocated from an HSA memory pool (hipMalloc, hipHostMalloc), or | |
| * mapped through the HSA virtual memory API (hipMemCreate + hipMemMap) */ | |
| static inline int uct_rocm_base_is_hsa_mem_type(hsa_amd_pointer_type_t type) | |
| { | |
| #if HAVE_DECL_HSA_EXT_POINTER_TYPE_HSA_VMEM | |
| if (type == HSA_EXT_POINTER_TYPE_HSA_VMEM) { | |
| return 1; | |
| } | |
| #endif | |
| return (type == HSA_EXT_POINTER_TYPE_HSA); | |
| } |
minor: avoids the dangling ; and adds the missing blank line before the function.
|
Residual coverage gap worth noting: the |
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 23, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 23, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 26, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 27, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
@edgargabriel @mshanthagit Routing for ROCm CODEOWNER review. This is the first of four PRs (#12137 to #12140) that make HIP VMM memory usable with UCX. It is the dependency for the rest of the series. Context: vllm-project/vllm#56492 (extensible KV cache). Please see the discussion on #12140 about the AMD parallel approach. |
This was referenced Oct 29, 2026
fix(rocr): Report the owning agent in pointer info for imported VMM mappings
ROCm/rocm-systems#13991
Draft
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
to vllm-project/vllm
that referenced
this pull request
Nov 2, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Treat
HSA_EXT_POINTER_TYPE_HSA_VMEMmemory as ROCm memory everywhereHSA_EXT_POINTER_TYPE_HSAis accepted:mem_query;rocm_copyregistration;rocm_copyzcopy.A configure check (
AC_CHECK_DECLS) guards it, since older ROCm headers lack the enum value. Adds a gtest helper that allocates ROCm VMM memory, and an md test.Why?
Device memory mapped with the HIP virtual memory API (
hipMemCreate+hipMemMap,hsa_amd_vmem_*in ROCr) can't be used with UCX today, while the equivalent CUDA VMM memory (cuMemCreate+cuMemMap) can. Frameworks that grow GPU buffers in place reserve an address range and map memory into it on demand. vLLM's extensible KV cache (vllm-project/vllm#56492) does this, and its NIXL KV connector registers that memory with UCX. On ROCm that fails at registration, then crashes inrocm_copy, androcm_ipccan't share the memory at all.ROCr reports VMM memory as
HSA_EXT_POINTER_TYPE_HSA_VMEM, with the GPU as its owner. UCX classified it as host memory, sorocm_copytried to lock it as host memory and registration failed:This mirrors the CUDA VMM support in
cuda_copy.This is one of four PRs that make VMM memory work on ROCm:
rocm_copyhsa_amd_vmem_map/unmaprocm_ipcHow?
uct_rocm_base_is_hsa_mem_type()accepts both pointer types and replaces the direct comparisons.The new test
rocm_cpy/test_rocm_copy_vmm_md.mem_typechecks three addresses: the start of a mapping, an interior address, and a second mapping. At each, it checksdetect_memory_typeandmem_query, thenmem_reg. Without this change it fails: the memory is reported as host memory andmem_regreturns an I/O error. With the change it passes.Tested on MI350X (gfx950) with the vLLM ROCm CI image (ROCm 10.0 TheRock wheels, ROCr 1.21.0). The existing
rocm_cpy,rocm_copy,rocm_ipc/test_mdandtest_rocm_signaltests pass.🤖 Generated with Claude Code