Repository navigation
[Hardware][Gaudi][Feature] Support Contiguous Cache Fetch - #12139
Merged
Merged
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
zhouyu5
changed the title
zhouyu5
force-pushed
the
hpu_cont_pa
branch
2 times, most recently
from
January 17, 2025 09:10
2bbc892 to
d622b98
Compare
Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Signed-off-by: zhouyu5 <yu.zhou@intel.com>
zhouyu5
force-pushed
the
hpu_cont_pa
branch
2 times, most recently
from
January 23, 2025 08:50
0952f55 to
81c362d
Compare
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
zhouyu5
marked this pull request as ready for review
January 23, 2025 08:59
zhouyu5
requested review from
alexm-redhat,
comaniac,
njhill,
youkaichao and
zhuohan123
as code owners
January 23, 2025 08:59
Author
|
/ready |
jikunshang
reviewed
Jan 24, 2025
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
Author
|
All test passed, could you help merge it? @comaniac |
xjpang
pushed a commit
to xjpang/vllm
that referenced
this pull request
Feb 20, 2025
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Akshat-Tripathi
pushed a commit
to krai/vllm
that referenced
this pull request
Mar 3, 2025
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
lulmer
pushed a commit
to lulmer/vllm
that referenced
this pull request
Apr 7, 2025
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>
shreyankg
pushed a commit
to shreyankg/vllm
that referenced
this pull request
May 3, 2025
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
philippesic
pushed a commit
to philippesic/vllm-semantic-cache
that referenced
this pull request
Jul 19, 2026
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
plasticchris
pushed a commit
to plasticchris/vllm
that referenced
this pull request
Jul 20, 2026
…ct#12139) Signed-off-by: yuzhou <yuzhou@habana.ai> Signed-off-by: zhouyu5 <yu.zhou@intel.com> Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 23, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 23, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 26, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 27, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 28, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 29, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Oct 30, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
njhill
added a commit
that referenced
this pull request
Nov 2, 2026
…cannot share it - Triton attention initialized the per-token-head quantized KV cache scales with `fill_(1.0)` over all `num_blocks`, but the extensible KV cache backs only a prefix of them during warmup, so the fill touched unmapped memory (illegal memory access). The attention kernels read scales only below each sequence's length, where they are written with their tokens, so they need no initialization. - The extensible KV cache's views do not keep it alive, and the model runner can outlive shutdown in an in-process engine, so its memory was still mapped afterwards (tests asserting that GPU memory is freed failed). Unmap it explicitly in `shutdown()`. - Skip freeing an extensible tensor's mappings during interpreter finalization: the process releases them on exit, and the HIP runtime may already be torn down, where a driver call aborts the process. - LMCacheMPConnector shares the KV cache through legacy CUDA IPC, which cannot export driver-mapped memory on any platform. Keep the extensible KV cache off with it by default, and reject an explicit request. - On ROCm, UCX does not yet recognize memory mapped with `hipMemMap` (`HSA_EXT_POINTER_TYPE_HSA_VMEM`) and ROCr cannot export it as a dma-buf for RDMA, so the NIXL, Mooncake and MoRIIO connectors fail to register the KV cache. Keep the extensible KV cache off on ROCm when one of them registers device KV cache memory (directly or within MultiConnector), until fixed releases are available (openucx/ucx#12137, #12138, #12139, #12140; ROCm/rocm-systems#13989). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Nick Hill <nickhill123@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contiguous cache fetching to avoid using costly gather operation on Gaudi3. Requires changes in vllm-hpu-extension (HabanaAI/vllm-hpu-extension#17) to work.
Introduces redundant calculations in decoding phase. Feature improves the performance of all tested workloads over the entire benchmark (5-12%) on Gaudi3. commit further improves the performance of this feature (9-22%). Feature negatively impacts the performance of Gaudi2.
Use VLLM_CONTIGUOUS_PA=true environment variable to enable.