Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions vllm/distributed/parallel_state.py
Original file line number Diff line number Diff line change
Expand Up @@ -1427,7 +1427,10 @@ def get_pcp_group() -> GroupCoordinator:


@contextmanager
def graph_capture(device: torch.device):
def graph_capture(
device: torch.device,
graph_capture_context: GraphCaptureContext | None = None,
):
"""
`graph_capture` is a context manager which should surround the code that
is capturing the CUDA graph. Its main purpose is to ensure that some
Expand All @@ -1440,8 +1443,13 @@ def graph_capture(device: torch.device):
the graph capture is running on a separate stream from the default stream,
in order to explicitly distinguish the kernels to capture
from other kernels possibly launched on background in the default stream.

A caller may pass an explicit ``graph_capture_context`` to control the
stream used (e.g. to capture on the default stream).
"""
context = GraphCaptureContext(torch.cuda.Stream(device=device))
context = graph_capture_context or GraphCaptureContext(
torch.cuda.Stream(device=device)
)
with get_tp_group().graph_capture(context), get_pp_group().graph_capture(context):
yield context

Expand Down
21 changes: 20 additions & 1 deletion vllm/v1/worker/gpu_model_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
from vllm.distributed.kv_transfer import get_kv_transfer_group, has_kv_transfer_group
from vllm.distributed.kv_transfer.kv_connector.utils import copy_kv_blocks
from vllm.distributed.parallel_state import (
GraphCaptureContext,
get_dcp_group,
get_pp_group,
get_tp_group,
Expand Down Expand Up @@ -6607,11 +6608,29 @@ def profile_cudagraph_memory(self) -> int:
per_graph_estimate = {}
encoder_memory_estimate = 0

# On ROCm, capture these throwaway profiling graphs on the current stream

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR LGTM for now, but should we check if there's a way to tell the model runner to delete the AITER persistent kernel buffers after CG capture? I wonder why this issue isn't seen on CUDA cc @dllehr-amd @AndreasKaratzas @tjtanaa

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Rohan138. I looked into the "free the AITER buffers after capture" idea — it doesn't recover the throughput, and I think the reason is worth spelling out because it's not a leak and not a graph-memory-release problem.

What the buffer is. The stranded block is a ~76 MiB (79,691,776 B) persistent scratch/workspace that AITER's sparse-MLA decode kernel allocates lazily on its first forward pass (dispatched from mla_decode_fwd at vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py:702). It's allocated inside AITER's compiled kernel code — a gc walk finds no owning Python torch.Tensor — so nothing on the vLLM side holds a handle to it. It's a legitimate runtime workspace: the decode kernels reuse it on every step, so it's meant to persist.

Why the profiling stage strands it. profile_cudagraph_memory() runs its throwaway forward+capture inside graph_capture(), which by default allocates a fresh side stream (parallel_state.py:600, torch.cuda.Stream(...)). torch's caching allocator pools free blocks per stream. So when that first ever sparse-MLA forward happens on the side stream, the 76 MiB workspace is created in the side stream's allocator pool rather than the default/runtime pool. Confirmed with memory_snapshot: the fast path has all segments on one (default) stream; the slow path has an extra segment (~0.07 GiB) owned by a second stream.

Why it degrades decode later. That side-stream block is allocated before the real KV cache. Its presence in a separate pool shifts the physical HBM placement of the KV/activation buffers allocated afterward. Every bandwidth-bound decode kernel then reads from a worse physical region — traces show ~+20–30% on bandwidth-bound kernels (allreduce, wvSplitK, concat_and_cache_mla) while compute-bound MFMA barely moves. Same kernels, same launch configs, same total reserved memory — purely a placement effect. Pre-#47366, ROCm never ran profiling, so this workspace was first allocated on the default stream during real warmup → good placement → full speed.

Why freeing after capture doesn't work (I benchmarked it, MI355 tp4, GLM-5.2-MXFP4):

approach tok/s
side-stream capture (broken) 69.5
side-stream capture + explicit gc.collect() + empty_cache() after capture 69.3
current-stream capture (this PR) 83.3

Two reasons it can't help: (1) empty_cache only returns unreferenced blocks — this one is still live (AITER holds it and decode needs it), so it's not reclaimed; and (2) even a working free is too late — the placement damage is done when the KV cache is allocated after profiling, and the side-stream pool already exists at that point. prewarm_default and reset_ws (in my root-causing) failed for the same ordering reason. Capturing on the current stream is the only approach that prevents the separate pool from ever existing, so KV placement matches the no-profiling path.

Why CUDA doesn't hit this. Two independent reasons: (1) the triggering workspace is allocated by AITER, which is ROCm-only — CUDA uses FlashAttention/FlashInfer backends that don't create this particular persistent buffer; and (2) the fix itself is ROCm-specific anyway — on CUDA current_stream() at init is the legacy default stream, on which graph capture can't begin, so CUDA is forced onto a side stream regardless. That's why the change is scoped to is_rocm().

Net: it's not a graph-memory-release issue (a profiling run that captures no graph at all still strands the buffer and is still slow), and we don't change any AITER pool — we just ensure that first forward runs on the runtime stream so the workspace lands in the right pool. Happy to file a follow-up if AITER wants to expose a way to pre-touch/own that workspace explicitly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let me know if it resolves your question. It is really an aiter related issue, cuda does not have it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well ... my question is that this workspace or its handles should likely be allocated by aiter.get_mla_metadata_v1 from vllm: https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py#L550

So while the allocation itself does live inside AITER, why doesn't torch "see" this workspace and gcit after cudagraph capture in https://github.com/vllm-project/vllm/blob/85f50eb/vllm/v1/worker/gpu_model_runner.py#L5315?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question — the key is that there are two separate buffer sets here, and the stranded block isn't the one get_mla_metadata_v1 touches.

The get_mla_metadata_v1 buffers are torch-visible — and they're not the problem. _mla_work_meta_data, _mla_work_info_set, _mla_work_indptr, mla_reduce* (allocated at rocm_aiter_mla_sparse.py:442-457) are plain torch.empty(...) tensors owned by the metadata builder. get_mla_metadata_v1 doesn't allocate them — it's passed them as out-params and just fills them in (:550-568). torch sees them perfectly. But they're persistent attributes on a live object held for the runner's lifetime, so gc/empty_cache correctly won't reclaim them — and they're small. These aren't what strands the 76 MiB.

The stranded 76 MiB comes from mla_decode_fwd, not get_mla_metadata_v1. The block is a scratch/workspace the AITER sparse-decode kernel allocates lazily inside compiled kernel code on its first forward (dispatched from mla_decode_fwd at :704). It's not wrapped by any Python torch.Tensor — I confirmed with a gc walk that no owning object exists. torch's caching allocator tracks the underlying segment (that's why it shows up in memory_snapshot), but there's no Python handle to it, so it's invisible to any Python-side cleanup.

So on your gpu_model_runner.py#L5315 question specifically — that's why the gc.collect() + empty_cache() there is a no-op on this block, for two independent reasons:

  1. empty_cache only returns free, unreferenced blocks to the OS. This block is live — AITER holds it internally and every decode step reuses it — so it's never on the free list to reclaim.
  2. Even if it were reclaimable, empty_cache doesn't relocate blocks or change which stream-pool owns them. The damage isn't that the buffer exists; it's that (during profiling) its first-ever allocation happened on the side stream, so it landed in the side-stream allocator pool, which shifts the physical placement of the KV cache allocated afterward. That placement is baked in before L5315 ever runs.

I did benchmark the explicit-free approach (in my earlier reply's table): side-stream capture + gc.collect() + empty_cache() after capture = 69.3 tok/s vs. 83.3 for capturing on the current stream. It doesn't recover throughput, for exactly these reasons.

Net: nothing to gc here — the visible buffers should persist, and the invisible one is native-owned + live. Capturing that first forward on the runtime stream is what keeps the workspace in the right pool in the first place. Happy to file an AITER follow-up if it'd help to expose a way to pre-own/pre-touch that decode workspace explicitly.

# instead of the fresh side stream graph_capture() allocates by default.
# torch's allocator pools free blocks per stream, so a side-stream forward
# strands a persistent aiter scratch buffer in a separate pool, shifting
# the physical placement of the real KV cache allocated afterward and
# slowing bandwidth-bound decode ~20%. The graphs are discarded, so a
# side stream is unnecessary here.
# cap_ctx=None keeps the side-stream path on CUDA, where the current
# stream is the legacy default stream, on which capture cannot begin.
cap_ctx = (
GraphCaptureContext(torch.cuda.current_stream(self.device))
if current_platform.is_rocm()
else None
)

# Cleanup-only guard: CUDA graph capture errors should still propagate
# because encoder graph capture is opt-in.
try:
set_cudagraph_capturing_enabled(True)
with self._freeze_gc(), graph_capture(device=self.device):
with (
self._freeze_gc(),
graph_capture(device=self.device, graph_capture_context=cap_ctx),
):
torch.accelerator.synchronize()
torch.accelerator.empty_cache()

Expand Down
12 changes: 7 additions & 5 deletions vllm/v1/worker/gpu_worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -492,11 +492,14 @@ def determine_available_memory(self) -> int:
)

# Profile CUDA graph memory if graphs will be captured.
# Skip on ROCm/HIP/XPU as graph pool handles and get_memory_info
# behave differently and can produce incorrect/negative estimates.
# ROCm is included: #44825 moved the profiler to
# torch.accelerator.get_memory_info (reliable on ROCm, as used by
# the AMD-CI mem tests), and graph_pool_handle resolves to the same
# torch.cuda handle the live capture path already uses on ROCm.
# XPU stays excluded (see #39977).
Comment on lines +495 to +499

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be deleted, we don't need now-irrelevant references to historical state.

cudagraph_memory_estimate = 0
if (
current_platform.is_cuda()
current_platform.is_cuda_alike()
and self.vllm_config.compilation_config.cudagraph_mode
!= CUDAGraphMode.NONE
):
Expand All @@ -512,8 +515,7 @@ def determine_available_memory(self) -> int:
+ profile_result.weights_memory
)

# On ROCm, cudagraph_memory_estimate is always 0 so this is a no-op.
# On CUDA, respect the opt-in flag as originally designed.
# Respect the opt-in flag as originally designed.
cudagraph_memory_estimate_applied = (
cudagraph_memory_estimate
if envs.VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS
Expand Down
Loading