Repository navigation
[ROCm] Re-enable cudagraph memory profiling, captured on the current stream #48526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
AndreasKaratzas
merged 3 commits into
vllm-project:main
from
peizhang56:fix/rocm-cudagraph-profiling-current-stream
Jul 15, 2026
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -492,11 +492,14 @@ def determine_available_memory(self) -> int: | |
| ) | ||
|
|
||
| # Profile CUDA graph memory if graphs will be captured. | ||
| # Skip on ROCm/HIP/XPU as graph pool handles and get_memory_info | ||
| # behave differently and can produce incorrect/negative estimates. | ||
| # ROCm is included: #44825 moved the profiler to | ||
| # torch.accelerator.get_memory_info (reliable on ROCm, as used by | ||
| # the AMD-CI mem tests), and graph_pool_handle resolves to the same | ||
| # torch.cuda handle the live capture path already uses on ROCm. | ||
| # XPU stays excluded (see #39977). | ||
|
Comment on lines
+495
to
+499
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this can be deleted, we don't need now-irrelevant references to historical state. |
||
| cudagraph_memory_estimate = 0 | ||
| if ( | ||
| current_platform.is_cuda() | ||
| current_platform.is_cuda_alike() | ||
| and self.vllm_config.compilation_config.cudagraph_mode | ||
| != CUDAGraphMode.NONE | ||
| ): | ||
|
|
@@ -512,8 +515,7 @@ def determine_available_memory(self) -> int: | |
| + profile_result.weights_memory | ||
| ) | ||
|
|
||
| # On ROCm, cudagraph_memory_estimate is always 0 so this is a no-op. | ||
| # On CUDA, respect the opt-in flag as originally designed. | ||
| # Respect the opt-in flag as originally designed. | ||
| cudagraph_memory_estimate_applied = ( | ||
| cudagraph_memory_estimate | ||
| if envs.VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR LGTM for now, but should we check if there's a way to tell the model runner to delete the AITER persistent kernel buffers after CG capture? I wonder why this issue isn't seen on CUDA cc @dllehr-amd @AndreasKaratzas @tjtanaa
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @Rohan138. I looked into the "free the AITER buffers after capture" idea — it doesn't recover the throughput, and I think the reason is worth spelling out because it's not a leak and not a graph-memory-release problem.
What the buffer is. The stranded block is a ~76 MiB (79,691,776 B) persistent scratch/workspace that AITER's sparse-MLA decode kernel allocates lazily on its first forward pass (dispatched from mla_decode_fwd at vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py:702). It's allocated inside AITER's compiled kernel code — a gc walk finds no owning Python torch.Tensor — so nothing on the vLLM side holds a handle to it. It's a legitimate runtime workspace: the decode kernels reuse it on every step, so it's meant to persist.
Why the profiling stage strands it. profile_cudagraph_memory() runs its throwaway forward+capture inside graph_capture(), which by default allocates a fresh side stream (parallel_state.py:600, torch.cuda.Stream(...)). torch's caching allocator pools free blocks per stream. So when that first ever sparse-MLA forward happens on the side stream, the 76 MiB workspace is created in the side stream's allocator pool rather than the default/runtime pool. Confirmed with memory_snapshot: the fast path has all segments on one (default) stream; the slow path has an extra segment (~0.07 GiB) owned by a second stream.
Why it degrades decode later. That side-stream block is allocated before the real KV cache. Its presence in a separate pool shifts the physical HBM placement of the KV/activation buffers allocated afterward. Every bandwidth-bound decode kernel then reads from a worse physical region — traces show ~+20–30% on bandwidth-bound kernels (allreduce, wvSplitK, concat_and_cache_mla) while compute-bound MFMA barely moves. Same kernels, same launch configs, same total reserved memory — purely a placement effect. Pre-#47366, ROCm never ran profiling, so this workspace was first allocated on the default stream during real warmup → good placement → full speed.
Why freeing after capture doesn't work (I benchmarked it, MI355 tp4, GLM-5.2-MXFP4):
Two reasons it can't help: (1) empty_cache only returns unreferenced blocks — this one is still live (AITER holds it and decode needs it), so it's not reclaimed; and (2) even a working free is too late — the placement damage is done when the KV cache is allocated after profiling, and the side-stream pool already exists at that point. prewarm_default and reset_ws (in my root-causing) failed for the same ordering reason. Capturing on the current stream is the only approach that prevents the separate pool from ever existing, so KV placement matches the no-profiling path.
Why CUDA doesn't hit this. Two independent reasons: (1) the triggering workspace is allocated by AITER, which is ROCm-only — CUDA uses FlashAttention/FlashInfer backends that don't create this particular persistent buffer; and (2) the fix itself is ROCm-specific anyway — on CUDA current_stream() at init is the legacy default stream, on which graph capture can't begin, so CUDA is forced onto a side stream regardless. That's why the change is scoped to is_rocm().
Net: it's not a graph-memory-release issue (a profiling run that captures no graph at all still strands the buffer and is still slow), and we don't change any AITER pool — we just ensure that first forward runs on the runtime stream so the workspace lands in the right pool. Happy to file a follow-up if AITER wants to expose a way to pre-touch/own that workspace explicitly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please let me know if it resolves your question. It is really an aiter related issue, cuda does not have it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well ... my question is that this workspace or its handles should likely be allocated by
aiter.get_mla_metadata_v1from vllm: https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py#L550So while the allocation itself does live inside AITER, why doesn't torch "see" this workspace and gcit after cudagraph capture in https://github.com/vllm-project/vllm/blob/85f50eb/vllm/v1/worker/gpu_model_runner.py#L5315?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question — the key is that there are two separate buffer sets here, and the stranded block isn't the one get_mla_metadata_v1 touches.
The get_mla_metadata_v1 buffers are torch-visible — and they're not the problem. _mla_work_meta_data, _mla_work_info_set, _mla_work_indptr, mla_reduce* (allocated at rocm_aiter_mla_sparse.py:442-457) are plain torch.empty(...) tensors owned by the metadata builder. get_mla_metadata_v1 doesn't allocate them — it's passed them as out-params and just fills them in (:550-568). torch sees them perfectly. But they're persistent attributes on a live object held for the runner's lifetime, so gc/empty_cache correctly won't reclaim them — and they're small. These aren't what strands the 76 MiB.
The stranded 76 MiB comes from mla_decode_fwd, not get_mla_metadata_v1. The block is a scratch/workspace the AITER sparse-decode kernel allocates lazily inside compiled kernel code on its first forward (dispatched from mla_decode_fwd at :704). It's not wrapped by any Python torch.Tensor — I confirmed with a gc walk that no owning object exists. torch's caching allocator tracks the underlying segment (that's why it shows up in memory_snapshot), but there's no Python handle to it, so it's invisible to any Python-side cleanup.
So on your gpu_model_runner.py#L5315 question specifically — that's why the gc.collect() + empty_cache() there is a no-op on this block, for two independent reasons:
I did benchmark the explicit-free approach (in my earlier reply's table): side-stream capture + gc.collect() + empty_cache() after capture = 69.3 tok/s vs. 83.3 for capturing on the current stream. It doesn't recover throughput, for exactly these reasons.
Net: nothing to gc here — the visible buffers should persist, and the invisible one is native-owned + live. Capturing that first forward on the runtime stream is what keeps the workspace in the right pool in the first place. Happy to file an AITER follow-up if it'd help to expose a way to pre-own/pre-touch that decode workspace explicitly.