Repository navigation
[CI][ROCm] Check skinny-GEMM guard dispatch without torch.profiler - #64655
Merged
AndreasKaratzas merged 2 commits intoOct 30, 2026
Merged
AndreasKaratzas merged 2 commits into
AndreasKaratzas merged 2 commits into
Conversation
The daily Full CI run arms the ci-infra kernel recorder, which registers its own ROCProfiler SDK tool via ROCP_TOOL_LIBRARIES. kineto then never registers, torch.profiler records nothing, and the profiler-based dispatch checks fail with kernels=[]. Spy on ops.wvSplitK instead. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Member
Author
|
/ci run |
AndreasKaratzas
marked this pull request as ready for review
October 30, 2026 04:42
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
✅ Triggered Buildkite CI #98564 for commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rocm_unquantized_gemm_impltook the skinny path by wrappingops.wvSplitK, instead of reading kernel names fromtorch.profilerThe four dispatch tests added in #63179 fail with
kernels=[]on the daily Full CI run on MI300 (build 98403, both attempts) but pass on per-commit builds. Only the daily run arms the kernel recorder from vllm-project/ci-infra#685, which registers its own ROCProfiler SDK tool throughROCP_TOOL_LIBRARIES. With that set, the SDK configures only the listed tool. Kineto'srocprofiler_configureinlibtorch_cpu.sonever registers, andtorch.profilerrecords nothing, not even a plaina @ a. The guard's contract is which branch runs, so the test now checks that directly, regardless of what else is profiling the process.Validation on MI355X, with the ci-infra recorder built from source and armed the way the daily run arms it:
pytest tests/model_executor/layers/test_lmhead_skinny_gemm_guard.pypasses 9/9 without the recorder and fails the same 4 tests as CI with it.VLLM_ROCM_USE_AITER=1.test_n5_big_weight_dispatch[None-False]; removing the n=1 carve-out failstest_n1_big_weight_dispatch_unchanged.pytest -v -s model_executor -m '(not slow_test) and (not distributed)'under the recorder: 1403 passed, 90 skipped, 0 failed (same count as per-commit CI).No open PR touches this test (searched
test_lmhead_skinny_gemm_guard,skinny gemm guard,kernrec). MI300 was not available locally; the CI group runs there.AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.