Repository navigation
[ROCm][CI] Pin Inductor pointwise configs in the MoRIIO dummy-weight P/D test - #62280
Merged
AndreasKaratzas merged 2 commits intoOct 19, 2026
Merged
AndreasKaratzas merged 2 commits into
AndreasKaratzas merged 2 commits into
Conversation
…P/D test Inductor's deterministic mode pins only reduction configs and still picks pointwise XBLOCK by timing. On ROCm the XBLOCK changes which multiply-adds of the fused QK-norm + RoPE kernel are contracted into FMAs, and the candidates are within timing noise, so separately compiled servers differ by one or two bf16 logit steps and the attention-only cases fail on the ROCm 10 image. Turn pointwise autotuning off as well. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Member
Author
|
/ci run |
|
✅ Triggered Buildkite CI #95692 for commit ℹ️ This PR is 2 commits behind upstream |
AndreasKaratzas
marked this pull request as ready for review
October 18, 2026 10:32
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
jackLei0901
added a commit
to jackLei0901/vllm_dev
that referenced
this pull request
Oct 19, 2026
Include vllm-project#62274, vllm-project#62280, vllm-project#62286 and vllm-project#62241. The CUTLASS PR diff is unchanged; updated-base runtime validation is left to CI. Assisted-by: OpenAI Codex Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
triton.autotune_pointwise) next to deterministic mode intest_write_dummy_e2e_rocm.py, and update the comment and docstring that explain whySince the ROCm 10 image became the default (#58761), the attention-only cases fail with first-token log-prob distances of one or two bf16 steps (0.03-0.14), e.g. https://buildkite.com/vllm/ci/builds/95663. P/D is not involved: two plain direct serves of the same model on the same GPU disagree by the same amount, while each server is bitwise stable across repeats. Inductor's deterministic mode only pins reduction configs and still picks the XBLOCK of pointwise kernels by timing. On ROCm the XBLOCK changes which multiply-adds of the fused QK-norm + RoPE kernel get contracted into FMAs (e.g. 4 unfused packed adds at XBLOCK 1024, none at 4096), and the candidates are within timing noise of each other, so each server ends up with a random config. Pinning every pointwise kernel's config in two fresh processes makes them agree bitwise; flipping only the QK-norm + RoPE kernel's XBLOCK reproduces the CI distances.
With this change the CI-failing
qwen3_dense-TRITON_ATTN-P_FULL_AND_PIECEWISE-D_FULL_AND_PIECEWISEcase passes 10/10 on MI355 with ROCm 10 (distances exactly 0.0, full external prefix-cache hits).AI assistance (Claude Code) was used to draft this change. Every line was reviewed by the submitter.