Repository navigation
[Model Loader] Keep norm layers neutral under dummy weight init - #61323
Merged
Merged
Conversation
Random +-1e-3 norm weights shrink every activation, which makes data-dependent kernels (e.g. sparse indexer top-k) time far from real weights. Norm layers now declare their identity value as dummy_weight_value. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
WoosukKwon
marked this pull request as ready for review
October 13, 2026 03:51
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #94370 for commit |
Member
|
Main CI found a related regression after #61134 merged, so this PR needs exact E2E Core validation before merge. In main #94362, both NVIDIA H200 MIG18 E2E Core shard 1 and AMD MI250 E2E Core fail:
Both platforms report This norm-neutral change may or may not repair the failure—the causal boundary still needs a hardware gate. Please run and require both exact H200 |
Collaborator
CI selector (shadow): 141 test steps (187 jobs) instead of 64 (80 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (141)
Would skip (today's rules run them) (6)
Would add (today's rules do not run them) (83)
AMD mirrors: would skip (6)
AMD mirrors: would add (65)
2 changed files · base |
Member
|
Update: the automatic MI250 replacement in main #94362 also failed both MRV1 variants with the identical key-576 mismatch. We now have three independent exact failures (H200 once, MI250 twice), so this is deterministic rather than retry noise. The selector shadow on this PR confirms |
6 tasks done
6 tasks done
AndreasKaratzas
pushed a commit
to AndreasKaratzas/vllm
that referenced
this pull request
Oct 16, 2026
…-project#61323) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
michael-gpu
pushed a commit
to practical-inference/vllm-ampere-extended
that referenced
this pull request
Oct 20, 2026
…-project#61323) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Keep normalization layers neutral under
--load-format dummy:RMSNorm,LayerNorm,RMSNormGatedandGemmaRMSNormtag their parameters withdummy_weight_value(their identity value), so dummy init no longer fills them with ±1e-3. Follow-up to #61134 (dtype-aware dummy init).Claims
1 + wweight 0) instead of random ±1e-3.topKPerRowPrefilltakes 27.8 ms per forward with dummy weights onmain, 10.7 ms with this PR, and 10.6 ms with real weights.Validation
DeepSeek-V4.1-Flash (40 layers), TP4 + EP on 4x GB200, MegaMoE, FlashInfer sparse MLA, MXFP4 indexer cache. One prefill step: 8 requests x 1024 new tokens at ~100K context each. GPU kernel time per forward from a profiled CUDA-graph replay of the model forward (3 replays):
topKPerRowPrefill(ms/forward)main(norms ±1e-3)The remaining forward gap to real weights is MoE routing (real routing from random token IDs vs. a balanced fixture), not this change. All other kernels differ by under 1 ms between this PR and real weights.
These numbers come from an exact-shape forward benchmark that isn't upstream yet; the per-kernel effect can also be seen in an
nsysprofile of any long-context DeepSeek-V4.1 prefill with--load-format dummy.Unit test (in
.buildkite/test_areas/model_executor.yaml):test_dummy_norm_weights_are_neutralfails onmain(RMSNormweight stays random) and passes with this PR.No model evaluation: only
--load-format dummyvalues change.Details
topKPerRowPrefillis a data-dependent radix select, so these degenerate logits are its slow case; real models never produce them.dummy_weight_valuehook ininitialize_single_dummy_weight, as DeepSeek-V4.1's engram already does for its own weights. Each norm declares the value its constructor already uses.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
Not a duplicate: open PRs were searched for dummy norm initialization (
dummy RMSNorm weight,dummy_weight_value norm,dummy init layernorm,load-format dummy norm weights); none change it.AI assistance (Claude Code) was used to write this change and to run the tests and validation above.
🤖 Generated with Claude Code