Repository navigation
Conversation
Per-tensor FP8 MoE requantizes w13 to one scale per expert before the kernel conversion, but Humming's per-tensor schema splits the w13 scale across the gate and up shards and asserts on the [E] tensor. Repeat the scale per shard for gated w13. Signed-off-by: lesj0610 <lesj0610@gmail.com>
lesj0610
requested review from
mgoin,
pavanimajety,
wzhao18 and
zyongye
as code owners
October 13, 2026 12:18
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
|
/ci run |
|
✅ @lesj0610, CI is now available for this PR.
|
|
✅ Triggered Buildkite CI #94516 for commit |
Contributor
|
@khluu Can you enable the humming-related lm eval tests which fail in main in the CI? so we verify this PR indeed fixes those issues. |
Contributor
|
/ci retry |
|
✅ Queued 1 failed job(s) for retry in Buildkite CI #94516. |
4 of 6 tasks
5 of 6 tasks
Member
|
Should be fixed by humming bump in #61556 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Fixes the Humming load crash for per-tensor FP8 MoE checkpoints, which currently fails the
Qwen3-30B-A3B-Fp8-v1-hummingandQwen3-30B-A3B-Fp8-v1-humming-act-fp8LM Eval configs on main (seen in the B200 and H100 Humming jobs on #60948). Regressed in #56997.Claims
--moe-backend hummingloads again instead of failing engine init withassert scale.size(0) == (num_experts or 1).fp8, ModelOpt, online FP8). Block, channel and MXFP8 scales and non-gated MoE are unchanged.Validation
nm-testing/Qwen3-30B-A3B-Fp8-v1with--moe-backend humming --linear-backend humming --enforce-eageron SM80:AssertionErrorinhumming/schema/base.pyQwen3-30B-A3B-Fp8-v1-hummingGSM8K (1319 q, 5-shot, threshold 0.86)The
-act-fp8config needs FP8 activations, which Humming rejects on SM80 by design. Locally it now gets past the weight conversion (w13 converts to a channel scale) and stops at that input-schema check, so its accuracy is left to the H100/B200 Humming jobs.Details
Since #56997, the Humming branch of
convert_to_fp8_moe_kernel_formatwrites the processed tensors back to the layer before the Humming conversion. For per-tensor FP8,process_fp8_weight_tensor_strategy_moehas already requantized w13 to the max scale of each expert, so the layer now holds a[E]w13 scale. Humming's per-tensor schema expects one scale per gate/up shard: with two n-stacks it views the scale as(-1, 2, 1), which turns[E]into[E/2, 2, 1]and trips the assertion.Dropping the write-back would bring back the old pairing of requantized weights with the per-shard checkpoint scales. Instead this repeats the requantized scale for both shards, which matches the requantized weight exactly.
numel == num_expertsonly holds for per-tensor scales, and non-gated w13 keeps a single n-stack. The existing Humming LM Eval configs already cover this path, so no new test.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
AI assistance: Claude