[Bugfix][Quantization] Fix online MoE weight conversion KeyError: 'weight_scale' with Humming

[waizuichougou]

This failure is deterministic whenever generic online FP8, MXFP8, or MXFP4
MoE quantization selects Humming: its converter reads the layer before the
generated quantized weights and scales have been registered there.

Purpose

Online FP8, MXFP8, and MXFP4 MoE quantization produces quantized weights and
scales as temporary tensors, then passes them to the selected backend's format
converter. Most converters consume those arguments directly. Humming instead
builds its conversion input from layer.state_dict().

At that point, the layer still contains the full-precision weights used for
online loading and does not contain the newly generated weight scales. As a
result, selecting Humming for an online-quantized MoE model fails during model
loading. For example, online MXFP8 loading reaches Humming's ModelOpt schema
without weight_scale and raises:

KeyError: 'weight_scale'

Stage the online-quantized weights and scales on the layer before invoking the
Humming converter. This gives Humming the same canonical quantized tensors that
the other converters receive as arguments. The normal post-conversion
replacement remains unchanged.

The fix covers all generic online MoE formats supported by Humming:

• FP8
• MXFP8
• MXFP4

The MXFP8 path also created two full-size FP8 tensors that were immediately
discarded before quantization. Remove those unused allocations. For the
Qwen3-30B-A3B MoE dimensions (128 experts, 2,048 hidden size, and 768 expert
intermediate size), this avoids 576 MiB of transient device allocations per
layer being processed.

Reproduction

On the parent commit, the following command deterministically fails while
loading the first MoE layer with KeyError: 'weight_scale':

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --quantization online \
  --quantization-config '{"moe":"mxfp8"}' \
  --moe-backend humming \
  --max-model-len 128 \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.95 \
  --enforce-eager

Mxfp8OnlineMoEMethod.process_weights_after_loading
→ convert_to_fp8_moe_kernel_format
→ convert_to_humming_moe_kernel_format
→ ModeloptMxfp8WeightSchema.convert_humming
→ KeyError: 'weight_scale'

--quantization online ensures that the generic online quantization path is
used, while the explicit MoE configuration selects MXFP8 for every routed
expert layer.

Why this is not a duplicate

No open PR currently fixes this conversion boundary. PR #42997 adds Humming's
native online quantization path; it does not bridge the generic online
quantization methods to Humming's layer-based format converter. PR #51808
addresses online quantization configuration overrides and does not change
backend conversion.

Test Plan

python -m pytest tests/quantization/test_online.py -q \
  -k 'stages_quantized_weights_for_humming'

pre-commit run --files \
  tests/quantization/test_online.py \
  vllm/model_executor/layers/quantization/online/fp8.py \
  vllm/model_executor/layers/quantization/online/moe_base.py \
  vllm/model_executor/layers/quantization/online/mxfp4.py \
  vllm/model_executor/layers/quantization/online/mxfp8.py

The regression tests exercise the shared FP8 setup path, MXFP8, and MXFP4 and
assert that Humming's converter observes the exact online-quantized weight and
scale tensors through layer.state_dict().

The model-loading path was also validated on an NVIDIA A100 with
Qwen/Qwen3-30B-A3B-Instruct-2507, using online MXFP8 for MoE and Humming as the
MoE backend.

Test Result

• Focused regression tests: 3 passed, 47 deselected.
• All changed-file pre-commit checks passed, including Ruff, formatting, mypy,
  SPDX, forbidden-import checks, and repository-specific validation.
• Before this change, the real model failed while processing the first MoE
  layer with KeyError: 'weight_scale'.
• After this change, all 16 weight shards and all 48 MoE layers completed
  Humming weight conversion. Weight loading used 30.78 GiB and completed in
  496.08 seconds; the original missing-scale failure did not recur.

Compatibility

Weight staging is limited to the Humming branch of generic online MoE
quantization. Other MoE backends retain their existing conversion behavior;
the only shared change is removal of the unused MXFP8 allocations. No
user-facing configuration or documentation changes are required.

AI-assisted contribution

This contribution was developed with AI assistance.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[waizuichougou]

@mgoin Hi, would you mind taking a look at this bug when you get a chance?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @waizuichougou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @waizuichougou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[khluu]

CI regression evidence for the rebase: merged PR #56997 partially copied this PR's staging into the generic convert_to_fp8_moe_kernel_format() path. That placement fixes online quantization but breaks pre-quantized gated FP8 MoE checkpoints.

Since #56997 merged, all four scheduled Humming eval lanes (H100/B200, default/activation) deterministically fail only nm-testing/Qwen3-30B-A3B-Fp8-v1. The last green was full CI #93069 (commit 6bbad6acdf); the first red was #93187 after #56997, and the failure has repeated through #94125. The representative stack is:

compressed_tensors_moe_w8a8_fp8.process_weights_after_loading
-> convert_to_fp8_moe_kernel_format
-> convert_to_humming_moe_kernel_format
-> humming.schema.compressed_tensors._convert_humming
-> BaseWeightSchema._may_process_global_scale
-> assert scale.size(0) == num_experts

For gated W13, the compressed-tensors path has already collapsed/requantized the two gate/up scales to one scale per expert. The new generic staging loop installs that collapsed tensor before Humming interprets shape_n_stacks=[..., ...], so it sees only half the expected stacked scale rows. Before #56997, pre-quantized loading left the original two-slot scale tensor on the layer for Humming and passed.

PR #56997's own Buildkite #93056 did not select any Humming LM-eval lane, so the regression escaped PR CI.

The clean rebase resolution appears to be this PR's original design: stage tensors only inside generic online MoE setup, and remove #56997's unconditional staging loop from oracle/fp8.py. Please include one pre-quantized gated FP8 regression test plus the exact Humming eval lane when rebasing. I am not opening a duplicate PR while this one owns the correct boundary.

[waizuichougou]

CI regression evidence for the rebase: merged PR #56997 partially copied this PR's staging into the generic convert_to_fp8_moe_kernel_format() path. That placement fixes online quantization but breaks pre-quantized gated FP8 MoE checkpoints.

Since #56997 merged, all four scheduled Humming eval lanes (H100/B200, default/activation) deterministically fail only nm-testing/Qwen3-30B-A3B-Fp8-v1. The last green was full CI #93069 (commit 6bbad6acdf); the first red was #93187 after #56997, and the failure has repeated through #94125. The representative stack is:

compressed_tensors_moe_w8a8_fp8.process_weights_after_loading
-> convert_to_fp8_moe_kernel_format
-> convert_to_humming_moe_kernel_format
-> humming.schema.compressed_tensors._convert_humming
-> BaseWeightSchema._may_process_global_scale
-> assert scale.size(0) == num_experts

For gated W13, the compressed-tensors path has already collapsed/requantized the two gate/up scales to one scale per expert. The new generic staging loop installs that collapsed tensor before Humming interprets shape_n_stacks=[..., ...], so it sees only half the expected stacked scale rows. Before #56997, pre-quantized loading left the original two-slot scale tensor on the layer for Humming and passed.

PR #56997's own Buildkite #93056 did not select any Humming LM-eval lane, so the regression escaped PR CI.

The clean rebase resolution appears to be this PR's original design: stage tensors only inside generic online MoE setup, and remove #56997's unconditional staging loop from oracle/fp8.py. Please include one pre-quantized gated FP8 regression test plus the exact Humming eval lane when rebasing. I am not opening a duplicate PR while this one owns the correct boundary.

Thanks for your feedback! I will resolve this as soon.

[waizuichougou]

@khluu Thanks again for your review. I have updated the PR accordingly by removing the unconditional staging from the shared Humming FP8 converter while keeping it in the online FP8/MXFP8/MXFP4 paths.

[vllm-agent]

A commit/parent A/B now confirms the Humming regression boundary:

• Exact B200 Humming Default build #94436, at merged [Quantization] Prefer Humming before Marlin backends on SM90 #56997 (d1f3d8b8), failed only Qwen3-30B-A3B-Fp8-v1-humming with the known humming/schema/base.py::_may_process_global_scale assertion (1 failed, 13 passed).
• The same exact lane on [Quantization] Prefer Humming before Marlin backends on SM90 #56997's parent (bb87d227) in #94437 passed all 14/14 cases, including that model.

That removes runner/model drift as an explanation and pins the regression to #56997's staging change. This is root-cause evidence, not validation of this PR's current head. Before merge, please still run the exact H100 and B200 Humming default + activation lanes at 6db79395 and require the formerly failing pre-quantized Qwen3 case to pass.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @waizuichougou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork