[Bugfix] Size and iterate w13 by shard count for non-gated MoE

[aoshen02]

Purpose

Non-gated MoE (is_act_and_mul=False, e.g. NemotronH's relu2_no_mul) fuses
only the up projection into w13, so w13 holds a single
intermediate_size_per_partition shard rather than two gate/up shards.

13 MoE quantization methods already conditionalize on this — modelopt.py,
compressed_tensors_moe_*, flashinfer, unquantized_fused_moe_method.py, … —
each with its own local w13_num_shards = 2 if self.moe.is_act_and_mul else 1.
14 other methods hardcode 2. For a non-gated model those:

1. over-allocate w13 to 2 * I rows when only I rows are ever written,
   so the second shard stays whatever the allocator handed back. On the online
   path reload/meta.py materializes with torch.empty_strided, so this is
   uninitialized memory that then flows into the block/per-tensor amax, the
   quantized weight, and the GEMM (0 × NaN = NaN);
2. over-allocate the per-shard scale and bias tensors (torch.ones(E, 2),
   torch.zeros(E, 2 * I), (E, 2 * I // block, …));
3. walk w13 as two shards when zeroing the roundup padding
   (Fp8PerBlockOnlineMoEMethod._zero_padding) or when requantizing
   (for shard_id in range(2) in quark_moe.py), so the pad band boundaries
   are computed from a half that is twice too small.

Failure mode 3 is the one that bites even where the allocation happens to be
tolerated: the pad rows are never zeroed, so uninitialized values contaminate
the shared per-block weight scale of real rows.

Rather than adding a 14th copy of the same local variable, this PR adds one
property next to is_act_and_mul and uses it everywhere:

@property
def w13_num_shards(self) -> int:
    """Number of shards fused into w13: gate and up for gated, up only."""
    return 2 if self.is_act_and_mul else 1

Gated models are bit-for-bit unaffected: the property returns 2 and every
changed expression reduces to the one it replaced.

Changes

vllm/model_executor/layers/fused_moe/config.py — add
FusedMoEConfig.w13_num_shards.

68 substitution sites across 13 files:

file	sites	what
quantization/quark/quark_moe.py	24	4 methods' w13 allocations, 5 per-shard (E, 2) scale/zero-point tensors, 3 range(2) requant loops
quantization/mxfp4.py	12	GptOssMxfp4MoEMethod + Mxfp4MoEMethod weights/scales/bias and the _setup_kernel shape asserts
quantization/fp8.py	4	w13_weight, w13_bias, per-tensor torch.ones(E, 2) scale, block scale, plus is_act_and_mul= passthrough to process_fp8_weight_tensor_strategy_moe
quantization/auto_gptq.py	4	qweight / qzeros / scales / g_idx
quantization/online/fp8.py	4	Fp8PerBlockOnlineMoEMethod._zero_padding, weight and bias
quantization/auto_awq.py	3	incl. the reversed intermediate_size_per_partition * 2 spelling
quantization/bitsandbytes.py	3
quantization/moe_wna16.py	3
compressed_tensors_moe_w4a8_fp8.py	3
quantization/humming.py	2
quantization/inc/schemes/inc_mxfp4_moe.py	2
compressed_tensors_moe_w4a4_mxfp4.py	2
quantization/online/moe_base.py	2	shared online base: w13_weight + w13_bias

Deliberately not changed:

• w13_weight_shape / w2_weight_shape torch.empty(num_experts, 2) in the
  compressed-tensors files — that 2 is a (rows, cols) shape descriptor, not
  a shard count.
• The 13 methods that already have a correct local w13_num_shards. Collapsing
  them onto the new property is a pure identity refactor and belongs in its own
  change, not in a bugfix.
• moe_wna16.py's weight loader param.data[expert_id, : shard_size // 2] —
  that 2 is loader shard semantics, only reachable with has_zp and
  non-gated, and needs its own analysis.

Why this does not duplicate an existing PR

Three open PRs touch part of this. This PR is a strict superset of their
substantive content in the weight-allocation / post-load-zeroing layer:

• [Bugfix] Fix FP8 MoE double memory allocation for non-gated models #44498 (littlecircle0730) — Fp8MoEMethod w13_weight, w13_bias,
  block scale. All three covered here, plus the per-tensor
  torch.ones(num_experts, 2) scale and the is_act_and_mul passthrough to
  process_fp8_weight_tensor_strategy_moe that it misses. Its second hunk adds
  an Fp8OnlineMoEMethod(Fp8MoEMethod) class that no longer exists after the
  online/ refactor, so that part is not carried over.
• [Bugfix] Support non-gated MoE in online quantization and Marlin MoE tile padding #48028 (puririshi98) — online/moe_base.py and
  _zero_padding; both equivalent to the versions here (that PR merges the
  weight and bias loops, this one keeps the original two-block structure).
• [Quantization] Preserve precision in online NVFP4 expert packing #50029 (S1ro1) — online/moe_base.py; equivalent.

All three authors are credited as co-authors.

Two adjacent problems are deliberately left out of scope because they are
independently reachable and already have owners:

• utils/marlin_utils_fp8.py ([Bugfix] Handle single-shard FP8 Marlin MoE padding #50568 mikekg, [Bugfix] Support non-gated MoE in online quantization and Marlin MoE tile padding #48028's second half) —
  _moe_pad_shard_rows hardcodes view(e, 2, n, …), so a non-gated w13
  cannot be viewed as two shards when Marlin pads a tile-misaligned
  intermediate. Same root assumption, but a different layer (kernel-format
  conversion) and live on main today via a path this PR does not touch:
  ModelOptFp8MoEMethod is one of the 13 already-correct allocators, so the
  crash reproduces standalone (shape '[128, 2, 464, 2688]' is invalid for input of size 159645696). [Bugfix] Handle single-shard FP8 Marlin MoE padding #50568 already validates its fix on the real
  nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 checkpoint at TP4 including
  GSM8K; folding it in here would replace validated work with unvalidated work.
• online/nvfp4.py ([Quantization] Preserve precision in online NVFP4 expert packing #50029's second half) — folding the per-expert global
  scale into the weights rounds the product back through weight.dtype before
  quantizing, which is a precision bug, not a shard-count one. Separate PR.

The remaining 11 quantization files in the table above are covered by no open
PR.

Also checked and non-overlapping: #46795 (TPU config only), #48624
(flashinfer_fp4_moe.py), #50196 (oracle/int_wna16.py), #43386 / #47106
(non-gated activation kernels).

Test Plan

Python-only diff, so it can be applied onto an installed vllm without a
rebuild. Every case loads a model, collective_rpcs a probe that dumps every
MoE expert-tensor shape and measures the w13 padding band, then
greedy-generates; baseline and patched runs are compared shape-by-shape and
token-by-token.

# image: vllm/vllm-openai:nightly  sha256:40e504de8278
#        vLLM 0.26.1rc1.dev245+ge2fa28594, built 2026-08-02
# hosts: 8x H200 (SM90)

# non-gated, online per-block fp8, intermediate 1856 -> padded 1920
run_case.sh nemotron_h_nogated  {baseline,patched}

# gated regressions
run_case.sh qwen3_moe_fp8_block   {baseline,patched}   # Fp8MoEMethod block
run_case.sh gpt_oss_mxfp4         {baseline,patched}   # GptOssMxfp4MoEMethod
run_case.sh qwen3_moe_gptq_int4   {baseline,patched}   # AutoGPTQMoEMethod
run_case.sh mixtral_ct_fp8_tensor {baseline,patched}   # untouched-file control

# gated, real weights, TP1, greedy, 4+4 runs
RedHatAI/Mixtral-8x7B-Instruct-v0.1-FP8

Synthetic cases are real upstream config.jsons with the layer and expert
counts shrunk, loaded with --load-format dummy --skip-tokenizer-init, which
exercises create_weights → process_weights_after_loading → kernel setup
without needing checkpoints.

Test Result

The fix does what it claims. nemotron_h routed experts are non-gated
(activation_without_mul("relu2") → relu2_no_mul) and
moe_intermediate_size=1856 is not a multiple of 128, so the online per-block
path rounds 1856 → 1920:

	baseline	patched
w13_weight	[8, 3840, 2688] (2 × 1920)	[8, 1920, 2688]
w13_weight_scale_inv	[8, 30, 21]	[8, 15, 21]
w13 pad band	42,663,936 elems, 93.5 % nonzero	1,376,256 elems, 0 % nonzero

The patched band is exactly 8 experts × 64 rows × 2688 — the 1856→1920 padding
— and fully zeroed. Baseline allocates twice the rows, leaves the entire extra
shard uninitialized, and hands it to the block-FP8 quantizer and the GEMM. The
93.5 % figure reconciles: the old code's two 64-row bands account for 2,752,512
zeros against 2,752,599 measured, i.e. only 87 incidentally-zero elements out of
the 39.9 M it never touched.

Neither run crashes. With dummy weights nothing validates the shape, so this is
the silent-contamination failure mode rather than a load error — which is why
the probe checks the band directly.

No regression on gated paths. status=ok, MoE tensor shapes byte-identical
baseline vs patched, identical generated token ids:

case	path exercised	result
offline_fp8_block	Fp8MoEMethod block branch	identical
offline_mxfp4	GptOssMxfp4MoEMethod incl. the new _setup_kernel asserts	identical
offline_gptq_int4	AutoGPTQMoEMethod	identical
offline_ct_fp8_tensor	CompressedTensorsW8A8Fp8MoEMethod (untouched file)	identical

Real weights, RedHatAI/Mixtral-8x7B-Instruct-v0.1-FP8, TP1, greedy, 4
baseline + 4 patched: MoE shapes identical in all 8 runs; 7/8 produce identical
text. The one divergent patched run is vLLM's run-to-run greedy
nondeterminism, not the diff — that model's MoE method lives in a file this PR
does not touch, and 3 further baseline + 3 further patched runs all agreed.

Since gated shapes and outputs are unchanged and the non-gated path previously
consumed uninitialized memory, there is no accuracy delta to measure on a
supported configuration; the shape and pad-band evidence above stands in for a
model eval.

Not covered on this hardware (stated explicitly rather than implied):

• Gated + padding on the online per-block path. Four gated configs with
  moe_intermediate_size % 128 != 0 (qwen3_moe, all-MoE qwen3_moe,
  glm4_moe_lite, deepseek_v2) fail identically before and after this diff,
  inside the linear block-scaled-mm kernel (the last dimension of x … must be divisible by group_size 128, plus a triton_scaled_mm scale-shape
  assert). Unrelated to this change. The gated _zero_padding slices are
  unchanged by construction — shard count 2 reproduces the same two bands.
• quark (AMD), bitsandbytes MoE, humming, inc (Gaudi) — no hardware.
• AWQ MoE, compressed-tensors w4a4_mxfp4 / w4a8_fp8 — no checkpoint or config
  available locally. These are shape-only substitutions that reduce to the
  previous expression when gated.

Disclosure for reproducibility: the patched runs applied this diff together
with an unrelated online/nvfp4.py change (the #50029 follow-up mentioned
above). That change is inert here — Nvfp4OnlineMoEMethod refuses non-SM100
and H200 is SM90 — so it cannot affect any result reported above.

AI assistance disclosure

This PR was developed with AI assistance (Claude Code), including the
cross-PR overlap analysis and the integration harness. The human submitter has
reviewed every changed line, ran the tests above, and can defend the change
end-to-end.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[aoshen02]

/run ci

[github-actions]

✅ @aoshen02, CI is now available for this PR.

• /ci run starts a CI build.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.

[aoshen02]

/ci run

[github-actions]

✅ Triggered Buildkite CI #82469 for commit e6fc021f8399.

[aoshen02]

Pushed a fix for the two quantization failures in build 82469 — both were tests that construct a MoE method without a FusedMoEConfig and relied on create_weights not reading self.moe:

• test_auto_gptq.py::test_auto_gptq_moe_creates_zero_initialized_expert_biases — object.__new__(AutoGPTQMoEMethod) bypasses __init__, so self.moe was never set (AttributeError: 'AutoGPTQMoEMethod' object has no attribute 'moe').
• test_auto_round.py::test_inc_mxfp4_moe_method_registers_weights_and_builds_kernel — passes moe=cast(Any, "moe-config"), a string sentinel (AttributeError: 'str' object has no attribute 'w13_num_shards').

Reading self.moe inside create_weights is the existing contract — 13 of the other create_weights implementations already do it (modelopt.py:801, the compressed-tensors MoE files, …) and FusedMoEMethodBase.__init__ takes the config precisely for this. So the fix is to give those two tests a stub with the attribute rather than to work around it in the production path, which keeps them CPU-only and sub-second as intended:

# test_auto_gptq.py
method.moe = SimpleNamespace(w13_num_shards=2)

# test_auto_round.py — bound to a name so the pass-through assertion still has a sentinel
expected_moe_config = SimpleNamespace(w13_num_shards=2)
method = INCMxfp4MoEMethod(moe=cast(Any, expected_moe_config))
...
assert captured["kernel_kwargs"]["moe_config"] is expected_moe_config

The second file needed the assertion updated too, since "moe-config" was doing double duty as the config and as the sentinel checked at line 727; is against the bound stub is a slightly stronger pass-through check than the previous ==.

This follows the pattern already in test_auto_gptq.py itself, which stubs moe_config = SimpleNamespace(is_act_and_mul=True, tp_rank=0, ...) in test_routed_experts_loads_per_expert_biases.

Verified on the vllm/vllm-openai:nightly image with this branch's source patched into the installed package:

$ pytest tests/quantization/test_auto_gptq.py::test_auto_gptq_moe_creates_zero_initialized_expert_biases \
         tests/quantization/test_auto_round.py::test_inc_mxfp4_moe_method_registers_weights_and_builds_kernel -q
2 passed in 5.24s

$ pytest tests/quantization/test_auto_gptq.py tests/quantization/test_auto_round.py tests/quantization/test_moe_wna16.py -q
7 failed, 62 passed, 1 skipped in 26.81s

The 7 remaining failures are all test_auto_round_model[...] / test_auto_gptq_quantization_method[TheBloke/...], which fail in huggingface_hub.snapshot_download because that container ran with HF_HUB_OFFLINE=1 and no model cache — not related to this change.

Also checked test_moe_wna16.py's two object.__new__(MoeWNA16Method) tests: they only exercise _setup_kernel and get_fused_moe_quant_config, not create_weights, so they were and remain unaffected.

pre-commit run mypy-3.12 --all-files --hook-stage manual passes, as do the standard hooks on the changed files. The pre-commit GitHub Actions checks on this PR are gated behind pre-run-check (needs a ready/verified label or 4+ merged PRs from the author), so they have not actually executed yet.

The two remaining red jobs in build 82469 look unrelated:

• quantization also had test_experts_int8.py::test_model_experts_int8_startup[4-bfloat16-ai21labs/Jamba-tiny-random] failing on a pydantic config assertion (Chunked prefill is required for mamba cache mode 'align').
• amd-spec-decode-eagle-mi300-1 failed with RuntimeError: invalid argument for fmha_v3_varlen_fwd in test_eagle_correctness_light[ROCM_AITER_FA-deepseek_eagle]. Neither log contains any reference to w13_num_shards.

[aoshen02]

/ci run

[aoshen02]

/ci run

[github-actions]

✅ Triggered Buildkite CI #82494 for commit 4beebe37d764.