[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method by juhi10071998 · Pull Request #49381 · vllm-project/vllm

juhi10071998 · 2026-07-21T23:53:04Z

TL;DR

ModelOpt linear quantization is six near-duplicate LinearMethod classes today, one per format (FP8 per-tensor, FP8 per-channel/per-token, FP8 block-weight-only, NVFP4 W4A4, NVFP4 W4A16, MXFP8). This PR replaces all six with one generic ModelOptLinearMethod, composed from per-QuantKey schemes and driven by a QuantSpec(weight, activation) pair. Adding a format becomes data — a resolve() row plus reusable schemes — not a new class.

Behavior-preserving for existing checkpoints (byte-level parity + GSM8K on all six formats).
MoE and mixed-precision are untouched. A couple of latent bugs get fixed as a side effect.

📄 Design doc: https://docs.google.com/document/d/14ao-WqOeeMIi0XPvjXXm4baxD4xQm_Jb/edit

Motivation

The six classes share ~80% of their structure but each re-implements create_weights / process_weights_after_loading / apply inline, with per-format quirks that are load-bearing and fail silently — a wrong assumption yields garbage output, no error. NVFP4 W4A4 and W4A16 are two whole classes differing only in whether activations are quantized; the three FP8 classes differ mainly in scale shape; kernel selection is scattered; and a new format has to be wired into every dispatch site.

The design question: which parts of a quantized linear layer are derivable from the format's numeric description, and which genuinely need per-format code? Almost everything is derivable from the (weight, activation) QuantKey pair; the true residue is small and named explicitly.

Design

A ModelOpt linear format is fully described by two QuantKeys. resolve(algo, config, prefix) maps the checkpoint's algo string to a QuantSpec(weight, activation) + CkptCtx (read-only over the config). One generic method runs a fixed lifecycle:

   resolve(algo) → QuantSpec(weight_key, activation_key)
            │
            ▼
   ModelOptLinearMethod
     wkey = SCHEME_FOR[weight_key]     akey = SCHEME_FOR[activation_key]
            │
   create_weights   → wkey.create_weights(WEIGHT) → akey.create_weights(ACT)
                      → select_linear_kernel(spec) → expose_input_quant_key
   process_weights  → wkey.process → akey.process → maybe_fuse_global_scales
                      → format_scheme hooks → kernel.process_weights_after_loading
   apply            → format_scheme wraps → kernel.apply_weights

QuantKeyScheme — one scheme per QuantKey, content-keyed and role-parameterized.
Each knows how to allocate/post-process its key in the weight or activation slot, and rejects any role it hasn't validated (no silent wrong-role fall-through). Schemes are shared — NVFP4 W4A4 and W4A16 use the same KNvfp4Static weight scheme. Seven schemes cover all six formats.
select_linear_kernel(spec) picks the kernel family from the weight key (fp4 /
mxfp8 / fp8) in one place.
Front-end. Each config exposes linear_algo() (just the algo string); the
LinearBase arm calls build_linear_method(config, algo, prefix). The config names a
format, not a method class — behavior is derived.

Two extension seams so future formats get tools, not forks:

FormatScheme — optional per-format hooks (extra_weights, pre/post_process, and
an apply wrapper for compute-time residue) that compose around the key schemes. Default no-op.
LINEAR_METHOD_BUILDERS — a registry (empty by default) so a format that genuinely
can't be a (weight, activation) pair registers its own LinearMethodBase by algo, through the single build_linear_method indirection.

MoE and mixed-precision are unchanged. MoE keeps its per-format methods (the
ModelOpt* class-name coupling routed_experts.py gates on is preserved); the mixed MoE fork is byte-identical. Only the linear arm changed — a mixed checkpoint runs new-linear beside old-MoE with no new plumbing.

Behavior preservation + fixes

Behavior-preserving for all six formats (evidence below). A few deltas fall out of unifying, all improvements:

FP8 PbWo bug fix. The old class skipped the block kernel's post-load via a misnamed
guard, so under compiled serving it ran the GEMM on un-repacked weights → garbage (acc 0.000, invalid-rate 1.0). The generic method runs the post-load → correct (0.897).
Blackwell FP8_PB_WO NaN fix. The config now implements has_blocked_weights() (on
both the homogeneous FP8 config and the mixed config), enabling +quant_fp8 so DeepGEMM gets UE8M0-packed scales instead of NaN.
MXFP8 → weight_loader_v2. MXFP8 was the one format missing from
WEIGHT_LOADER_V2_SUPPORTED; the generic method self-registers. Byte-identical.

W4A16 checkpoints that ship an on-disk `input_scale`

A W4A16_NVFP4 (weight-only) checkpoint can still carry activation input_scale tensors (e.g. a relabeled W4A4 export, or injected downstream). Since resolve("W4A16_NVFP4") sets activation=None, no such param is registered and the tensor is orphaned. A small _DropInputScale FormatScheme registers a placeholder, loads the scale into it, and drops it after load — a no-op for genuine exports (which carry none), kept for backward compatibility with already-published ModelOpt checkpoints. Serves weight-only via Marlin.

Partial-block FP8_PB_WO (absorbs #53132)

A block-FP8 weight whose output width isn't a multiple of 128 (a partial trailing block — e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is handled by a _Fp8PbWoPartialBlock FormatScheme: pad the weight to a block boundary before the kernel post-load, run the GEMM on the padded weight, and trim the output back to the logical width (bias added after). This is #53132's approach, re-expressed as compute-time residue on the generic method (via the new FormatScheme.apply hook) rather than a per-format class. No-op for the common block-aligned case. Verified bit-identical to a no-pad path at width 2624.

Adding a format (developer guide, in-code)

Composes as a (weight, activation) pair → add a QuantKeyScheme per new key to
SCHEME_FOR + a resolve() row. No new class.
Needs format-wide residue → also return a FormatScheme from that row.
Genuinely can't be a key pair → write a LinearMethodBase, register it in
LINEAR_METHOD_BUILDERS by algo.

Testing

Rebased current onto main (which includes #53132 Kimi K3 nvfp4).

Unit — pytest tests/quantization/test_modelopt.py -k "not checkpoint_setup":
33 passed. Dispatch/config tests assert the resolved QuantSpec + ModelOptLinearMethod;
plus the extension-seam builder test, a faithful port of #53132's partial-block test, and the upstream mixed-precision/mapper tests. ruff check + ruff format --check clean; typos clean.

Per-format evidence — Qwen3-8B. Parity is the deterministic gate
(VLLM_BATCH_INVARIANT=1 + --enforce-eager): per-layer weight/kernel/expose byte-hash + per-token prefill logit-diff (old vs generic) against a proven-0.0 old-vs-old baseline.
GSM8K (1319 Q · 5-shot · T=0 · 512 tok) is the coarse end-to-end check.

Format	weight/kernel/expose hash	logit Δ	GSM8K	Verdict
NVFP4 W4A4	byte-identical	`0.0`	0.89765	✅ identical
W4A16 NVFP4	byte-identical	`0.0`	0.87794	✅ identical
FP8 per-tensor	byte-identical	`0.0`	0.88476	✅ identical
FP8 PcPt	byte-identical	`0.0`	0.89689	✅ identical
MXFP8	byte-identical	`0.0`	0.89538	✅ identical
FP8 PbWo	weights differ ¹	`0.0` (eager)	0.89689	✅ bug fix

Logits are byte-identical for all six → behavior-preserving. GSM8K deltas across builds are ≤2 Q (torch.compile autotune noise, not math), invalid-rate 0.

¹ PbWo weights differ by design: the generic method runs the block-kernel post-load the old class skipped (a misnamed-guard bug). Identical under eager; under compiled serving the old class produces garbage (0.000) while the generic path is correct — the fix above.

Not duplicating existing work

Replaces the in-tree ModelOpt linear methods; not a parallel implementation.

[Bugfix][Quantization] Run block kernel post-processing for ModelOpt FP8_PB_WO #48422 (FP8_PB_WO post-load bug) — fixed here as a consequence of unifying; if it
lands first, the PbWo behavior here becomes a no-op and we rebase.
Support kimi k3 nvfp4 checkpoint #53132 (Kimi K3 nvfp4) — merged; its partial-block FP8_PB_WO support is absorbed
above.
[Quantization][ModelOpt] Serve modelopt_mixed checkpoints with FP8_PB layers #50617 (serve modelopt_mixed with FP8_PB layers) — will be closed in favor of
this PR. Its has_blocked_weights() fix is already incorporated here; any remaining FP8_PB serving work is reintroduced on top of the generic method.

AI assistance

AI assistance (Claude Code) was used; the diff and results were human-reviewed.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

mergify · 2026-07-31T07:12:11Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @juhi10071998.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mgoin · 2026-08-03T17:37:22Z

@@ -1412,6 +1416,13 @@ def load_weights(
                param = getattr(self.get_submodule(submodule), attr, self)
            else:
                param = getattr(self, name, self)
+            if param is self:
+                # `name` is not a real parameter on this layer (the getattr
+                # above fell back to the module itself) — e.g. an on-disk
+                # activation input_scale for a weight-only quantized layer that
+                # registers no such param. Skip it instead of crashing in
+                # self.weight_loader(self, ...).
+                continue


This changes general behavior for the linear layers and I think shouldn't be needed if we respect the checkpoint format that the model's config define. Basically I don't understand when we would make parameters that don't load right?

Thanks Michael for the discussion this morning.
As discussed, this was added to support the w4a16_nvfp4 checkpoint from ModelOpt with silently injected input_scales. I'm meeting some folks internally tomorrow to convey our discussions below-

quant_algo in the config to honor and map the weights present in checkpoint --> config should be able to load the model weights

Uniform QuantKeyScheme behavior across the checkpoints.

That said, this will be a breaking change for prev ModelOpt w4a16_nvfp4 ckpts with input_scales. The redesign just surfaced those inconsistencies as it was easier to patch methods previously. We just need more thought/ justification and a better place to add those patches now with the QuantKey based redesign.

mgoin · 2026-08-03T17:38:53Z

+    # Drop an on-disk activation input_scale on weight-only NVFP4 (W4A16), where
+    # resolve() sets activation=None and registers no such param. Covers the
+    # simple linears (down_proj, o_proj) via the loader's unexpected-key check;
+    # merged linears (qkv_proj, gate_up_proj) also rely on the LinearBase
+    # `param is self` skip.
+    _ignore_unexpected_suffixes = QuantizationConfig._ignore_unexpected_suffixes + (
+        ".input_scale",
+    )


As we covered offline, the quantization config override should be handled within the kernel oracle, not the checkpoint frontend itself. So I think we don't need this

yes, aligns with previous point

mgoin · 2026-08-03T17:57:14Z

+    def linear_algo(self) -> str:
+        return "MXFP8"


Why do we need this function if the other variants just return return self.quant_method? Seems like we could just access self.quant_method

you're right- makes sense, it was not symmetrically present for all the QuantizationConfig classes, just added for the MXFP8.

mgoin · 2026-08-03T18:00:36Z

+# ===========================================================================
+# Generic QuantKey-driven linear method
+#
+# One ``ModelOptLinearMethod`` replaces the six per-format linear method
+# classes. It composes a per-QuantKey weight scheme + activation scheme (from
+# the ``QuantSpec`` pair produced by ``resolve``), runs a fixed create/process
+# lifecycle, selects the kernel from the pair, and applies. See
+# ``linear_design_concrete.md`` for the design and caveats C1-C13.
+#
+# Adding a format (developer guide):
+#   * Composes as a (weight, activation) key pair -> add a QuantKeyScheme per
+#     new key to SCHEME_FOR, plus a resolve() row returning the QuantSpec. No
+#     new method class. (This is how all six existing formats are built.)
+#   * Needs format-wide residue but the same lifecycle -> also return a
+#     FormatScheme subclass from that resolve() row (extra_weights / pre_process
+#     / post_process hooks).
+#   * Genuinely cannot be a key pair (different lifecycle) -> write a bespoke
+#     LinearMethodBase and register it in LINEAR_METHOD_BUILDERS by algo.
+#   In all cases add the algo to the owning config's linear_algo()/validation.
+# ===========================================================================


Please clean up the in-progress specific comments, such as "replaces the six...", the design file and CX markers that don't exist.

mgoin · 2026-08-03T18:02:29Z

+# Weight-loader "unloaded shard" marker — FP8 family fills scales with it; the
+# NVFP4/MXFP8 families deliberately do not (C3, load-bearing asymmetry).
+SENTINEL = torch.finfo(torch.float32).min


Cruft. I'm sure we have this sentinel in some fp8 utils elsewhere

Good point, I couldn't find a predefined sentinel value. The value is torch.finfo(torch.float32).min and it's open-coded in 7 places, each defining its own:

▎ - fp8_utils.py:1302 — inside create_fp8_scale_parameter()
▎ - fp8_utils.py:1317 — inside create_fp8_input_scale()
▎ - fbgemm_fp8.py:138 — weight_scale
▎ - quark_w8a8_fp8.py:171 — weight_scale
▎ - quark_w8a8_fp8.py:180 — input_scale
▎ - quark_w4a8_mxfp4_fp8.py:140 — input_scale
▎ - modelopt.py — updated

I've added FP8_SCALE_SENTINEL to fp8_utils.py and switched ModelOpt to it. I can update all of these in this PR itself if you'd prefer, or do it as a separate follow-up.

mgoin · 2026-08-03T18:03:05Z

+class CkptCtx:
+    """Per-checkpoint facts a QuantKey cannot carry."""
+
+    serialized: bool


Isn't it always serialized as a checkpoint format? When is this information needed?

You're right, it isn't used from what i understand

It existed in the previous code but only as dead branches — weight_dtype = fp8 if is_checkpoint_fp8_serialized else params_dtype, and the if serialized: around registering weight_scale/input_scale — plus a guard in ModelOptFp8PbWoLinearMethod and asserts in the MoE methods that can't fire.

vllm/vllm/model_executor/layers/quantization/modelopt.py

Line 561 in c810937

if not self.quant_config.is_checkpoint_fp8_serialized:

Nothing can reach the False case: a ModelOpt config is only built when override_quantization_method matches a quant_algo that's already in the checkpoint, so there's no path that constructs one for an unquantized model.

I've deleted it entirely here — the CkptCtx.serialized field, requires_serialized and its guard, and the four branches. CkptCtx now just carries group_size. [and placeholder for any future specific ckpt ctx quirks]

@pavanimajety — just double checking if you see any concerns here.

mgoin · 2026-08-03T18:07:24Z

+    # Whether the base advertises the kernel's input_quant_key on the layer
+    # (enables upstream activation-quant fusion). Behavior-preserving per-format:
+    # the old NVFP4 W4A4 method exposed it, the old FP8 method did NOT — and the
+    # FP8 kernel *does* return a static key, so exposing there flips activation
+    # quant into a fused path and diverges (C2). Read off the weight scheme;
+    # default True (NVFP4), False on the FP8 schemes to preserve today's
+    # behavior. Adopting FP8 fusion is a separate deliberate change.
+    exposes_input_quant_key: bool = True


There is no need for the checkpoint format to be aware of this. How expose_input_quant_key(layer, self.kernel) resolves only relies on the kernel that is selected and whether it supports this feature

Makes sense — should we just call it unconditionally then, like the CT schemes do?

I'd added the flag to preserve current behavior.
FP8 per-tensor is the one that actually differs: the cutlass/flashinfer scaled_mm kernels return kFp8StaticTensorSym, but the existing ModelOptFp8LinearMethod never exposed it.
The other schemes turned out to be no-ops either way — their kernels return None regardless, so the flag wasn't suppressing anything.

Dropped it and made the call unconditional to match CT. will re-run the gsm8k again

mergify · 2026-08-04T00:34:50Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @juhi10071998.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

juhi10071998 · 2026-08-04T02:49:38Z

Thanks @mgoin, I've addressed your current comments, and rebased onto current main.

On the ones we discussed:

linear_algo() is gone — ModelOptMxFp8Config just sets self.quant_method like the others.
Cleaned up the CX markers, design-file references and other in-progress comments. Kept the "adding a format" guide and pointed it at quant_method validation.
FP8_SCALE_SENTINEL now lives in fp8_utils.py (details in the thread — happy to convert the other call sites too).
Dropped serialized/requires_serialized. You were right that it's always True, so the branches were dead. Kept CkptCtx itself for group_size.
Dropped exposes_input_quant_key and call expose_input_quant_key unconditionally like the CT schemes.

Two bugs turned up while testing this, both of which reproduce on unmodified main:

Mxfp8LinearKernel is the only linear kernel base that doesn't declare input_quant_key() — MMLinearKernel, NvFp4LinearKernel and ScaledMMLinearKernel all do. So calling the helper on an MXFP8 kernel raises AttributeError instead of no-op'ing. Added the same None default to that base.
Serving an FP8_PB_WO checkpoint on Blackwell dies in DeepGEMM with CUDA_ERROR_LAUNCH_FAILED. ModelOptFp8Config implements neither weight_block_size nor has_blocked_weights(), so +quant_fp8 never gets enabled, QuantFP8 falls back to forward_native and emits unpacked fp32 group scales where DeepGEMM wants UE8M0-packed ones. VLLM_USE_DEEP_GEMM_E8M0=0 isn't a workaround — DeepGEMM then says "Unsupported architecture or scaling factor types", so packed really is the only option here. Fp8Config gets this via weight_block_size and CompressedTensors implements the method; ModelOpt was the last block-FP8 config without it. Fixed and confirmed by serving.

Also worth flagging: the rebase conflict was the three deleted FP8 classes vs #48861, which had landed out_dtype = model_config.dtype inside each of them. Resolving it as "keep the deletion" would have quietly reverted that fix, so I carried it into ModelOptLinearMethod.

GSM8K, 1319 questions / 5-shot / T0, before vs after these changes:

format	before	after	Δ
NVFP4 W4A4	0.87794	0.87794	identical, same output tokens
FP8 PcPt	0.89613	0.89613	identical, same output tokens
FP8 PbWo	0.90296	0.90296	identical, same output tokens
MXFP8	0.90068	0.90068	same accuracy
W4A16 NVFP4	0.88021	0.88476	+6 Q
FP8 per-tensor	0.89158	0.88552	−8 Q

The three byte-identical ones are the formats batch-invariance actually stabilises. The two that moved are the two it doesn't: W4A16 pins Marlin, and ModelOpt FP8 per-tensor has no VLLM_BATCH_INVARIANT path (native Fp8LinearMethod has one that dequantises to BF16 — ModelOpt never did, before or after this PR). Re-running FP8 per-tensor twice on the same build gives −3 Q / −996 tokens, so −8 is that format's noise, not a change in behaviour.

pre-commit (ruff check + format) is clean on all changed files, and tests/quantization/test_modelopt.py + tests/fusion/test_quant_activation_contract.py pass (26 passed, 1 skipped).

cc @pavanimajety for viz.

juhi10071998 · 2026-08-04T03:58:40Z

Dropped the _ignore_unexpected_suffixes entry and both param is self guards in linear.py as well. This means that now the ModelOpt checkpoints need to contain exact set of weight keys as specified by the quant_algo for a given layer.

This means that the w4a16_nvfp4 checkpoint containing a4 activation scales will fail to load. The new redesign just surfaced the inconsistencies we previously had because we were patching each Quant method class with changes restricted to ModelOpt but now we align more with vLLM fundamental design pattern.

If such a checkpoint ever shows up, the config is misdescribing its own contents and that's a producer-side fix likely.

juhi10071998 · 2026-08-05T19:42:34Z

hi @pavanimajety could i get some reviews on this PR when you get a chance, would really value your inputs here since this changes the ModelOpt vllm flow significantly. Wanted to make sure everything is captured

juhi10071998 · 2026-08-05T19:43:27Z

thanks for your review @mgoin , I've addressed the existing comments, please let me know if there's anything missing/ not clear, happy to provide more details

Mxfp8LinearKernel is the only linear-kernel base that does not implement input_quant_key(). MMLinearKernel, NvFp4LinearKernel and ScaledMMLinearKernel all declare it returning None, meaning "this kernel quantizes its own input". A caller that routes an MXFP8 kernel through expose_input_quant_key therefore raises AttributeError instead of getting the intended no-op. Add the same default to the MXFP8 base so the fusion bridge can be used uniformly across kernel families. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

- Drop linear_algo(). ModelOptMxFp8Config now sets self.quant_method like the other configs, so get_quant_method reads the attribute directly instead of going through a method that three of four configs implemented identically. - Remove the in-progress comments, design-document references and caveat markers. The "adding a format" guide stays and now points at quant_method validation. - Add FP8_SCALE_SENTINEL to fp8_utils and use it in place of a local copy of torch.finfo(torch.float32).min. The value is open-coded in several other places; converting those is left to a follow-up. - Drop CkptCtx.serialized and QuantKeyScheme.requires_serialized. A ModelOpt config is only constructed from a quant_algo already present in the checkpoint, so the non-serialized case is unreachable and the branches guarding it were dead. The MoE methods already assert the same invariant. CkptCtx keeps group_size. - Drop QuantKeyScheme.exposes_input_quant_key and call expose_input_quant_key unconditionally, as the CompressedTensors schemes do. Whether a layer can accept a pre-quantized activation is a property of the selected kernel, and the kernel already answers it by returning None from input_quant_key(); a per-format flag could only override that from the wrong layer. The last item is the only observable difference: FP8 per-tensor layers now advertise layer.input_quant_key. No code reads that attribute today, so the kernels still quantize their own inputs exactly as before. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Serving a ModelOpt FP8_PB_WO checkpoint on Blackwell fails in DeepGEMM: RuntimeError: CUDA driver error (jit/handle.hpp:154): 719 (CUDA_ERROR_LAUNCH_FAILED, unspecified launch failure) VllmConfig enables the "+quant_fp8" custom op only when the quantization config reports weight_block_size or implements has_blocked_weights(). ModelOptFp8Config has neither, so QuantFP8 falls back to forward_native and emits unpacked fp32 group scales, while DeepGemmFp8BlockScaledMMKernel asks DeepGEMM for UE8M0-packed ones. Serving with VLLM_USE_DEEP_GEMM_E8M0=0 is not a workaround: DeepGEMM then reports "Unsupported architecture or scaling factor types", i.e. UE8M0 scales are required on this architecture, so the packed layout is the only correct one. Implement has_blocked_weights() so the op is enabled and forward_cuda produces packed scales. Only FP8_PB_WO reports True, leaving FP8 and FP8_PER_CHANNEL_PER_TOKEN unchanged. Fp8Config already satisfies the same check through weight_block_size, and CompressedTensorsConfig implements the method; ModelOpt was the remaining block-FP8 config without it. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Carry forward vllm-project#48861, which fixed out_dtype in ModelOptFp8LinearMethod, ModelOptFp8PcPtLinearMethod and ModelOptFp8PbWoLinearMethod. Those classes are replaced by ModelOptLinearMethod here, which still took out_dtype from torch.get_default_dtype() while input_dtype already came from the model config. The two are the same value, so read the model config once and use it for both. Without this the generic method would silently reinstate the fp32-vs-bf16 mismatch that vllm-project#48861 removed. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Restores the blank line that ruff-format expects before the CkptCtx dataclass, lost when the SENTINEL constant above it was deleted. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Removes the ".input_scale" entry from ModelOptQuantConfigBase's _ignore_unexpected_suffixes and the two "param is self" guards in MergedColumnParallelLinear.weight_loader and QKVParallelLinear.weight_loader. Both existed to tolerate a W4A16_NVFP4 checkpoint that ships activation input_scale tensors the weight-only path never registers. Checking a real ModelOpt W4A16 export (Qwen3-8B, nvfp4_weight_only-kv_fp8_cast) shows 904 tensors and zero named input_scale, so the case does not arise: a checkpoint that declares W4A16_NVFP4 does not carry activation scales. Guarding against it also meant changing the shared weight loader for every quantization method to accommodate one hypothetical ModelOpt checkpoint. If such a checkpoint ever appears, the config is misdeclaring its own contents and should be fixed at the producer. linear.py is now limited to removing the deleted classes from WEIGHT_LOADER_V2_SUPPORTED. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

vllm-project#51093 restored input_dim/output_dim on the transposed weight in ModelOptFp8LinearMethod.process_weights_after_loading. That class is replaced by ModelOptLinearMethod here, so port the fix into KFp8StaticTensor.process, which performs the same transpose. Replacing the ModelWeightParameter with a plain Parameter drops the dim attributes that Humming reads; after the transpose the layout is [in, out], so input_dim is 0 and output_dim is 1. Also rewrites the accompanying test against the generic method. Verified it still guards the behaviour: removing the two lines fails it with "AttributeError: 'Parameter' object has no attribute 'input_dim'". Note vllm-project#51093 fixed only the per-tensor path. ModelOptFp8PcPtLinearMethod performed the same transpose without setting the dims, and KFp8StaticChannel inherits that, so per-channel FP8 likely has the same gap upstream. Left alone here to keep this PR behaviour-preserving. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

One source of truth for the supported linear algos. LINEAR_ALGOS maps each algo to its owning config and to the mixed-precision sub-config that holds its checkpoint parameters. QUANT_ALGOS, both configs' quant_algo validation, their error messages, and the mixed-precision lookup are all derived from it, so adding a format is one row here plus one row in resolve(). That replaces three hand-maintained lists, two of which restated themselves in the error string and could drift from the tuple they described. Deriving the mixed-precision lookup also fixes a bug: it listed only four algos, so an FP8_PER_CHANNEL_PER_TOKEN or FP8_PB_WO layer in a MIXED_PRECISION checkpoint fell through to UnquantizedLinearMethod, which registers no scale parameters and fails weight loading. It was a faithful port of the old four-way fork over the per-format classes; the generic method had already removed the reason for it, since resolve() reads nothing from the sub-config for those algos. The lookup is now complete by construction, and a parametrised test covers every entry so a new format cannot miss it. Not grouping the algos by weight bit width, since that is derivable from the QuantSpec resolve() already returns; a second table would be one more thing to keep in sync. Also: - Use the conventional shape names in Shapes. It had invented its own vocabulary -- out_parts, in_, out, nparts -- where the rest of vLLM says output_partition_sizes, input_size_per_partition and output_size_per_partition; in_ carried a trailing underscore only to avoid the keyword. The MXFP8 error message is rewrapped for the longer name and now says "input size" rather than "in", which only read as a word when the field was called in_. - Drop the three "format is experimental and could change" warnings (FP8, NVFP4, MXFP8). They fire on every load and no longer earn the line. - Reword the NVFP4 global-scale warnings: "for parallel layers" -> "for fused weights", and "a shared global scale" -> "the same global scale", since "shared" can read as one parameter serving all shards rather than equal values. - Warn when the static FP8 input scale differs across fused shards. The weight path handles disagreement by requantizing against the max, which repairs the values, so it needs no warning. The activation path cannot: activations do not exist at load time, so collapsing to the max leaves every smaller-range shard permanently quantized against too large a scale. NVFP4 already warned here; per-tensor FP8 was silent. - State the key invariants so mypy can see them. QuantSpec's fields are QuantKeyField, so a weight key may statically be a name or None even though resolve() only ever yields concrete QuantKeys, and self.kernel is assigned in create_weights rather than __init__. Adds narrowing asserts and an annotation, and checks quant_algo for None before the mixed-precision lookup rather than relying on dict.get(None) missing. No behaviour change. mypy on this file goes from 126 errors on main to 70, adding none. - Simplify the "adding a format" note. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

…cale Genuine ModelOpt W4A16 exports carry no activation input_scale, but some legacy/mislabeled checkpoints ship one. Restore the FormatScheme hook that registers a placeholder input_scale (also satisfying the fused qkv/gate_up loaders), drops it after load, and warns once when a scale was actually present. resolve("W4A16_NVFP4") returns _DROP_INPUT_SCALE as its format scheme. No shared-loader changes; a no-op for genuine exports. Verified: a qwen3 W4A16 checkpoint carrying 108 input_scale loads via MarlinNvFp4LinearKernel (weight-only), the deprecation warning fires, and GSM8K scores 0.89538 (invalid-rate 0.0). AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

…ompat Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Upstream parity: vllm-project#50273 added both to ModelOptNvFp4W4A16LinearMethod and the generic method dropped them. LinearBase sets both itself, so this only matters for ParallelLMHead, which does not -- humming reads them in prepare_humming_linear_layer_config. Verified on a qwen3 W4A16 NVFP4 checkpoint: --linear-backend=auto selects MarlinNvFp4LinearKernel and humming selects HummingNvFp4LinearKernel, both generating correctly. AI assistance (Claude Code) was used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

…mpotency comment Mixed-precision checkpoints with an FP8_PB_WO layer need +quant_fp8 enabled on Blackwell just like the homogeneous config, or QuantFP8 falls back to forward_native and the DeepGEMM launch fails. Mirror ModelOptFp8Config's has_blocked_weights on ModelOptMixedPrecisionConfig, resolved per layer. Also expand the MXFP8 process idempotency comment to name the env flag (VLLM_MXFP8_EMULATION_DEQUANT_AT_LOAD) and the reload case, so it is clear the scheme is validate-only and the load-time dequant lives in the emulation kernel, not here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

…trim FormatScheme A FP8_PB_WO output width that is not a multiple of 128 (a partial trailing block; e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is padded up to a block boundary with zeros before the kernel post-load, the GEMM runs on the padded weight, and the output is trimmed back to the logical width with bias added after. This is wei-zhao vllm-project#53132's approach, expressed as a FormatScheme (_Fp8PbWoPartialBlock) on the generic method rather than a per-format class -- post_process pads the weight, apply wraps the kernel to trim the output. A new FormatScheme.apply hook makes this compute-time residue expressible without touching ModelOptLinearMethod. KFp8Block128 sizes the weight_scale by cdiv to match the (padded) block count. No-op for the common block-aligned case. Chosen over a no-pad variant because it handles any width and does not depend on DeepGEMM's undocumented N-alignment tolerance -- it is the logic upstream already merged. Test is a faithful port of vllm-project#53132's test_modelopt_fp8_pb_wo_hides_output_padding for the generic method. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

The rebase onto vllm-project#53132/Kimi K3 brought in mla_attention.py's _get_kv_b_proj_input_dtype, which imported and isinstance-checked the deleted ModelOptFp8PbWoLinearMethod (a runtime ImportError for MLA + block-FP8 models, caught by mypy). Map it to the generic ModelOptLinearMethod + the block-FP8 weight key (kFp8Static128BlockSym), same absorption as deep_gemm_warmup. Also reword a test docstring (mis-scaled -> wrong scales) to satisfy the typos hook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

mgoin · 2026-09-01T16:26:10Z

/ci run

github-actions · 2026-09-01T16:26:29Z

✅ Triggered Buildkite CI #86648 for commit d0aac93e36f0.

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin · 2026-09-01T18:23:23Z

/ci run

github-actions · 2026-09-01T18:23:45Z

✅ Triggered Buildkite CI #86675 for commit ec7fa343c283.

…ey-driven method (vllm-project#49381) Signed-off-by: Juhi Mittal <juhim@nvidia.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: mgoin <mgoin64@gmail.com>

…cales (linear + MoE) Rebased onto the vllm-project#49381 ModelOpt LinearMethod redesign. NVFP4 per-tensor global scales (weight_scale_2 / input_scale) were allocated with torch.empty and never validated, so a checkpoint that omits one folds uninitialized memory into the dequant math and silently corrupts the layer instead of failing. Linear: the generic ModelOptLinearMethod builds NVFP4 weights via the KNvfp4Static / KNvfp4Dynamic QuantKey schemes. NaN-init their global scales and reject any scale still carrying NaN / zero / non-finite values in process(), before it is folded into the runtime global scale. W4A16 only builds the weight scheme (activation key is None), so only weight_scale_2 is required there; the input scale is dropped as before. MoE: ModelOptNvFp4FusedMoE gets the same NaN-init plus a per-expert check that validates only the scales the selected backend consumes. HUMMING and MARLIN drop the input scales (and MARLIN serves the W4A16 MoE path whose checkpoints legitimately omit them), so only the weight global scales are required for them; every other (W4A4) backend folds both input scales in. Signed-off-by: Aditya Jha <4adityajha@gmail.com>

…ey-driven method (vllm-project#49381) Signed-off-by: Juhi Mittal <juhim@nvidia.com> Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: mgoin <mgoin64@gmail.com>

juhi10071998 marked this pull request as ready for review July 21, 2026 23:54

juhi10071998 requested review from AndreasKaratzas, mgoin, pavanimajety, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and zyongye as code owners July 21, 2026 23:54

claude Bot reviewed Jul 21, 2026

View reviewed changes

juhi10071998 changed the title ~~[ModelOpt] Replace 6 per-format linear methods with one generic QuantKey-driven method~~ [ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method Jul 21, 2026

juhi10071998 force-pushed the modelopt-linear-generic branch from 04cd6ab to 519bf5e Compare July 21, 2026 23:57

mergify Bot added the quantization label Jul 23, 2026

Qyx5-5 mentioned this pull request Jul 24, 2026

[Core][Kernel] Support dynamic activation scaling for ModelOpt NVFP4 #49715

Open

4 tasks

juhi10071998 force-pushed the modelopt-linear-generic branch from 519bf5e to 3b6dcd5 Compare July 27, 2026 18:05

mergify Bot added the needs-rebase label Jul 31, 2026

juhi10071998 force-pushed the modelopt-linear-generic branch from 3b6dcd5 to a68032b Compare July 31, 2026 17:26

mergify Bot removed the needs-rebase label Jul 31, 2026

juhi10071998 force-pushed the modelopt-linear-generic branch from 8b03848 to a8e10ce Compare August 3, 2026 17:34

mgoin reviewed Aug 3, 2026

View reviewed changes

mergify Bot added the needs-rebase label Aug 4, 2026

juhi10071998 force-pushed the modelopt-linear-generic branch from a8e10ce to 8df04f8 Compare August 4, 2026 02:49

mergify Bot removed the needs-rebase label Aug 4, 2026

juhi10071998 force-pushed the modelopt-linear-generic branch 2 times, most recently from 7b44d16 to e4b575c Compare August 4, 2026 19:21

juhi10071998 and others added 14 commits September 1, 2026 15:50

[ModelOpt] Note _DropInputScale is for exported-checkpoint backward c…

4d3c539

…ompat Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

juhi10071998 force-pushed the modelopt-linear-generic branch from 905b7e0 to d0aac93 Compare September 1, 2026 15:51

Merge main into modelopt-linear-generic

ec7fa34

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin approved these changes Sep 1, 2026

View reviewed changes

mgoin merged commit 7a977c0 into vllm-project:main Sep 1, 2026
125 checks passed

huthvincent mentioned this pull request Sep 2, 2026

[Bugfix][Quantization] Run block kernel post-processing for ModelOpt FP8_PB_WO #48422

Closed

This was referenced Sep 3, 2026

[Bugfix][Quantization] Reject NVFP4 checkpoints with missing global scales (linear + MoE) #54444

Open

[RFC]: Unify ModelOpt MoE methods (mirror of #49381) #55339

Open

ima-helikoptaaa mentioned this pull request Sep 9, 2026

[ModelOpt] Unify MoE methods onto QuantSpec + ModelOptMoEMethod #55997

Open

GirasoleY mentioned this pull request Sep 14, 2026

[Bugfix][MLA] Keep MXFP8 prefill activations in model dtype #56911

Open

4 tasks

Uh oh!

Uh oh!

Conversation

juhi10071998 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

Motivation

Design

Behavior preservation + fixes

W4A16 checkpoints that ship an on-disk input_scale

Partial-block FP8_PB_WO (absorbs #53132)

Adding a format (developer guide, in-code)

Testing

Not duplicating existing work

AI assistance

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

mergify Bot commented Jul 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juhi10071998 Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juhi10071998 Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juhi10071998 Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

juhi10071998 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juhi10071998 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

juhi10071998 commented Aug 5, 2026

Uh oh!

juhi10071998 commented Aug 5, 2026

Uh oh!

mgoin commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

mgoin commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

juhi10071998 commented Jul 21, 2026 •

edited

Loading

W4A16 checkpoints that ship an on-disk `input_scale`

juhi10071998 Aug 3, 2026 •

edited

Loading

juhi10071998 Aug 3, 2026 •

edited

Loading

juhi10071998 Aug 4, 2026 •

edited

Loading

juhi10071998 commented Aug 4, 2026 •

edited

Loading

juhi10071998 commented Aug 4, 2026 •

edited

Loading