[Quant][Prototype] Manual fusion AR+RMS+Quant by mgoin · Pull Request #42597 · vllm-project/vllm

mgoin · 2026-05-14T03:55:53Z

Builds on #42469.

Manual fusion of all-reduce + residual-add + RMSNorm + activation-quant at decoder layer boundaries. Under TP>1, those ops at each layer's input and post-attention norms collapse into a single FlashInfer kernel.

How it works

fused_ar_rms_norm_quant (in vllm/model_executor/layers/fusion/ar_rms_quant.py) picks one of four paths:

FlashInfer AR + add + RMS + Quant — one kernel, returns a QuantizedActivation the downstream linear consumes directly.
FlashInfer AR + add + RMS — same kernel without quant; downstream linear quantizes itself.
Explicit AR + fused RMS+Quant (vLLM C++) — when path 1 isn't applicable but a fused RMS+quant kernel exists.
Explicit AR + plain RMSNorm — fallback (CPU, ROCm, no quant key, no flashinfer).

A linear kernel declares via kernel.input_quant_key() whether it can accept pre-quantized activations; the scheme propagates this to layer.input_quant_key and the helper reads it to choose between the quant-capable (1/3) and no-quant (2/4) paths. Non-CUDA kernels inherit None and always land in path 4.

Wired up

llama.py, qwen2.py, qwen3.py: o_proj and down_proj set reduce_results=False; decoder forward routes through the helper at input norm, post-attention norm, and final norm.
FP8 (kFp8StaticTensorSym) on Cutlass and FlashInfer scaled-MM kernels; NVFP4 (kNvfp4Dynamic) on the FlashInfer Cutlass NVFP4 kernel. Other FP8 strategies (block, per-token-channel) and other linear kernels fall through to path 4.

Test plan

Llama-3.2-1B-FP8 TP=1 + TP=2 — manual fusion fires; output coherent.
Llama-3.2-1B (unquant) TP=1 + TP=2 — AR-fused RMS fires.
NVFP4 checkpoint on B200.
CPU / ROCm smoke (path 4).
Qwen3 FP8 TP=2.

AI-assisted.

Introduce an opt-in path for model code to pre-quantize activations for quantized linears, so RMSNorm + input-quant fusion can be expressed directly in model code rather than relying on torch.compile passes. - Add QuantizedActivation + rms_norm_input_quant helper. - Split CutlassNvFp4LinearKernel.apply_weights into apply_weights (chain) and apply_quantized (pre-quantized entry point). - Have CompressedTensorsW4A4Fp4 advertise layer.input_quant_key and implement quantize_input; route QuantizedActivation through to the kernel's apply_quantized. - Wire Qwen3 decoder layer to call rms_norm_input_quant before qkv_proj and gate_up_proj. Only the Cutlass NVFP4 kernel opts in; FlashInfer/fbgemm fall back to the legacy in-apply quant path. Other quant methods and other models are unchanged. Requires --enforce-eager (or piecewise compile) to take effect; the isinstance(QuantizedActivation) dispatch breaks the torch.compile graph by design, which is the direction this rework is moving toward. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

…env var Extend the manual-input-quant hoisting prototype to compressed_tensors static-per-tensor FP8 (kFp8StaticTensorSym) and wire the Llama decoder layer to use it. Smoke-tested on RedHatAI/Llama-3.2-1B-FP8 with byte-identical output vs. the legacy path. - Gate scheme opt-in on VLLM_HOIST_INPUT_QUANT=1; default off keeps the torch.compile + rms_quant_fusion pass active as today. - CompressedTensorsW8A8Fp8: opt in only when static-tensor activation; quantize_input calls scaled_fp8_quant(layer.input_scale), apply_weights routes QuantizedActivation by passing the FP8 tensor through (kernel already has a "skip quant if x.dtype == fp8" branch). - CompressedTensorsW4A4Fp4: same gate retroactively applied so the NVFP4 path also defaults off. - llama.py LlamaDecoderLayer.forward: call rms_norm_input_quant before qkv_proj and gate_up_proj. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

Add a scheme-level rms_norm_quantize_input that the helper prefers over the chained norm+quantize path. CompressedTensorsW8A8Fp8 implements it by calling torch.ops._C.rms_norm_static_fp8_quant / fused_add_rms_norm_static_fp8_quant directly, so manual-fusion mode hits the same C++ kernel as the compile-fusion path. CompressedTensorsLinearMethod gains a method-level forwarder that delegates to the scheme's fused entry point if present, otherwise falls back to chained norm + scheme.quantize_input. NVFP4 keeps the chained path since no real fused kernel exists yet. Tested on RedHatAI/Llama-3.2-1B-FP8 across all four mode combinations (eager/compile x HOIST=0/1) with byte-identical generated text; verified under fullgraph torch.compile that Dynamo traces into the new path. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

Both manual-fusion and compile paths produce byte-identical output and both reach the same fused C++ kernel, so the env-var gate is just dead weight. Each scheme keeps its own narrow correctness gate (NVFP4: only when the kernel exposes apply_quantized; FP8: only static-per-tensor), so non-supported configurations still fall through to the legacy path. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

…o kernel Strip the NVFP4 + Qwen3 changes to focus the prototype on static FP8. Move the QuantizedActivation-vs-Tensor dispatch from CompressedTensorsW8A8Fp8.apply_weights down into FP8ScaledMMLinearKernel.apply_weights so the scheme and the CompressedTensorsLinearMethod are pure passthroughs; the isinstance check lives in exactly one place. Drop the now-unused CompressedTensorsW8A8Fp8.quantize_input and CompressedTensorsLinearMethod.quantize_input forwarder, and collapse rms_norm_quantize_input on the method to a one-line delegation. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

Make quant_fusion agnostic of the LinearMethod/Scheme abstractions: it reads layer.input_quant_key and dispatches directly to the right fused op (today: kFp8StaticTensorSym → torch.ops._C.{rms_norm,fused_add_rms_norm}_static_fp8_quant). Remove the now-unused rms_norm_quantize_input methods from CompressedTensorsW8A8Fp8 and CompressedTensorsLinearMethod. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

Make the rms_norm_input_quant signature kwarg-explicit so model code documents what's being pulled from the downstream linear instead of handing the whole module to the helper to introspect. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

…all site" This reverts commit 5c84221.

Make rms_norm_input_quant transparent to AllReduce: when residual is not None and tp_size>1 the helper calls tensor_model_parallel_all_reduce on the input before the (potentially fused) add+rms_norm+quant op. The discriminator residual-is-None matches the decoder-layer convention: layer 0's input_layernorm has no residual and no AR; every other site has both. Llama and Qwen3 decoder layers now construct o_proj and the MLP's down_proj with reduce_results=False so the AR happens inside the helper. At tp=1 this is a strict no-op (RowParallelLinear already skipped AR when tp_size=1). Smoke-tested RedHatAI/Llama-3.2-1B-FP8 on tp=1 (output unchanged) and tp=2 (correct, coherent output in both eager and torch.compile). Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

gemini-code-assist

Code Review

This pull request introduces fused RMSNorm and FP8 quantization for Llama and Qwen models, utilizing a new QuantizedActivation dataclass and a rms_norm_input_quant utility to optimize the activation path. However, the review identifies critical issues regarding the hardcoding of reduce_results=False in the model architectures. This change incorrectly moves the All-Reduce operation to the fusion utility without accounting for the final model layer, which will result in partial sums and incorrect logits in multi-GPU configurations. Furthermore, the logic for triggering manual All-Reduce within the fusion utility is flagged as fragile because it does not verify if tensor parallelism is actually enabled for specific layers, potentially leading to incorrect scaling of results.

gemini-code-assist · 2026-05-14T03:59:54Z

+    if residual is not None and get_tensor_model_parallel_world_size() > 1:
+        x = tensor_model_parallel_all_reduce(x)


The logic for triggering All-Reduce based on residual is not None is fragile. While this proxy works for standard Llama/Qwen architectures where the first layer has no residual and subsequent layers follow a RowParallelLinear, it fails to account for cases where disable_tp=True is set on specific layers or the model as a whole. If TP is globally enabled but a layer is not sharded, this will perform an incorrect All-Reduce on a complete tensor, leading to incorrect results (values multiplied by tp_size). Consider passing an explicit flag or checking if the preceding linear layer was actually sharded.

Drop the residual-is-not-None heuristic in favor of inspecting the producer linear directly: rms_norm_input_quant now AR's iff a prev_linear is supplied and that linear has tp_size>1 and reduce_results=False. disable_tp=True (which leaves a linear at tp_size=1) short-circuits cleanly. Route LlamaModel/Qwen2Model's final norm through the same helper, passing the last local layer's down_proj as prev_linear. Plain Qwen2 keeps reduce_results=True so the helper is a no-op for it; Llama and Qwen3 set reduce_results=False so the helper completes the AR before normalizing. This fixes a correctness bug where the final norm was operating on a per-rank partial sum, producing wrong logits at tp>1. Eager and compile outputs now agree on Llama-3.2-1B-FP8 at tp=2. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

When the manual fusion path needs an AllReduce, prefer the FlashInfer single-kernel op (torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm) over the two-kernel chain. Falls back to the chain when FlashInfer isn't loaded or when the input isn't 2D. The op needs a max_token_num for its workspace; capture it on the layer in process_weights_after_loading (where vllm_config is still in context) rather than calling get_current_vllm_config() at forward time. Tested on RedHatAI/Llama-3.2-1B-FP8 tp=2 eager + compile: identical generated text in both modes (workspace log line "Initialized FlashInfer Allreduce norm quantization fusion workspace with backend=trtllm" confirms the fused path fires). Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: mgoin <mgoin64@gmail.com>

quant_fusion now owns a `QUANT_IMPLS` dispatch table (FP8 static per-tensor, NVFP4) and a `FusedAllReduceRMSQuant` module that pre- resolves the impl + max_token_num at construction time. Decoder layers instantiate one per (norm, consumer-linear) pair instead of calling a helper at forward time — keeps the fusion choice on the fused op, not smeared across the linear. FlashInfer wired directly (CUDA-only top-level import; no try/except fallback chain). Linear methods set `layer.input_quant_key` once in `create_weights` — both compressed-tensors W4A4 NVFP4 and ModelOpt NVFP4 — so the fused-op constructor in the decoder layer can resolve its dispatch before forward. NVFP4 path: `_nvfp4` allocates the FP4 quant_out + swizzled scale_out via `create_fp4_output_tensors`, calls `kARResidualRMSNormFP4Quant` with `scale_factor=input_global_scale_inv`, packages into a `QuantizedActivation`. `FlashInferCutlassNvFp4LinearKernel` now unpacks `QuantizedActivation` on the fast path and skips the redundant `scaled_fp4_quant` call. Smoke-tested on B300 (sm_103) with Llama-3.2-1B FP8 and inference-optimization/Llama-3.2-1B-Instruct-NVFP4 at TP=1/2 in eager + compile; FlashInfer fused workspace initializes and the fused single-kernel fires for both FP8 and NVFP4 paths. Signed-off-by: mgoin <mgoin64@gmail.com>

mergify · 2026-05-20T19:39:35Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: mgoin <mgoin64@gmail.com>

Manual fusion (fused_ar_rms_norm_quant) keys off layer.input_quant_key to decide whether the upstream RMSNorm should emit the pre-quantized activation. That key was hard-coded in the W8A8 FP8 scheme and only set on the scheme object (not the layer) for the NVFP4 schemes. Both have problems: - On non-CUDA, the W8A8 scheme would still mark the layer with kFp8StaticTensorSym and dispatch into the CUDA-only flashinfer AR+RMS+FP8 path. - The NVFP4 schemes set self.input_quant_key on the scheme rather than the layer, so getattr(consumer, "input_quant_key", None) at the fusion call site always returned None and the fast path was dead. Move the source of truth to the kernel: - FP8ScaledMMLinearKernel.input_quant_key() defaults to None. CutlassFP8 and FlashInferFP8 override to return kFp8StaticTensorSym when the config is static per-tensor. All other kernels (AITER, ROCm, CPU, XPU, Marlin, PerTensorTorch) inherit None. - Fp8BlockScaledMMLinearKernel.input_quant_key() returns None (block scales are dynamic per-group; nothing to hoist). - NvFp4LinearKernel.input_quant_key() was already in place; FlashInferCutlassNvFp4LinearKernel returns kNvfp4Dynamic. - Schemes assign layer.input_quant_key = kernel.input_quant_key() in create_weights when the result is non-None. Result: non-CUDA platforms never set layer.input_quant_key, so the manual-fusion helper falls through to AR + norm.forward. Signed-off-by: mgoin <mgoin64@gmail.com>

The previous import block assumed FlashInfer was always available on CUDA. ``allreduce_rms_fusion`` only exposes ``ar_fusion_patterns`` / ``flashinfer_trtllm_fused_allreduce_norm`` when ``flashinfer_comm`` is importable, so the module-level import would raise on CUDA boxes without FlashInfer installed. Wrap the import in ``contextlib.suppress(ImportError)``, default both names to None, and gate every reference behind ``is not None``. The fused-quant impls fall through to their non-flashinfer paths (which use vLLM's own C++ rms-norm kernels), and the no-quant impl falls through to ``tensor_model_parallel_all_reduce`` + ``RMSNorm.forward``. This also fixes the latent issue where the no-quant ``_allreduce_rms_norm`` unconditionally called the flashinfer op on non-CUDA platforms whenever ``needs_ar and x.ndim == 2`` held -- relevant now that every TP model goes through this helper (down_proj / o_proj use reduce_results=False). Signed-off-by: mgoin <mgoin64@gmail.com>

…base Two changes that travel together: 1. Drop the engine-config lookup for the flashinfer fused-AR-RMS kernel's ``max_token_num`` arg. The free-function refactor of the helper pushed ``get_current_vllm_config()`` into the call path, which runs inside the torch.compile region during ``_dummy_run`` -- where ``set_current_vllm_config`` is not active -- and raised: AssertionError: Current vLLM config is not set ... model forward time when config is not set. Compute the value at the call site instead, using the same math ``FlashInferAllReduce._ensure_workspace`` uses: per-(world_size, capability) workspace MB cap divided by ``hidden_dim * element_size``. The MB table is resolved once at import time (its lookup pulls in lazy imports dynamo can't trace through), and the per-call sizing is a pure int op. TP=1 didn't trip the original assertion because ``do_allreduce`` was False and the lookup was short-circuited. 2. Pull ``input_quant_key()`` up to ``MMLinearKernel`` (and mirror on ``ScaledMMLinearKernel``) so every linear kernel inherits the default-None contract. Removes the explicit overrides on ``Fp8BlockScaledMMLinearKernel`` and ``FP8ScaledMMLinearKernel`` (still inherited). Signed-off-by: mgoin <mgoin64@gmail.com>

The old per-quant-key impls (_allreduce_rms_norm_fp8_static_tensor, _allreduce_rms_norm_nvfp4, _allreduce_rms_norm) each interleaved a fused-kernel attempt with an inline naive fallback at the tail. The decision tree was buried inside each impl; "what path did we actually take" was hard to read. Classify the available fusions first, then dispatch. Four explicit, ordered paths: A. Single kernel for AR + add + RMSNorm + activation-quant (flashinfer kARResidualRMSNormFP[48]Quant). Returns QA. B. Single kernel for AR + (add +) RMSNorm, no quant (flashinfer kARResidualRMSNorm). Returns plain tensor. C. Explicit AR (if any) + fused (add +) RMSNorm + activation-quant (vLLM C++ rms_norm_static_fp8_quant variants). Returns QA. D. Explicit AR (if any) + plain RMSNorm (vLLM C++ on CUDA, RMSNorm.forward on other platforms). Returns plain tensor. Two small dispatch tables -- _FUSED_AR_RMS_QUANT_IMPLS for path A and _FUSED_RMS_QUANT_IMPLS for path C -- key impls by activation quant key. QuantizedActivation is returned only when a fused kernel actually produced the pre-quantised output (A and C); B and D return plain tensors and let the downstream linear quantise its own input. Adding a ROCm/AITER backend later is a matter of populating the dispatch tables -- the dispatcher itself doesn't need to change. Signed-off-by: mgoin <mgoin64@gmail.com>

The helper's 4th argument was renamed to consumer_linear; update the positional calls in llama / qwen2 / qwen3 to use the keyword form (consumer_linear=...). The final-norm sites passed consumer=None as a keyword and would have raised TypeError under the new signature. Signed-off-by: mgoin <mgoin64@gmail.com>

Signed-off-by: mgoin <mgoin64@gmail.com>

- Path A (flashinfer AR+RMS+quant) now handles residual=None via the zero-residual + throwaway norm_out trick, matching the compile pass's AllReduceFusedRMSNormStaticQuantFP8Pattern. The AR'd input (left in x) becomes the downstream residual. No current model wiring hits this (do_allreduce always coincides with a real residual today), but it removes the asymmetry with the no-quant path and is forward-looking. - Guard fused_ar_rms_norm_quant with `assert type(norm) is RMSNorm`. The fused kernels apply `gamma * x_normed` directly, so a modified norm (e.g. Gemma's `(1 + gamma)`) would silently produce wrong results. - Assert `x.quant_key == self.input_quant_key()` in the FP8 and NVFP4 apply_weights QA branches so a mismatched QuantizedActivation fails loudly instead of corrupting. - Drop the unquantized op-count test case: the no-quant TP=1 path is a plain RMSNorm passthrough that defers to norm.forward, so it codegens natively under the test's IrOpPriority and no custom op fires to count. Signed-off-by: mgoin <mgoin64@gmail.com>

It was using a two-branch form (full kernel call duplicated per residual case) while the quant impls used inline buffer-selection. Behaviorally identical, just restructured to match: one (kernel_residual, norm_out, out_residual) selection + a single kernel call. The no-quant output is norm_out when allocated (no residual) else x, since this path has no quant_out to return. Signed-off-by: mgoin <mgoin64@gmail.com>

The flashinfer fused-AR workspace is hard-capped in MB (a one-shot FlashInfer limit), so it holds at most MB_cap / (hidden * elem) tokens. The compiler pass guards this per compile-range (is_applicable_for_range: range.end <= max_token_num) and leaves oversize ranges unfused. Manual fusion call sites have no such guard, so the profiling run (max_num_batched_tokens) on a wide-hidden model overran the workspace and tripped the assert (e.g. Llama-4-Scout-FP8: 8192 tokens vs a 6553 token capacity at hidden=5120). Replace the assert with a runtime fallback inside the op: when the batch is wider than the workspace, do a standard all-reduce + the matching C++ rms-norm[+quant] kernels, writing the same output buffers the fused kernel would. The check uses the real num_tokens (the op body is eager / opaque to inductor), so it's per-batch -- fused on small decode steps, unfused only on the oversize prefill/profile -- and needs nothing from VllmConfig. The compiler pass is unaffected (its range guard means it never reaches the fallback). all_reduce is out-of-place, so the fallback runs the norm/quant kernels on the freshly all-reduced tensor and only copies back into allreduce_in when the caller reads it (the no-quant norm result, or the unreached no-residual case). The common FP8/NVFP4 with-residual path is copy-free, and NVFP4 writes its outputs in place via scaled_fp4_quant.out. Handles no-quant, FP8-static, and NVFP4 patterns, honoring the norm_out/residual_out aliasing. Signed-off-by: mgoin <mgoin64@gmail.com>

Baking the manual fusion into the shared Llama/Qwen2 base classes broke tensor parallelism for models that reuse them: - LlamaAttention hardcoded o_proj reduce_results=False, expecting the decoder forward to do the all-reduce. Models reusing LlamaAttention with their own (unfused) decoder forward -- mistral, nemotron_nas (DeciLM), arcee, llama_eagle, llama_eagle3 -- silently dropped the attention all-reduce at TP>1. - Qwen2's final norm called fused_ar_rms_norm_quant(do_allreduce=tp>1) while Qwen2DecoderLayer was never converted (reduce_results=True), so the last layer's output was all-reduced twice (confirmed: garbage output at TP=2 on Qwen2.5-0.5B/1.5B). - aria inherited the converted forward but its MoE mlp has no gate_up_proj -> AttributeError. Fix: make fusion opt-in. - LlamaAttention gains a reduce_results param (default True, safe for reusers); the decoder only sets it False when fusing. - LlamaDecoderLayer gains fuse_allreduce (default True) that gates both reduce_results on o_proj/down_proj and the forward path (fused vs the original plain path). The 4 subclasses that don't fully convert (mistral, llama_eagle, llama_eagle3, aria) pass fuse_allreduce=False, restoring their original behavior (also fixes eagle's Identity-norm crash and aria's gate_up_proj crash). - DeciLM/Arcee build attention directly, so the True default fixes them with no per-model change. - Revert the Qwen2 final-norm call to self.norm(...); drop the now-unused import and tp_size. - Defensive getattr(self.mlp, "gate_up_proj", None) in the fused path. Tested TP=1 vs TP=2 (greedy): Qwen2.5-1.5B now matches (was garbage); Llama-3.2-1B-FP8 fused path unchanged; Mistral-7B coherent (AR restored; its code path is now identical to pre-fusion main). Manual-fusion op-count test still passes. Signed-off-by: mgoin <mgoin64@gmail.com>

Signed-off-by: mgoin <mgoin64@gmail.com>

mergify · 2026-05-29T17:17:41Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mgoin and others added 9 commits May 12, 2026 23:16

Revert "[Quant][Prototype] Pass quant_key/input_scale explicitly at c…

1c6157f

…all site" This reverts commit 5c84221.

mgoin mentioned this pull request May 14, 2026

[Quant][Prototype] Expand FP8 manual fusion: AR + more models mgoin/vllm#2

Closed

3 tasks

mergify Bot added llama Related to Llama models qwen Related to Qwen models labels May 14, 2026

gemini-code-assist Bot reviewed May 14, 2026

View reviewed changes

mgoin and others added 2 commits May 14, 2026 16:01

mgoin changed the title ~~[Quant][Prototype] Expand FP8 manual fusion: AR + more models~~ [Quant][Prototype] Manual fusion AR+RMS+Quant May 18, 2026

mergify Bot added the nvidia label May 20, 2026

github-project-automation Bot added this to NVIDIA May 20, 2026

mergify Bot added the needs-rebase label May 20, 2026

Change to functional approach and no quant support

b11c165

Signed-off-by: mgoin <mgoin64@gmail.com>

A1c0r-Z mentioned this pull request May 22, 2026

[Kernel] Add native CUDA fused RoPE + KV cache write op, opt-in for flash_attn #43355

Closed

5 tasks

mgoin added 3 commits May 26, 2026 21:35

Merge branch 'main' into fp8-manual-fusion-expand

8e14d9e

Signed-off-by: mgoin <mgoin64@gmail.com>

mergify Bot removed the needs-rebase label May 27, 2026

mgoin added 3 commits May 27, 2026 17:17

mgoin force-pushed the fp8-manual-fusion-expand branch from 86f5d5b to b0d1410 Compare May 27, 2026 19:53

mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label May 27, 2026

mgoin marked this pull request as ready for review May 27, 2026 20:00

mgoin requested review from pavanimajety, robertgshaw2-redhat, sighingnow, tlrmchlsmth, vadiklyutiy, yewentao256 and zyongye as code owners May 27, 2026 20:00

Fix up

c95561f

Signed-off-by: mgoin <mgoin64@gmail.com>

depthfirst-app Bot reviewed May 28, 2026

View reviewed changes

Comment thread vllm/model_executor/layers/fusion/ar_rms_quant.py Outdated

Comment thread vllm/model_executor/models/qwen2.py Outdated

mgoin added 6 commits May 28, 2026 21:00

Merge branch 'main' into fp8-manual-fusion-expand

39b6bad

Tests

c3cb1c7

Signed-off-by: mgoin <mgoin64@gmail.com>

mgoin requested review from BoyuanFeng, ProExpertProg, patrickvonplaten, youkaichao and zou3519 as code owners May 29, 2026 17:16

mergify Bot added mistral Related to Mistral models speculative-decoding labels May 29, 2026

mergify Bot added the needs-rebase label May 29, 2026

HumphreySun98 mentioned this pull request Jun 12, 2026

[Quant] Port RMSNormQuantFusionPass to manual fusion (fp8 static per-tensor + dynamic per-token) #45364

Open

mgoin mentioned this pull request Jun 16, 2026

[Kernel] Manual TP fusion via ResidualStream (AR+RMSNorm+quant) #45855

Open

mergify Bot added the quantization label Jul 23, 2026

Navigation Menu

Uh oh!

Uh oh!

[Quant][Prototype] Manual fusion AR+RMS+Quant - #42597

[Quant][Prototype] Manual fusion AR+RMS+Quant#42597
mgoin wants to merge 26 commits into
vllm-project:mainfrom
mgoin:fp8-manual-fusion-expand

mgoin commented May 14, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot May 14, 2026

Uh oh!

mergify Bot commented May 20, 2026

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		if residual is not None and get_tensor_model_parallel_world_size() > 1:
		x = tensor_model_parallel_all_reduce(x)

Uh oh!

Uh oh!

Conversation

mgoin commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Wired up

Test plan

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot May 14, 2026

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented May 20, 2026

Uh oh!

Uh oh!

Uh oh!

mergify Bot commented May 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mgoin commented May 14, 2026 •

edited

Loading