Repository navigation
🚧 [WIP][llm][npu][quant] Add W4A4 MXFP4 quantization support for Qwen3.5 MoE on Ascend NPU - #32602
Draft
TallMessiWu wants to merge 75 commits into
Draft
TallMessiWu wants to merge 75 commits into
TallMessiWu wants to merge 75 commits into
Conversation
Add NPUMXFP8FusedMoEMethod (online weight quant via npu_dynamic_mx_quant with 3D input) + npu_fused_experts_mxfp8 kernel (routing -> act quant -> swiglu_quant_v2 -> grouped_matmul -> finalize_routing, mirroring vllm-ascend A5 path). Wire dispatch in Fp8Config.get_quant_method for FusedMoE layers (is_npu + use_mxfp8). Weights kept as strided transpose views — kernel contract verified by A5 probe.
ModelSlimMXFP8MoEScheme for --quantization modelslim with W8A8_MXFP8 pre-quantised checkpoint. NPUMXFP8FusedMoEMethod now detects fp8 dtype and takes offline transpose-only branch. Registered via FusedMoeWeightScaleSupported.BLOCK.
Offline modelslim (W8A8_MXFP8) checkpoints quantise mlp.gate to float8_e4m3fn + weight_scale, but Qwen3MoeSparseMoeBlock hard-coded the gate as ReplicatedLinear(quant_config=None) -> bf16. The fp8 gate was then cast to bf16 WITHOUT its block scale and the scale key dropped, scrambling the router (probe: norm 70195 vs 31.4) -> wrong experts -> garbage. Online (bf16 gate) was unaffected. Pass quant_config to the gate only when it is the offline ModelSlim path (get_name()=='modelslim'); ModelSlimConfig.get_quant_method then dequantises a quantised gate correctly or skips a FLOAT/absent one. The online Fp8/mxfp8 path keeps gate_quant_config=None -> bf16, unchanged. Mirrors vllm-ascend's description-driven dispatch.
ascend_npu_quantization.mdx: mark MXFP8 MoE supported on A5, add an MXFP8-for-LLM-MoE section (online + offline launch commands, kernel path, and the router-gate description-driven caveat). quantization.mdx: extend the mxfp8 Ascend cell to cover LLM MoE (FusedMoE/TP). Mirrors the doc pattern from PR sgl-project#23795.
Add an opt-in fused path to npu_fused_experts_mxfp8: when SGLANG_ENABLE_NPU_FUSED_MOE_ROUTING_QUANT is on and the op supports it, npu_moe_init_routing_v2(quant_mode=3) does the MXFP8 activation quant inline (e4m3 payload + e8m0 block scale as ret[3]), replacing the separate npu_dynamic_mx_quant pass and saving one kernel launch. Gated by SGLANG_ENABLE_NPU_FUSED_MOE_ROUTING_QUANT (default off) plus a cached runtime capability probe that falls back to the two-step path when quant_mode=3 is unsupported. A5 probe (Q9, torch_npu 2.10.0.post2): quant_mode=3 accepted with x_dtype=None; ret[3] scale is 2D [N,K/32] e8m0 so _normalize_mxfp_scale reshapes it to [N,K/64,2]; fused output is byte-identical to the two-step path (cos=1.0). DeepEP path (apply_without_routing_weights) does not go through init_routing and is unchanged.
A5 online+offline e2e verified, so flip SGLANG_ENABLE_NPU_FUSED_MOE_ROUTING_QUANT default from False to True. Behavior change: the MXFP8 MoE TP path now fuses activation quant into npu_moe_init_routing_v2 (quant_mode=3) by default; the runtime capability probe still falls back to the two-step path if unsupported, and SGLANG_ENABLE_NPU_FUSED_MOE_ROUTING_QUANT=0 forces the two-step path.
A5 e2e validated the fused quant_mode=3 path (npu_fused_experts_mxfp8), so keep it as the only path -- remove the two-step (quant_mode=-1 + npu_dynamic_mx_quant) fallback branch, the _supports_fused_mxfp8_routing capability probe, and the SGLANG_ENABLE_NPU_FUSED_MOE_ROUTING_QUANT env switch.
Port MXFP8 MoE onto the refactored Ascend MoE stack. Upstream sgl-project#25663 (8ed82af) deleted fused_moe_method_npu.py -- the file this branch extended -- and split the monolithic NPU MoE path into four layers: routing/act-quant in AscendTPDispatcher, orchestration in AscendRunnerCore, per-gmm kernels in quantization/moe_methods.py, and weight creation in the schemes. NPUMXFP8FusedMoEMethod and npu_fused_experts_mxfp8 had no place in that shape, so the conflict is resolved by accepting the deletion and porting the implementation rather than restoring the file. - moe/init_routing.py: NPUMoEInitRouting_v2 already threads quant_mode, so quant_mode=3 needs only the e8m0 scale reshape ([N,K/32] -> [N,K/64,2]), which _normalize_mxfp_scale now does there. - moe/utils.py + token_dispatcher/ascend_tp.py: new DispatcherOutputDtype.MXFP8 selects NPUMoEInitRouting_v2(quant_mode=3); the selector raises on unknown dtypes, so a missing branch fails loudly. - quantization/moe_methods.py: NPUMXFP8MoEMethod is the per-gmm kernel (online and offline branch by weight dtype, strided transpose views preserved); NPUMXFP8FusedMoEMethod is the online --quantization mxfp8 entry point. - moe_runner/ascend.py: gmm1 uses npu_grouped_matmul_swiglu_quant_v2, which fuses gate/up + swiglu + requant and does not decompose into the runner's gmm1 -> activation -> gmm2 split. Keep the fusion via a dedicated branch; the activation is None for MXFP8 and DeepEP is rejected explicitly (the activation quant comes from init_routing, which DeepEP bypasses). - modelslim_mxfp8_moe.py: rewritten per weight group to match the new scheme contract; drops apply_weights/create_moe_runner (forward is the runner's now). - The DeepEP branch (apply_without_routing_weights) is dropped: its upstream counterpart is gone, and MoE was documented TP-only and never validated. Reuses upstream's _get_float8_e8m0fnu_dtype from linear_method_npu behind a non-null assert, since a None scale dtype would silently change how the MXFP8 ops read the scales.
Port regression from the main merge (e983b8f). The pre-merge implementation in fused_moe_method_npu.py resolved the block-scale dtype as getattr(torch_npu, "float8_e8m0fnu", getattr(torch, ...)), torch_npu first. The port to moe_methods.py routed it through the dense helper _get_float8_e8m0fnu_dtype, which only reads torch. npu_grouped_matmul_swiglu_quant_v2 / npu_grouped_matmul validate their scale dtype arguments against torch_npu's own enum (float8_e8m0fnu == 293 on A5) and reject torch's dtype object: "weight_scale_dtype only supports float8_e8m0fnu or None, but the actual value is Float8_e8m0fnu" -- so every MXFP8 MoE forward dies on the first gmm1, online and offline alike. Restore torch_npu-first resolution with a lazy import and a module-level cache, keeping torch as fallback and the loud raise when neither exposes it. The dense helper stays as is: npu_quant_matmul accepts either source.
--quantization mxfp8 makes ServerArgs override moe_runner_backend from 'auto' to
'flashinfer_trtllm' unconditionally, including on NPU where flashinfer does not
exist. NPUMXFP8FusedMoEMethod.create_moe_runner only substitutes ASCEND when the
backend is still 'auto', so it built a flashinfer runner and every decode died
with 'Unexpected quant_info type for flashinfer_trtllm: AscendQuantInfo' during
cuda-graph capture.
The value is not confined to runner selection. FusedMoE keys two more behaviours
off flashinfer_trtllm: the w1/w3 shard swap ('flashinfer assumes w31 format for
w13_weight', layer.py:939) for a listed set of methods, and a 128-alignment
round-up of intermediate_size_per_partition. The current method subclasses
FusedMoEMethodBase so it escapes the shard-swap list today, but it is one
inheritance change away from loading every expert with gate and up exchanged --
silent degenerate output rather than a loud TypeError.
Skip the override on NPU so the backend stays 'auto' and resolves to ASCEND.
ModelSlimMoEMethod.apply builds AscendQuantInfo by reading
layer.{w13,w2}_weight_offset unconditionally, which the int8 and int4 MoE
schemes all register. MXFP8 is a pure scale format -- the e8m0 block exponent
carries everything and there is no zero point -- so the offline scheme never
created one and every decode raised AttributeError: 'FusedMoE' object has no
attribute 'w13_weight_offset' during cuda-graph capture.
Before the main merge the scheme delegated apply to the fused method directly
and never went through the shared builder, so the assumption held.
AscendQuantInfo already types the field Optional, so register it as None -- the
attribute then exists and resolves to 'no offset'. Follows the register_parameter
(name, None) convention used for bias in linear.py and input_scale in fp8.py; a
None parameter is skipped by named_parameters(), so no weight loader looks for
it.
Review feedback on sgl-project#30768: --quantization modelslim does not need to be passed for a msmodelslim checkpoint. Confirmed in the loader. ModelConfig._find_quant_modelslim_config() reads quant_model_description.json and injects quant_method='modelslim', and _verify_quantization() adopts it whenever --quantization was not given -- the comment there calls the field 'required for flagless model loading'. Drop the flag from the offline MoE example and say the description file selects both the ModelSlim path and the scheme. This was the only --quantization modelslim left in the docs.
…type Review feedback on sgl-project#30768: make the hidden-states quant a reusable kernel wrapper like the matmuls, so other quantization schemes can pick it up. Rename npu/moe/hidden_states_quant.py to quant.py (moe_methods.py was its only importer) and route HiddenStatesDynamicQuant by dtype: float8_e4m3fn goes to npu_dynamic_mx_quant, int8/quint4x2 keep npu_dynamic_quant, anything else raises. The two existing call sites use quint4x2 and int8, so behaviour is unchanged; the mxfp8 branch has no consumer yet and is wired up in a later commit. Written fresh rather than cherry-picked from sgl-project#29762: that version does a module-level import torch_npu, which breaks the CUDA/CPU/AMD/XPU CI jobs since the module is imported on every platform. Going through torch.ops.npu.* keeps it importable everywhere, and torch.float8_e4m3fn is a plain torch dtype so no torch_npu reference is needed here at all.
…y prefix Review feedback on sgl-project#30768, two items. Move npu_grouped_matmul_swiglu_quant_v2 out of NPUMXFP8MoEMethod into a GroupedMatmulSwigluQuant wrapper in npu/moe/matmul.py, next to GroupedMatmul, so other block-scaled schemes can reuse it. Two deviations from GroupedMatmul worth knowing: it returns (quantized_activations, block_scale) rather than a single tensor, because the requant is part of the kernel, and it takes no output_dtype (that comes from quant_dtype in scale_args). The count -> cumulative group_list conversion moves inside the wrapper, since this op wants cumulative while npu_grouped_matmul keeps the count form the dispatcher produces. NPUMXFP8MoEMethod.__init__ now takes weight_prefix and picks the matmul there -- GroupedMatmulSwigluQuant for w13, GroupedMatmul for w2 -- instead of gmm1 reaching for the op directly. The instances were already per-prefix (layer.w13_kernel / layer.w2_kernel, one ModelSlimMXFP8MoEScheme per weight group), so only the two construction sites had to pass the prefix. apply() and process_weights_after_loading() keep their weight_prefix parameter, since the runner calls every kernel through that same interface. The resulting op call is argument-for-argument identical to the previous one.
Review feedback on sgl-project#30768: the hard rejection at runner construction was wrong. DeepEP works with mxfp MoE even without quantization inside the dispatcher -- it simply returns bf16, and the method can quantize the activations itself. Three pieces: - AscendRunnerCore no longer raises for DeepEP + MXFP8. activation stays None either way, since gmm1 fuses gate/up + swiglu + requant. - process_weights_after_loading sets the dispatcher output dtype per backend. DispatcherOutputDtype has no MXFP8 entry in deepep.py's config_map (only BF16/FP8/INT8/NVFP4), so asking for 'mxfp8' there would KeyError -- hence an explicit bf16 branch rather than letting it through. - apply_fused_gmm1_swiglu takes pertoken_scale as Optional and, when it is None, quantizes the hidden states through the HiddenStatesDynamicQuant wrapper from the previous commit. Both dispatchers therefore reach the fused kernel with the same e4m3 + e8m0 input; only where the activation quant happens differs. ascend_tp is unaffected: is_deepep() is False there, so the dispatcher dtype stays 'mxfp8' and pertoken_scale is never None. Docs updated to match. Only the ascend_tp path is validated end-to-end on A5 -- I have no EP setup, so in particular it is unverified whether low-latency dispatch's 3D hidden states are accepted by npu_grouped_matmul_swiglu_quant_v2.
…oint Review feedback on sgl-project#30768: the online entry point duplicated create_weights, process_weights_after_loading, create_moe_runner and apply from the unquantized method, and the name did not distinguish it from the per-gmm kernel. NPUMXFP8FusedMoEMethod becomes NPUMXFP8OnlineMoEMethod(UnquantizedFusedMoEMethod) and overrides only create_moe_runner, to attach the MXFP8 kernels before MoeRunner is built (AscendRunnerCore.__init__ reads layer.w2_kernel to pick its activation). That removes ~85 lines. Verified the inherited pieces against the current unquant.py rather than assuming: create_weights produces the same [E, 2I, H] / [E, H, I] parameters given is_gated and no bias; the aiter / cpu-amx / deep_gemm / flashinfer branches of process_weights_after_loading are all False on NPU, leaving the _is_npu branch that calls the kernels; apply -> forward_npu -> runner.run(dispatch_output, layer) works because layer carries the same w{13,2}_weight* attribute names AscendQuantInfo did. It has to live in a new module: unquant.py imports moe_methods.py at module scope, so subclassing there would be a circular import. moe_methods.py does not import unquant.py, so the new chain stays acyclic. Two supporting changes: - unquant.py no longer hardcodes dispatcher_output_dtype 'bf16' right after calling the kernels. It duplicated what NPUUnquantMoEMethod already does for w13, and it clobbered the value a quantized subclass kernel had just set. There are no other subclasses, so unquant is unaffected. - create_moe_runner rejects a non-Ascend runner backend loudly. Inheriting UnquantizedFusedMoEMethod puts this method in FusedMoE's w1/w3 shard-swap list (layer.py:942), so an explicit --moe-runner-backend flashinfer_trtllm would load every expert with gate and up exchanged -- silent degenerate output. The automatic override was already fixed in 9ac21f5; this covers the manual one. The other two isinstance sites (layer.py:365 and the weight-update hook at :904) are both gated on is_flashinfer_trtllm_routed() and inert on NPU.
Answers the NZ question raised in review of PR sgl-project#30768. On A5 (torch_npu 2.10.0.post2, Ascend950PR, Qwen3-30B-A3B shapes with 128 experts) this gains 1.4% on decode and 3.8% on prefill against a 0.2-0.3% noise floor, with outputs identical to before. Nowhere near the ~10% the int8 path reported, but consistent across both token shapes. The cast must precede the transpose. gmm1 asserts that weight and weight_scale share a transpose flag (CheckMXTranspose), and the cast returns a physically retiled, non-transposed tensor -- so casting the already-transposed view fails outright. That is why this cannot follow the int8 MoE methods in the same file, which transpose first but carry no MX scale to keep in sync; the dense W4A8 path in linear_method_npu.py already uses this order. Measurements also rule out .contiguous() on either side: making both contiguous costs 6.2% on decode. _is_nz_aligned gains an fp8 branch sharing the int8 column rule, since e4m3 is likewise single-byte. It previously fell through to the catch-all True, which no caller exercised -- the packed-FP4 callers return early via customize_dtype and dense MXFP8 never format-casts -- so this only guards the weights introduced here.
# Conflicts: # python/sglang/srt/layers/quantization/modelslim/schemes/__init__.py
npu_gemma_rms_norm has no kernel registered for socVersion ascend950 (A5), causing graph capture to fail with "does not support opType [GemmaRmsNorm]". Gemma RMSNorm equals normalize(x) * (1 + weight), so a plain npu_rms_norm with gamma = 1 + weight is mathematically equivalent and is implemented across A2/A3/A5.
Only Ascend A5 (soc_version 260 / ascend950) lacks the npu_gemma_rms_norm kernel; A2/A3 (910) support it. Branch on is_npu_a5() so A5 uses the equivalent npu_rms_norm(x, 1 + weight) fallback while A2/A3 keep the native fused op, mirroring vllm-ascend's per-device dispatch.
Add online and ModelSlim offline paths, reuse generic FusedMoE plumbing, and cover scheme/layout contracts with CPU tests.
Pass BF16 [1, N] bias to A8W4 matmul for online and offline paths while preserving bias-free Qwen3 behavior.
10 tasks
Upstream sgl-project#30318 landed the offline ModelSlim W4A8_MXFP MoE path while this branch was open, so the branch's parallel implementation is dropped and only the online entry point is kept: - Delete ModelSlimMXFP4W4A8MoEScheme and NPUMXFP4W4A8MoEMethod; the registry entry for W4A8_MXFP was already shadowed by upstream's ModelSlimW4A8MXFP4MoE. - Teach upstream's NPUW4A8MXFP4MoEMethod to quantize BF16 experts at load time, keyed on weight dtype, so offline and online share one re-layout. - Rename the online FusedMoE entry to NPUW4A8MXFP4OnlineMoEMethod, matching the MXFP8 one next to it. - Drop NPUSwigluMXFP8Quant: upstream dispatches BF16 and quantizes per token in apply(), so no separate swiglu requant step is needed. - Replace the is_npu_a5() Gemma RMSNorm branch with the sgl-kernel-npu provider import, keeping provider selection at wheel-build time per RFC sgl-project#35709. - Keep the linear-layer bias/scale-padding fixes and the qwen3_vl ModelSlim quant_config propagation, rebased on main's language_model_only gate.
Upstream sgl-project#30319 landed the offline ModelSlim W4A4_MXFP4 MoE path, so this branch's parallel implementation follows W4A8 and keeps only the online entry: - Delete ModelSlimMXFP4W4A4MoEScheme and NPUMXFP4W4A4MoEMethod; upstream's ModelSlimW4A4MXFP4MoE already owned the W4A4_MXFP4 registry entry. - Teach upstream's NPUW4A4MXFP4MoEMethod to quantize BF16 experts at load time, and lift the shared online-quantization helper onto _NPUMoEMethodBase now that W4A8 and W4A4 both use it. - Rename the online FusedMoE entry to NPUW4A4MXFP4OnlineMoEMethod. - Revert the AscendRunnerCore W4A4 branches: upstream dispatches BF16 and quantizes per token in apply(), so gmm1 is not fused with swiglu here. - Resolve the quant op in HiddenStatesDynamicQuant on first use. Binding torch.ops.npu in __init__ made every MoE method that holds a quantizer un-constructible on CPU, which blocks the scheme tests in this PR. - Drop the docs: upstream's Ascend quantization page already carries the MXFP8 MoE section this branch was still adding under the old docs_new/ tree.
HiddenStatesDynamicQuant bound torch.ops.npu.* in __init__, so constructing any MoE method that holds a quantizer -- including this PR's NPUW4A8MXFP4MoEMethod, and through it ModelSlimW4A8MXFP4MoE -- raised AttributeError off-device and made the scheme untestable on CPU CI. Keep the dtype validation eager and look the op up on first call, where a missing torch_npu still fails loudly.
The branch tip carried an accuracy experiment (6bcf2af) that routed every LinearBase through UnquantizedLinearMethod so only the experts were quantized. Against main that reads as a regression: main's --quantization mxfp_w4a8 does quantize Linear, and the PR would have taken that away. Restore the Linear branch, so the flag is now purely additive over main: - LinearBase keeps NPUMXFP4W4A8LinearMethod, with the K % 32 == 0 guard that falls unaligned layers (Qwen3.5 vision linear_fc2, K=4304) back to BF16 instead of crashing the FP4 npu_quant_matmul. - FusedMoE experts now get NPUW4A8MXFP4OnlineMoEMethod instead of main's warn-and-fall-back-to-BF16. To isolate the experts again for an accuracy run, set --quantization mxfp_w4a8 with the non-expert layers in ignored_layers rather than editing the dispatch.
The unfused W4A4 route spends four kernels per MoE layer -- grouped matmul to BF16, npu_swiglu, npu_dynamic_mx_quant back to fp4, grouped matmul -- and two BF16 round trips through HBM ([T, 2*inter] and [T, inter]) that only exist to carry gmm1's output to the requant. npu_grouped_matmul_swiglu_quant_v2 does all three in one kernel and hands the w2 gmm the fp4 activations plus their e8m0 block scale directly. MXFP8 already takes this route through apply_fused_gmm1_swiglu; W4A4 now joins it: - NPUW4A4MXFP4MoEMethod holds a GroupedMatmulSwigluQuant next to its plain GroupedMatmul, so one instance still serves both weight groups and the ModelSlim scheme keeps its no-argument constructor. - AscendRunnerCore resolves the choice once into fuses_gmm1_swiglu. DeepEP keeps the unfused route: its grouped-row dispatch has not been validated against the fused fp4 kernel, and unlike the previous TP-only revision of this branch it stays supported rather than raising. - apply() re-pairs a per-token scale only when it arrives flat. The fused kernel already returns the paired form, which the MXFP8 w2 gmm likewise forwards untouched; re-pairing it would corrupt gmm2's scales. Needs an A5 run to confirm the gmm2 scale layout end to end -- the non-None pertoken_scale branch in apply() was unreachable before this change.
Contributor
|
Hi! We are really interested in this functionality. Are you planning on opening up this PR? |
replay_with_input_update rebound the captured NPU graph's seq_lens from a
background thread while the main thread issued graph.replay(), joining
the thread only after the replay had been sent. Nothing ordered the
rebind before the execution it was meant to feed: the replay reaches the
driver's BindSqCq while the rebind is still in flight, the driver refuses
the inconsistent state ("Stream not inited or stream_mem not match" in
plog), and rtModelExecute fails on a decode that succeeded on the
previous run.
Rebind on the calling thread, then replay.
Reproduced on Ascend with Qwen3.5-27B: serving crashed intermittently on
the first decode graph replay with rtModelExecute retCode 0x7020023,
surfaced as Insufficient_Resources(EL0006) although the failure is not
about memory. It hit both BF16 and ModelSlim W8A8 MXFP8, on different
devices, with no other process on the card. Repeated runs with the
rebind serialized no longer reproduce it, and restoring the concurrent
rebind brings the failure straight back. The same stack has been
reported elsewhere with retCode 0x7020004 / Invalid_Argument(EL0003).
The lint gate rejects new files under test/registered/kernels/ unless they register a *-kernel-* suite, and no such suite runs on CPU: the workflows define base-b-kernel-unit-test-* on GPU runners only. This test mocks sgl_kernel_npu and asserts registry dispatch and error messages, so it is a CPU unit test rather than a kernel test. The rejection failed lint, which gates pr-gate, so every test job in the matrix was skipped -- including base-a-test-cpu, where this test runs. Path and suite now agree: test/registered/unit/npu/, next to the other NPU CPU unit tests, keeping register_cpu_ci(suite="base-a-test-cpu"). All 11 tests still pass.
Drop the graph-rebind fix: upstream sgl-project#39589 fixes the same race, and more completely. Both versions order the input rebind before the replay -- ours by doing it on the calling thread, upstream's by blocking on the future -- but upstream reuses one device-bound worker whose executor initializer calls set_device, instead of creating a thread per replay. Take upstream's file whole; nothing of ours is left to carry. The other ten files merged without conflicts.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extend Ascend
--quantization mxfp4to FusedMoE throughNPUW4A4MXFP4OnlineMoEMethodandNPUW4A4MXFP4MoEMethod. Experts use single-level packed MXFP4 weights and FP4 activations. Online Dense layers keep dual-level W4A4; ignored or group-unaligned Dense layers stay BF16. Single-level and dual-level scale contracts remain separate.Dependencies and review scope
Updated on 2026-10-10 with ordinary merges; base remains
main. Branch head:425b0a9abb. Includes official mainc55790d38b.Merge order: sgl-project/sgl-kernel-npu#638 (Gemma API/wheel) → #32745 → #32601 → #32602. This branch contains updated #32601 at
ded4671577, including #32745 atcf48b477db. #32266 stays closed. #36426 remains a separate validation dependency.Review the W4A4 increment against the refreshed #32601 branch.
Execution contracts
swiglu_limitbecause its activation paths do not implement that clamp.Current validation
Applicable pre-commit hooks passed. Isolated CPU checks passed W4A4 online/offline post-load layout, BF16 dispatch, fused-vs-unfused and clamp guards, and Dense L0 scale shapes (2D/3D/singleton). The inherited W4A8 method matches #32601 exactly and its regression checks pass. Shared MXFP8 scale checks pass.
These are CPU/static checks, not 910/950 operator, model-load, warmup, accuracy, or performance validation. Full SRT test collection was unavailable in this environment because its serving dependencies are not installed. Target hardware validation remains required.
Remains Draft. Current PR-test gates reject Draft PRs; no Ready conversion is part of this refresh.
Earlier benchmark results (not rerun on the refreshed head)
Performance
Note: Each configuration was measured once after one warm-up run. All runs completed 500 requests with a maximum request concurrency of 32 and identical input/generated token counts. Throughput improvement =
(quantized / bf16 - 1) × 100%; latency reduction =(1 - quantized / bf16) × 100%. Positive values indicate improvement.Accuracy
Note: Single-run GSM8K exact match. Δ = quantized − baseline.
CI States
Latest PR Test (Base): ❌ Run #38040905028
Latest PR Test (Extra): ❌ Run #38040904898
Latest PR Test (AMD ROCm 10): ❌ Run #38040905114