[JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked) by DarkSharpness · Pull Request #30924 · sgl-project/sglang

DarkSharpness · 2026-07-12T08:23:32Z

(generated by claude)

Naming: the kernel ships as plain per_token_group_quant (no version suffix) — it is the default; the v2 JIT kernel remains only as the benchmark perf baseline and is marked deprecated.
Rebased onto main after #30784 (RFC Phase 2.5, quantization kernels moved to sglang.kernels.ops.quantization); all touched entry points now live at the new paths.

Stacked on #30838 ([JIT] Refactor dtype traits into DTypeTrait and unify warp reductions). The first 4 commits here belong to that PR — please review only the last commit ([JIT] Add trait-driven per_token_group_quant) until #30838 merges, after which this branch will be rebased onto main.

Motivation

The per-token-group quant family has accumulated many parallel implementations, and the old kernels leave performance on the table:

sgl-kernel AOT: per_token_group_quant_8bit.cu (v1), per_token_group_quant_8bit_v2.cu
JIT: per_token_group_quant_8bit (v1), per_token_group_quant_8bit_v2, dsv4/fp8_wo_a, minimax_quant_ue8m0
Triton: _per_token_group_quant_8bit / _colmajor / _fuse_silu_and_mul, per_token_group_quant_mla_deep_gemm_masked_fp8, _per_token_group_quant_fp8_hopper_moe_mn_major (dead), int8 variants
plus a 5-way runtime dispatcher (_run_per_token_group_quant_8bit_kernel) and the SGLANG_OPT_USE_JIT_PER_TOKEN_GROUP_QUANT env flag

This PR introduces one trait-driven JIT kernel intended to subsume all of them. Every quant property is a compile-time axis of a single QuantTrait kernel body:

axis	values
input dtype	bf16 / fp16
output dtype	fp8_e4m3 / int8
group size	16 / 32 / 64 / 128 / 256 (template axis; v2 runtime-switched, capped at 128)
scale format	fp32 / packed UE8M0 (4-per-int32, aligned + pack-tail-zeroing unaligned)
scale layout	row-major / column-major (TMA-aligned view)
fused activation	none / `silu_and_mul`
schedule	flat 2D / masked EP-MoE 3D (`masked_m`, optional `expected_m` grid hint)

Design notes:

One subwarp (group_size / vec_size lanes) per group; 16-byte (32-byte on Blackwell) vectorized loads/stores; absmax reduced in the packed 16-bit domain via the shared warp::reduce_max from [JIT] Refactor dtype traits into DTypeTrait and unify warp reductions #30838; no cross-warp communication.
The masked schedule grid-strides the token axis up to the device-side masked_m[e], so expected_m is only a launch-size hint — a wrong hint can never drop tokens (pinned by a test).
PDL (programmatic dependent launch) supported on both schedules.
masked_m accepted as int32 or int64 (low-word read) without a second instantiation.

Correctness

test/registered/jit/test_per_token_group_quant.py (92 cases, registered for 1-gpu-large and 4-gpu-b200):

UE8M0 paths (the DeepGEMM / EP-MoE production paths) are bit-exact against a pure-torch reference (the pow-2 multiplier is exact), including the non-4-aligned pack-tail layouts (hidden=768) in both row- and col-major packing.
fp32 / int8 scale paths pin the exactly-reproducible stored scale (amax/max_q) bit-exact and bound the dequant round-trip error (codes are not bit-reproducible under --use_fast_math division).
Masked schedule: written rows bit-exact, padding rows untouched, expected_m under-hint covered.
References are pure torch, not the v1/v2 kernels, so the tests outlive the kernels this replaces.

Performance

Measured on B200 vs the v2 JIT kernel (bench_per_token_group_quant{,_masked}.py, both included):

Flat (hidden=4096, col-major UE8M0, group 128):

num_tokens	v2 (us)	new (us)	v2 (GB/s)	new (GB/s)
1	1.474	1.433	7.8	8.0
32	1.638	1.618	224	227
2048	5.778	4.562	4067	5151

Masked EP-MoE (DeepSeek-V3 shape, 4 GPUs, balanced routing): v2 launches a fixed grid over the padded token axis, the new kernel sizes the grid from expected_m and grid-strides to masked_m[e]:

num_tokens	v2 (us)	new (us)	speedup
1	35.50	2.17	16.4x
128	35.48	2.93	12.1x

(Full sweep across group sizes / layouts / models / imbalance in the two committed benchmarks; the new kernel is uniformly >= v2, typically 29-41% faster on the dense grid at batch sizes that saturate bandwidth.)

Call-site migration (done in this PR, per RFC #29630)

per_token_group_quant is exposed through sglang.kernels.ops.quantization (KernelSpec registered) and is now the only CUDA group-quant path:

_run_per_token_group_quant_8bit_kernel collapses to: MUSA → AOT v2 op, CUDA → the new kernel. JIT v1 (kernel + test + bench) deleted; SGLANG_OPT_USE_JIT_PER_TOKEN_GROUP_QUANT, the old-wheel AOT-v1 fallback and all enable_v2 plumbing removed. the kernel bakes eps / quant ranges in — drifted constants assert loudly.
Whole-row group quant (group_size == hidden: per-channel weight quant, w8a8 dynamic per-token) reroutes to the dedicated sgl_per_token_quant_fp8 kernel.
_varlen_deep_gemm_silu_mul_quant: SGLANG_MASKED_GEMM_FAST_ACT (v2) branch + plain-silu Triton fallback → one per_token_group_quant call with expected_m, scales emitted directly in deep_gemm's layout (packed-int32 col-major UE8M0 / TMA-aligned col-major fp32) so the get_mn_major_* transform short-circuits. it beats the DSV4 JIT activation kernel by 25–60% on balanced loads (B200); the flag is removed.
int8 (sglang_per_token_group_quant_int8) → the new kernel. Codes shift ≤1 on ~0.2% of elements vs the old AOT-v1 path (fast-math division boundary ties); scales bit-exact.
Row-major UE8M0-as-fp32 scales (no callers) now error loudly; fixed a 3D-shape slice bug in create_per_token_group_quant_fp8_output_scale (col-major TMA-aligned fp32).
Perf: fp32-scale quant multiply via __fmul2_rn on SM100+ (16 FMUL → 8 FMUL2, +0.5–1% at large batch); flat load tiling fixed at 32B/lane (Hopper regression fix; Blackwell unchanged).

Verified on B200: kernel suite 92/92; migration smoke 15/15 (col-UE8M0 / fp32 / masked-fused bit-exact vs v2, int8 scales bit-exact vs AOT, whole-row, fp32-input, group 256, row-padded, Hopper-style fp32 col-major masked vs pure-torch). Pre-existing (unchanged by this PR): test_v2_jit_matches_aot fails 66 full-grid cases outside the CI subset.

Later PRs:

Triton _packed_fwd (gemm1_alpha oai-swiglu) → add an activation-kind axis to QuantTrait
delete the v2 JIT kernel (kept as benchmark baseline for now) and dead Triton variants (_hopper_moe_mn_major); fold minimax_quant_ue8m0 (scatter variant stays)

Explicitly out of scope (kept as-is): the DSV4 JIT silu kernels for swiglu_limit / swizzled gran=8 layouts, the DSV4 fp8_wo_a group-major quant, the generic activation.py family, and Triton fallbacks for non-CUDA backends (ROCm/MUSA/XPU).

🤖 Generated with Claude Code

CI States

Latest PR Test (Base): ⏳ Run #29829135569
Latest PR Test (Extra): ❌ Run #29829135438

gemini-code-assist

Code Review

This pull request introduces the per_token_group_quant_v3 JIT kernel, refactors the type trait system (dtype_trait to DTypeTrait), and modernizes warp reduction primitives and host-side runtime checks (introducing CHECK_HOST and CHECK_CUDA). The review feedback highlights critical safety issues in per_token_group_quant_v3.cuh where negative indices (e.g., -1, -2) are passed to tvm::ffi::TensorView::size() and stride(). Since these methods typically accept unsigned size_t parameters, passing negative values can cause implicit casting to large unsigned integers, potentially leading to out-of-bounds memory access or undefined behavior.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Unify the per-token-group quant kernel family behind one QuantTrait template: fp8/int8 output, fp32 or packed-ue8m0 scales, row/col-major scale layouts, optional fused silu_and_mul, and a masked EP-MoE schedule with an expected_m grid hint, all as compile-time axes of a single kernel body. vs the v2 JIT kernel: group size becomes a template axis (16..256, v2 runtime-switched and capped at 128), the masked schedule grid-strides so a wrong expected_m hint can never drop tokens, and both kernels share one subwarp-per-group codepath. 29-41% faster than v2 across the dense and masked benchmark grids on B200. Tests pin ue8m0 paths bit-exact against a pure-torch reference (exact pow-2 multiplier) and pin the fp32/int8 paths via exact stored scales + dequant round-trip, so they outlive the v1/v2 kernels they replace. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ane tiling Deriving the per-lane vector width from the arch's native max vector load (kMaxVecBytes: 16B on Hopper, 32B on Blackwell) made a 128-group span 16 lanes on Hopper vs 8 on Blackwell. The doubled lane count doubles the per-lane fixed cost (work-id div/mod, one extra reduction step, redundant scale math/store), which ncu shows as ~50% more executed instructions and 74% issue-slot occupancy for the same bytes moved -- a 5-14% throughput loss vs the v2 kernel at bandwidth saturation, and a longer latency-bound critical path at small token counts. Fix each lane at 32 bytes via Vec32B: one native 256-bit vector on Blackwell (codegen unchanged), two back-to-back 128-bit vectors elsewhere. Each lane owns a contiguous 32B span so the contiguous out_vec_t store writes exactly the elements the lane loaded. On idle H200, v3 now matches or beats v2 on every benchmark row: ~3.8TB/s at bandwidth saturation for group 128, all low-latency rows ahead, and the group 32/64 leads unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…project#29630) Expose per_token_group_quant_v3 via sglang.kernels.ops.quantization and make it the only CUDA path: - _run_per_token_group_quant_8bit_kernel collapses to two lines: MUSA -> AOT v2 op, CUDA -> v3. The JIT v1 kernel, the SGLANG_OPT_USE_JIT_PER_TOKEN_GROUP_QUANT flag, the old-wheel AOT-v1 fallback and the enable_v2 plumbing are removed; v3 bakes eps/quant ranges in, so drifted constants assert instead of silently differing. - Whole-row group quant (group_size == hidden, the per-channel weight and w8a8 dynamic per-token paths) now routes to the dedicated sgl_per_token_quant_fp8 kernel instead of needing arbitrary group sizes. - _varlen_deep_gemm_silu_mul_quant: the SGLANG_MASKED_GEMM_FAST_ACT (v2) branch and the plain-silu Triton fallback collapse into a v3 call that emits scales directly in deep_gemm's layout (packed-int32 col-major UE8M0 / TMA-aligned col-major fp32), letting the follow-up get_mn_major transform short-circuit and carrying the expected_m grid hint. gemm1_alpha stays on Triton until v3 grows an activation-kind axis; swiglu_limit/swizzle stay on the DSV4 JIT kernel. v3 beats the DSV4 kernel 25-60% on balanced loads (B200). - int8 (sglang_per_token_group_quant_int8) routes to v3; codes shift by <=1 on ~0.2% of elements vs the old AOT-v1 path (fast-math division boundary ties), scales are bit-exact. - create_per_token_group_quant_fp8_output_scale: fix the col-major TMA-aligned fp32 slice for batched (3D masked) shapes. - v3 fp32-scale quant multiplies via __fmul2_rn on SM100+ (16 scalar FMUL -> 8 FMUL2, +0.5-1% at large batch); flat load tiling fixed at 32B/lane (Hopper regression fix, no change on Blackwell). - Row-major UE8M0-as-fp32 scales (no callers) now error loudly. Deleted: JIT v1 kernel + its test/benchmark. The v2 JIT kernel stays as the benchmark baseline; its production call sites are gone. Known pre-existing: test_v2_jit_matches_aot fails 66 full-grid cases (v2 JIT vs AOT mismatch outside the CI subset) before and after this change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge silu_and_mul_masked_post_quant_packed_fwd into silu_and_mul_masked_post_quant_fwd: the output_scale dtype now selects the schedule (int32 -> packed UE8M0 dense flat-work grid, float32 -> row-major grid-stride), mirroring how v3 infers scale layout from the buffer. num_real_tokens/topk become optional kwargs required only by the packed schedule. The deep_gemm gemm1_alpha branch collapses to one call. Verified on B200: both schedules produce bit-identical codes and scales on written rows, scales bit-exact vs a torch oai-swiglu reference, padding untouched. Also spell out in the v2 test note that only the fp32-pow-2 storage flavor of row-major UE8M0 is rejected by the srt entry; the packed int32 row-major layout is supported by v3 and pinned bit-exact in its own suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Rename per_token_group_quant_v3 -> per_token_group_quant everywhere (files, symbols, custom-op and KernelSpec ids, tests); it is the default kernel, not a versioned variant. Mark the v2 JIT kernel deprecated: a module docstring plus benchmark comments state it is kept only as the perf baseline (benchmark impl keys are now jit_v2/current), and no sglang runtime code may call it. Also adapts to the post-rebase tree: sgl-project#30784 (RFC sgl-project#29630 Phase 2.5) moved fp8_kernel/int8_kernel to sglang.kernels.ops.quantization, so the branch-added tests/benchmarks/wrapper now import from the new paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ntry per_token_group_quant only packs UE8M0 scales as int32, so routing the srt wrapper unconditionally to it broke callers that quantize into the fp32 power-of-two storage flavor of row-major UE8M0 (deep_gemm ceil_to_ue8m0 convention): _infer_scale_layout raised "scale_ue8m0=True requires an int32-packed output_s". Caught by jit-kernel-b200-test via test_fp8_wo_a, whose flat reference calls sglang_per_token_group_quant_fp8(scale_ue8m0=True) row-major. Route that flavor (float32 x_s with scale_ue8m0) to the retained JIT v2 baseline, which carries it; everything else stays on per_token_group_quant. Verified on B200: test_fp8_wo_a 5/5 and test_per_token_group_quant 38/38 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016pk3EdEe2mHeP9ZLQF9nio

…-trip (SM90 deepep capture crash) Since sgl-project#30924 the masked plain-silu path emits the down-gemm activation scale already MN-major TMA-aligned. On SM90 (DEEPGEMM_NEED_TMA_ALIGNED_SCALES) the runner rebinds it through deep_gemm's get_mn_major_tma_aligned_tensor, whose short-circuit branch returns a NON-owning alias across the tvm-ffi boundary (entry-time torch::from_blob borrow, re-exported as the result); the rebind then drops the scale's only owner, freeing its storage mid-forward. The next allocation reuses the block and the down gemm converts a dangling pointer, failing decode CUDA-graph capture with "the specified pointer resides on host memory" (base-c-test-deepep-4-gpu-h100, 100% deterministic since sgl-project#30924; B200/H200 take the UE8M0 branch and never rebind). Route both masked-gemm scale rebinds through a keep-owner wrapper: call the transform, and when it returns an alias of the input (same data_ptr while both are alive), keep the owning original instead of the alias. This detects the aliasing outcome rather than mirroring deep_gemm's alignment condition, so no deep_gemm layout logic is duplicated in sglang and wheel-side condition changes cannot silently reintroduce the use-after-free. Row-major scales keep taking the owned copy path unchanged.

…gl-project#30924) A hard 'git revert 8bb0d8d' fails: later PRs (sgl-project#32072/sgl-project#32045/sgl-project#32015) finalized the sglang.jit_kernel -> sglang.kernels namespace migration, so the paths sgl-project#30924 wrote to no longer exist. Instead, short-circuit the runtime dispatcher (_run_per_token_group_quant_8bit_kernel) to always route the CUDA path to the deprecated-but-still-present v2 JIT kernel (per_token_group_quant_8bit_v2), which achieves the same behavioral revert without touching moved files. Suspected root cause on Hopper (H20): - PR sgl-project#30924 author's own note: 'flat load tiling fixed at 32B/lane (Hopper regression fix)' -- meaning the new kernel had a Hopper regression they tried to fix, but 66 non-bit-exact-vs-AOT cases remain outside CI coverage. - Every activation quant on GLM 5.2 NVFP4 goes through this dispatcher; a tiny numerical drift in the new kernel compounds through the MoE and shows up as low MTP draft accept rate. Scope of change: - fp8_kernel.py _run_per_token_group_quant_8bit_kernel: CUDA + group_size in {16,32,64,128} -> v2 kernel (v2 does not support group_size=256, so fall through to the new kernel there). - MUSA path unchanged (already AOT v2). - deep_gemm._varlen_deep_gemm_silu_mul_quant bypasses this dispatcher and calls the new per_token_group_quant() directly; NOT covered by this revert. Only affects users of the EP-MoE fused varlen path (--enable-ep-moe with masked schedule); plain TP MoE decoders (the reproducer in PR sgl-project#32209) are fully covered.

whybeyoung · 2026-07-27T18:03:27Z

This fp8 kernel may cause glm5.2 performance decrease a lot

DarkSharpness · 2026-07-27T18:12:42Z

@whybeyoung on which GPU (and which kernel)? I mainly focused on Blackwell. On Hopper, I only tested once (with no regression)

whybeyoung · 2026-07-27T18:35:20Z

Hopper H20, i'll submit a issue when i wake up....zzzz

whybeyoung · 2026-07-27T22:10:45Z

@whybeyoung on which GPU (and which kernel)? I mainly focused on Blackwell. On Hopper, I only tested once (with no regression)

you can find my log above :
stepinto@4f45d01
i use it to get back and become normal @DarkSharpness

…nel (PR sgl-project#30924) Extends 4f45d01: my earlier fp8_kernel.py revert only covered callers that go through _run_per_token_group_quant_8bit_kernel. The EP-MoE fused-silu-mul-quant default plain-silu path in _varlen_deep_gemm_silu_mul_quant bypasses that dispatcher and calls the unified JIT per_token_group_quant() directly, so it still hit the two PR sgl-project#30924 Hopper bugs (scale-tensor use-after-free from returning non-owning views for already-aligned scales; missing non-finite quant sanitization -- NaNs propagate to fp8 NaN codes instead of saturating). Observed symptom on GLM 5.2 NVFP4 MTP on H20 with --enable-ep-moe: - After 4f45d01, accept rate improved from ~0.20 -> ~0.32 (still low; MTP target is >0.5). Log evidence: DP0-DP7 EP0-EP7 all reporting accept_len 2.13-3.18, accept_rate 0.23-0.44. - The plain-silu masked EP-MoE fused-quant path was still on the buggy new kernel. Fix: allocate output_q and output_s inline using create_per_token_group_quant_fp8_output_scale (same layouts deep_gemm consumes: packed-int32 col-major for UE8M0, TMA-aligned col-major fp32 otherwise -- caller's get_mn_major transform still short-circuits) and call per_token_group_quant_8bit_v2 with fuse_silu_and_mul + masked_m. Notes: - v2 has full feature parity for (fuse_silu_and_mul + masked_m + scale_ue8m0 + col-major output_s), verified in the v2 custom op. - expected_m (a PR sgl-project#30924 grid-bound optimization) is dropped; v2 uses tokens_pad as the grid bound -- numerically identical, marginally slower. - gemm1_alpha and swiglu_limit/swizzle branches are untouched (they already use legacy silu_and_mul_masked_post_quant kernels, not the new JIT unified path). - Upstream long-term fix: PR sgl-project#32188 + PR sgl-project#32296. This is a debug revert until those are rebased in.

…ransform Fixes the H100 deepep CI breakage (TestTBOWithTPAttn NaN logits / "pointer resides on host memory" during CUDA graph capture) exposed by sgl-project#30924. Root cause: sgl-deep-gemm (<= 0.1.4.post1) implements get_mn_major_tma_aligned_tensor behind TVM-FFI. Its input crosses the boundary via a non-owning torch::from_blob view, and the already-aligned fast path returns that view directly, so the result never owns the CUDA storage. Since sgl-project#30924 the masked fused-SiLU quant emits scales already in MN-major/TMA-aligned layout, hitting that fast path for the first time. The caller rebinds the result over its only reference (x = get_mn_major_tma_aligned_tensor(x)), the storage is freed to the allocator, and any later same-stream allocation reuses the block before the down GEMM reads it -- use-after-free. TBO makes the reuse near-certain because the other micro-batch's layer is enqueued (with many small allocations) between the rebind and the GEMM. Fix: in the sglang wrapper, hand back the input tensor itself whenever deep_gemm returns an alias of it, preserving ownership. The real transform path (row-major input) is unchanged. This keeps the unified JIT masked quant kernel on Hopper (its 92-case unit suite passes on H200) instead of reverting to the legacy Triton path. Verified on H200: the production-path repro (JIT masked quant -> transform rebind -> allocator churn -> DeepGEMM masked GEMM) produces 688 NaN rows on main and 0 with this fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…drop __frcp_rn from silu Two independent fixes to the trait-driven quant kernel, both traced back to sgl-project#30924 and both verified on H200 (sm_90a). 1. fp16 + UE8M0 quantized whole groups to +-448. The power-of-two quant multiplier was narrowed to the input dtype so the scaling could be one packed __hmul2. That multiplier reaches kMaxValue / eps = 4.5e12, which bfloat16 holds (it has fp32's exponent range) but float16 does not (max 65504). Any fp16 group whose absmax fell below 448/65504 = 6.8e-3 therefore got an `inf` multiplier, and every element in the group quantized to +-448 -- an all-zero row went through `0 * inf` to NaN, which the sanitizing clamp also turned into +448. Measured 4095/4096 codes wrong, 100% of them +-448, silently. The condition is now derived at compile time from DTypeTrait<T>::kFloatMax and a newly-named kAmaxFloor, keeping the packed half multiply where it is provably safe and scaling in fp32 otherwise. The bf16 production path is codegen-identical: SASS for the bf16/ue8m0/g128 col-major flat kernel is byte-for-byte unchanged at 200 instructions. fp16 + ue8m0 goes 128 -> 160 instructions, the cost of correctness on a path no bf16 model uses. Regression test verified to fail before the fix (3 fp16 cases red, 3 bf16 green) and pass after. 2. details::silu used __frcp_rn where the v2 kernel used a plain divide. Under --use_fast_math `a / b` is MUFU.RCP + FMUL (2 instructions); __frcp_rn is a Newton refinement (MUFU.RCP + 2 FFMA + FADD) behind a branch into an out-of-line special-value fixup (~10 instructions + divergence). silu runs per ELEMENT, not once per group like the quant multiplier, so this dominated the fused path: the bf16 fused masked kernel goes 608 -> 384 SASS instructions and BSSY/CALL 35 -> 2, worth 1.23x geomean (1.13-1.26x) on graph-captured EP-MoE decode shapes. It also restores bit-exactness with the AOT v2 op, which uses the same divide. Non-fused kernels are unaffected (silu is only instantiated for kFuseSiluAndMul). Note sgl-project#32616 made exactly this substitution back for the quant multiplier but missed silu; the multiplier is once per group, so it was the smaller half. Verified: 781 passed across the quantization + moe kernel suites; the current kernel is bitwise equal to the pre-sgl-project#30924 v2 kernel across 26 combinations (fp32/ue8m0/int8 x row/col-major x bf16/fp16 x plain/fused/masked).

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization jit-kernel labels Jul 12, 2026

gemini-code-assist Bot reviewed Jul 12, 2026

View reviewed changes

Comment thread python/sglang/jit_kernel/csrc/gemm/per_token_group_quant.cuh

Comment thread python/sglang/jit_kernel/csrc/gemm/per_token_group_quant.cuh

Comment thread python/sglang/jit_kernel/csrc/gemm/per_token_group_quant.cuh

DarkSharpness force-pushed the per-token-group-quant-v3 branch from 1d4eec4 to e59961c Compare July 13, 2026 08:42

DarkSharpness changed the title ~~[JIT] Trait-driven per_token_group_quant_v3: unify the quant kernel family (flat + masked)~~ [JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked) Jul 13, 2026

claude added 5 commits July 21, 2026 04:14

DarkSharpness force-pushed the per-token-group-quant-v3 branch from e59961c to 3ccaf26 Compare July 21, 2026 11:24

DarkSharpness marked this pull request as ready for review July 21, 2026 11:24

DarkSharpness requested review from Fridge003, HaiShaw, Ying1123, ispobock and merrymercy as code owners July 21, 2026 11:24

DarkSharpness added the run-ci label Jul 21, 2026

DarkSharpness requested a review from ch-wan as a code owner July 21, 2026 11:24

DarkSharpness added the bypass-fastfail label Jul 21, 2026

DarkSharpness requested review from BBuf, Edwardf0t1, HydraQYH, celve and yuan-luo as code owners July 21, 2026 11:24

BBuf approved these changes Jul 21, 2026

View reviewed changes

BBuf merged commit 8bb0d8d into sgl-project:main Jul 22, 2026
156 of 188 checks passed

mmangkad mentioned this pull request Jul 22, 2026

[CI] Fix stale per-token group quant callers #32047

Merged

BBuf mentioned this pull request Jul 22, 2026

[Bugfix] Fix Hopper masked quant scale lifetime #32051

Closed

Jialin mentioned this pull request Jul 22, 2026

[Bugfix] Skip the aligned-scale round-trip that frees the masked-path scale (SM90 deepep capture crash) #32097

Closed

5 tasks

amd-bot mentioned this pull request Jul 22, 2026

[CI Monitor] Daily Report - 2026-07-22 bingxche/sglang-ci-bot#143

Open

ch-wan mentioned this pull request Jul 22, 2026

Revert RuntimeContext config-namespace reads/roles (#31813–#31817) #32100

Merged

This was referenced Jul 23, 2026

[Fix] Two root causes of the H100 deepep TBO CI break: scale-tensor use-after-free + missing non-finite quant sanitization #32188

Merged

[Kernel] Add fused activation + FP8 quantization JIT kernel for Triton MoE path #31712

Open

whybeyoung mentioned this pull request Jul 27, 2026

[Bug] Glm5.2 Decode's Perf decrease a lot #32582

Closed

5 tasks

1e4ves mentioned this pull request Jul 28, 2026

[JIT] Restore the previous division behavior in per-token group quantization #32616

Merged

Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026

[JIT] Trait-driven per_token_group_quant: unify the quant kernel fami…

a191283

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

jinzhenfan pushed a commit to jinzhenfan/sglang that referenced this pull request Jul 29, 2026

[JIT] Trait-driven per_token_group_quant: unify the quant kernel fami…

9e32a11

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

DarkSharpness mentioned this pull request Aug 4, 2026

[JIT] per_token_group_quant: fix the fp16 UE8M0 multiplier overflow, 1.23x on the fused EP-MoE path, unify the scale allocation #33533

Open

5 tasks

jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026

[JIT] Trait-driven per_token_group_quant: unify the quant kernel fami…

5ff100e

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026

[JIT] Trait-driven per_token_group_quant: unify the quant kernel fami…

37730f3

…ly (flat + masked) (sgl-project#30924) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Navigation Menu

Uh oh!

[JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked) - #30924

[JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked)#30924
BBuf merged 6 commits into
sgl-project:mainfrom
DarkSharpness:per-token-group-quant-v3

DarkSharpness commented Jul 12, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whybeyoung commented Jul 27, 2026 •

edited

Loading

Uh oh!

DarkSharpness commented Jul 27, 2026

Uh oh!

whybeyoung commented Jul 27, 2026 •

edited

Loading

Uh oh!

whybeyoung commented Jul 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

DarkSharpness commented Jul 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Correctness

Performance

Call-site migration (done in this PR, per RFC #29630)

CI States

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

whybeyoung commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DarkSharpness commented Jul 27, 2026

Uh oh!

whybeyoung commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

whybeyoung commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DarkSharpness commented Jul 12, 2026 •

edited by github-actions Bot

Loading

whybeyoung commented Jul 27, 2026 •

edited

Loading

whybeyoung commented Jul 27, 2026 •

edited

Loading

whybeyoung commented Jul 27, 2026 •

edited

Loading