[Kernel] Migrate scattered quantization kernels to sglang.kernels (RFC #29630, Phase 2.5, 1/7) by BBuf · Pull Request #30784 · sgl-project/sglang

BBuf · 2026-07-10T14:10:24Z

Motivation

Phase 2.5 of RFC #29630 (see the updated migration plan in the issue): a full-tree audit after #30044 found ~280 Triton kernels plus CuTe DSL / TileLang kernels still scattered outside the three canonical locations. This is sweep 1 of 7: the quantization group.

Modifications

Byte-identical file moves + import rewrites only (git diff -M shows 100% renames):

From (srt/layers/quantization)	To	Kernels
`fp8_kernel.py`	`sglang/kernels/ops/quantization/fp8_kernel.py`	12 Triton
`int8_kernel.py`	`sglang/kernels/ops/quantization/int8_kernel.py`	3 Triton
`awq/awq_triton.py`	`sglang/kernels/ops/quantization/awq_triton.py`	2 Triton
`mxfp8_amd_gfx95.py`	`sglang/kernels/ops/quantization/mxfp8_amd_gfx95.py`	2 Triton
`nvfp4_gemm_swiglu_nvfp4_quant.py`	`sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`	CuTe DSL (SM100)
`configs/` (159 w8a8 tuned JSONs)	`sglang/kernels/ops/quantization/configs/`	used only by fp8/int8_kernel

mxfp4_flashinfer_trtllm_moe.py is a quant-method module with one embedded kernel: PackTopkIds + its Triton kernel are extracted verbatim into sglang/kernels/ops/moe/pack_topk_ids.py (MoE-routing functionality, also consumed by moe_runner/flashinfer_trtllm.py); the quant-method glue stays in srt.

Also:

Registered all migrated public entry points as KernelSpec inventory (TRITON / CUTE_DSL backends).
Rewrote all in-tree importers (82 files across python/, test/, benchmark/).
The only content change inside a moved file is fp8_kernel's lazy sibling import of int8_kernel.
Extended test_kernels_namespace.py expected-ops set.

Verification

import sglang.kernels stays metadata-only (import-purity test passes: no sgl_kernel / sglang.jit_kernel import).
Registry check: 24 quantization/moe ops registered with correct backends.
test_kernels_namespace.py + test_fused_op.py: 33 passed.
Existing quant correctness tests (test/registered/quant/*) now import through the new path and run unchanged in their CI lanes.

Checklist

Part of the RFC #29630 Phase 2.5 series (1/7). Next: moe, top-level layers strays, generic attention, dsa+dsv4, linear-attention family, vendored fla/mamba.

🤖 Generated with Claude Code

Kernel-migration verification (no perf / config regression)

All moved modules are byte-identical relocations (git rename similarity R100, except fp8_kernel.py at R099 whose only delta is an int8_kernel import-path rewrite — no kernel logic changed), so per-kernel latency is unchanged by construction.

Import fixes in this PR: awq/__init__ and fp8_utils still imported awq_triton / fp8_kernel / mxfp8_amd_gfx95 from pre-migration paths → repointed to sglang.kernels.ops.quantization.*.

moved kernel	rename	note
awq_triton	R100	identical
fp8_kernel	R099	only int8_kernel import path changed
int8_kernel	R100	identical
mxfp8_amd_gfx95	R100	identical
nvfp4_gemm_swiglu_nvfp4_quant	R100	identical

Tuning configs: 158 fp8/int8 block-shape config JSONs moved together with get_w8a8_block_fp8_configs; the loader resolves them via os.path.dirname(__file__)/configs, so co-locating loader + configs preserves the lookup.

Empirical A/B (H200, w8a8_block_fp8_matmul, N=1536 K=7168, block=[128,128], fp16 out): both main and PR load the tuned config (CONFIG_HIT=True, 18 entries) and latency matches within noise:

M	main µs	PR µs	Δ
64	41.85	42.04	+0.5%
512	40.29	40.59	+0.7%
4096	118.71	118.59	−0.1%

→ tuning config still hit at the new location, no latency regression.

CI States

Latest PR Test (Base): ⏳ Run #29226631377
Latest PR Test (Extra): ❌ Run #29226631233

…sgl-project#29630, Phase 2.5, 1/7) First of the seven Phase 2.5 sweeps: move the Triton / CuTe DSL quantization kernels still living under srt/layers/quantization into sglang.kernels.ops.quantization, byte-identical except import rewrites. - fp8_kernel.py (12 Triton kernels), int8_kernel.py, awq/awq_triton.py, mxfp8_amd_gfx95.py, and the CuTe DSL nvfp4_gemm_swiglu_nvfp4_quant.py move wholesale; the w8a8 tuned-config JSON directory (used only by fp8_kernel/int8_kernel) moves with them. - mxfp4_flashinfer_trtllm_moe.py is a quant-method module with one embedded kernel: extract PackTopkIds + its Triton kernel into sglang.kernels.ops.moe.pack_topk_ids (it is MoE-routing functionality, also consumed by moe_runner/flashinfer_trtllm); the quant-method glue stays in srt. - Register the migrated public entry points as KernelSpec inventory (TRITON / CUTE_DSL backends) and extend the namespace test's expected set. - Rewrite all in-tree importers (82 files across python/, test/, benchmark/). Only content change inside moved files is fp8_kernel's lazy sibling import of int8_kernel following the move. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

awq/__init__ and fp8_utils still imported awq_triton / fp8_kernel / mxfp8_amd_gfx95 from their pre-migration paths; repoint to sglang.kernels.ops.quantization.*. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ization

BBuf · 2026-07-12T14:09:05Z

/rerun-failed-ci

BBuf · 2026-07-13T01:40:49Z

/rerun-failed-ci

BBuf · 2026-07-13T02:04:30Z

https://github.com/sgl-project/sglang/actions/runs/29175506877/job/86717337482?pr=30784

tuning_block_wise_kernel.py still defaulted --save-path to the pre-migration python/sglang/srt/layers/quantization/configs (now empty; the 159 tuned JSONs and the get_w8a8_block_{fp8,int8}_configs loader moved to python/sglang/kernels/ops/quantization/configs). Repoint the default so tuned configs land where the runtime loader reads them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ization

BBuf · 2026-07-13T08:16:55Z

https://github.com/sgl-project/sglang/actions/runs/29175506877/job/86717337482?pr=30784

…t#30784 (moe) Each side relocated a different kernel into sglang.kernels.ops; resolution takes the union of both new-namespace imports and drops now-relocated old paths. Adopts main's removal of the DSA fused-metadata flag (attention). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t#30784 (attention) Each side relocated a different kernel into sglang.kernels.ops; resolution takes the union of both new-namespace imports and drops now-relocated old paths. Adopts main's removal of the DSA fused-metadata flag (attention). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…t#30784 (dsa-dsv4) Each side relocated a different kernel into sglang.kernels.ops; resolution takes the union of both new-namespace imports and drops now-relocated old paths. Adopts main's removal of the DSA fused-metadata flag (attention). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…nflict (vendored) Union of vendored fla relocation + main's quantization relocation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename per_token_group_quant_v3 -> per_token_group_quant everywhere (files, symbols, custom-op and KernelSpec ids, tests); it is the default kernel, not a versioned variant. Mark the v2 JIT kernel deprecated: a module docstring plus benchmark comments state it is kept only as the perf baseline (benchmark impl keys are now jit_v2/current), and no sglang runtime code may call it. Also adapts to the post-rebase tree: sgl-project#30784 (RFC sgl-project#29630 Phase 2.5) moved fp8_kernel/int8_kernel to sglang.kernels.ops.quantization, so the branch-added tests/benchmarks/wrapper now import from the new paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

#29630 #30784) (#31089) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Rename per_token_group_quant_v3 -> per_token_group_quant everywhere (files, symbols, custom-op and KernelSpec ids, tests); it is the default kernel, not a versioned variant. Mark the v2 JIT kernel deprecated: a module docstring plus benchmark comments state it is kept only as the perf baseline (benchmark impl keys are now jit_v2/current), and no sglang runtime code may call it. Also adapts to the post-rebase tree: sgl-project#30784 (RFC sgl-project#29630 Phase 2.5) moved fp8_kernel/int8_kernel to sglang.kernels.ops.quantization, so the branch-added tests/benchmarks/wrapper now import from the new paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…sgl-project#29630, Phase 2.5, 1/7) (sgl-project#30784) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

sgl-project#29630 sgl-project#30784) (sgl-project#31089) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…sgl-project#29630, Phase 2.5, 1/7) (sgl-project#30784) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

sgl-project#29630 sgl-project#30784) (sgl-project#31089) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

BBuf and others added 2 commits July 12, 2026 09:36

Merge remote-tracking branch 'origin/main' into kernels/phase25-quant…

266771f

…ization

BBuf and others added 3 commits July 13, 2026 10:30

Merge remote-tracking branch 'origin/main' into kernels/phase25-quant…

13b6f29

…ization

Merge remote-tracking branch 'origin/main' into kernels/phase25-quant…

a70cbdc

…ization

BBuf merged commit 874fc07 into sgl-project:main Jul 13, 2026
135 of 186 checks passed

DarkSharpness mentioned this pull request Jul 13, 2026

[JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked) #30924

Merged

8 tasks

BBuf mentioned this pull request Jul 14, 2026

[Kernel] Hotfix: update sgl-kernel imports of relocated fp8_kernel (RFC #29630 #30784) #31089

Merged

BBuf added a commit that referenced this pull request Jul 14, 2026

[Kernel] Hotfix: update sgl-kernel imports of relocated fp8_kernel (RFC

4c99731

#29630 #30784) (#31089) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

yctseng0211 mentioned this pull request Jul 17, 2026

[AMD] Fix stale fp8_kernel import path in test_fused_fp8_kv_write after kernels migration #31562

Draft

5 tasks

amd-bot mentioned this pull request Jul 18, 2026

[CI Monitor] Daily Report - 2026-07-18 bingxche/sglang-ci-bot#139

Open

BBuf mentioned this pull request Jul 22, 2026

[RFC] Introduce a unified sglang.kernels namespace for kernel organization and dispatch #29630

Open

jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026

[Kernel] Migrate scattered quantization kernels to sglang.kernels (RFC …

77830eb

…sgl-project#29630, Phase 2.5, 1/7) (sgl-project#30784) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Navigation Menu

Uh oh!

[Kernel] Migrate scattered quantization kernels to sglang.kernels (RFC #29630, Phase 2.5, 1/7) - #30784

[Kernel] Migrate scattered quantization kernels to sglang.kernels (RFC #29630, Phase 2.5, 1/7)#30784
BBuf merged 7 commits into
sgl-project:mainfrom
BBuf:kernels/phase25-quantization

BBuf commented Jul 10, 2026 •

edited by github-actions Bot

Loading

Uh oh!

BBuf commented Jul 12, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

BBuf commented Jul 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verification

Checklist

Kernel-migration verification (no perf / config regression)

CI States

Uh oh!

BBuf commented Jul 12, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

BBuf commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

BBuf commented Jul 10, 2026 •

edited by github-actions Bot

Loading