Skip to content

[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups into kernels.ops (RFC #29630) - #32045

Merged
BBuf merged 4 commits into
sgl-project:mainfrom
BBuf:bbuf/kernels-phase4-batch3
Jul 22, 2026
Merged

[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups into kernels.ops (RFC #29630)#32045
BBuf merged 4 commits into
sgl-project:mainfrom
BBuf:bbuf/kernels-phase4-batch3

Conversation

@BBuf

@BBuf BBuf commented Jul 22, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

RFC #29630 Phase 4 — batch-3 (tangled subsystems + new groups + splits, no shims: delete + rewrite call sites).

Stage 1 — clean moves: 19 attention ops (flash_attention{,_v3,_v4}, concat_mla, cutedsl_gdn/kda/paged_mqa, rope, fused_qknorm_rope/minimax_qknorm_rope, hadamard, clamp_position, add_constant, mla_kv_pack_quantize_fp8, ...), 8 moe ops, timestep_embedding; flash_attn/ → attention; new groups lplb/, kv_canary/; dsv32/ → quantization.

Stage 2 — tangled:

  • dsa/, dsv4/ → kernels.ops.attention (merged into existing Phase-2.5 dirs, __init__ re-exports merged)
  • trtllm_lora_temp/ (+ its self-contained data/csrc+data/include) → kernels.ops.moe
  • diffusion/ (+ nested cutedsl/flydsl/triton) → kernels.ops.diffusion
  • minimax_m3 split: rmsnorm→layernorm, qk_norm_rope→attention, swiglu→moe
  • triton/ split: gdn_fused_proj→attention, hash_topk/sigmoid_gate_mul→moe
  • inkling_* → new kernels.ops.model.inkling group

All sglang.jit_kernel.* refs rewritten to the new paths (268 files across both stages); lplb/kv_canary/model added to _GROUPS. Namespace import invariant CLEAN; 68 CPU kernels tests pass; lint clean.

After this, jit_kernel/ holds only the 8 batch-1 compat shims + build infra (csrc/include/main) + benchmark/tests — handled by the structural finale + a shim-cleanup PR.

🤖 Generated with Claude Code

CI States

Latest PR Test (Base): ⏳ Run #29923048180
Latest PR Test (Extra): ⏳ Run #29923047574

BBuf and others added 2 commits July 22, 2026 17:59
@BBuf @claude
[Kernel] Phase 4 batch-3 stage-1: clean attention/moe/diffusion ops +…
b6daa5a 
… new groups (RFC sgl-project#29630)

Moves the clean single-file operators and non-colliding subdirs (no shims,
delete+rewrite): 19 attention ops (flash_attention*, concat_mla, cutedsl_gdn/kda/
paged_mqa, rope, fused_qknorm_rope, hadamard, clamp_position, add_constant,
mla_kv_pack_quantize_fp8, ...), 8 moe ops, timestep_embedding, flash_attn/ ->
attention, new groups lplb/ and kv_canary/, dsv32/ -> quantization. 161 call
sites rewritten. Invariant CLEAN; tests pass; lint clean.

Stage-2 (dsa/dsv4/trtllm_lora_temp __init__ merges, diffusion/ nested subdir,
minimax_m3 & triton splits, inkling -> new model group) follows on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@BBuf
[Kernel] Phase 4 batch-3 stage-2: tangled subsystems, splits, new gro…
975c06a 
…ups (RFC sgl-project#29630)

- dsa/, dsv4/ -> kernels.ops.attention (merged into existing Phase-2.5 dirs)
- trtllm_lora_temp/ (+ its self-contained data/csrc) -> kernels.ops.moe
- diffusion/ (+ nested cutedsl/flydsl/triton) -> kernels.ops.diffusion
- minimax_m3 split: rmsnorm->layernorm, qk_norm_rope->attention, swiglu->moe
- triton/ split: gdn_fused_proj->attention, hash_topk/sigmoid_gate_mul->moe
- inkling_* -> new kernels.ops.model.inkling group
- new groups lplb/, kv_canary/ registered in _GROUPS
All refs rewritten (no shims). Invariant CLEAN; 68 tests pass; lint clean.
@gemini-code-assist

gemini-code-assist Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@BBuf BBuf added the run-ci label Jul 22, 2026
@BBuf
BBuf requested a review from YAMY1234 as a code owner July 22, 2026 10:19
@BBuf
BBuf requested a review from rainj-me as a code owner July 22, 2026 10:19
@BBuf
BBuf requested a review from HydraQYH as a code owner July 22, 2026 10:19
@github-actions github-actions Bot added documentation Improvements or additions to documentation quant LLM Quantization amd dependencies Pull requests that update a dependency file lora Multi-modal multi-modal language model deepseek blackwell SM100/SM120 npu diffusion SGLang Diffusion apple-silicon jit-kernel labels Jul 22, 2026
@BBuf

BBuf commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

https://github.com/sgl-project/sglang/actions/runs/29912754833/job/88923162846?pr=32045

@BBuf
BBuf merged commit 74338e9 into sgl-project:main Jul 22, 2026
19 of 20 checks passed
This was referenced Jul 22, 2026
Merged
Open
@mmangkad mmangkad mentioned this pull request Jul 22, 2026
Merged
mattteochen added a commit to mattteochen/sglang that referenced this pull request Jul 22, 2026
@mattteochen @claude
Merge branch 'main' into dsv4/fuse-c128-cleanup
7728432 
Resolve the migration conflict from sgl-project#32045
(jit_kernel.dsv4 -> kernels.ops.attention.dsv4):

- Port the fused C128 cleanup (C128DraftCleanup + fused kernel) onto the
  relocated kernels/ops/attention/dsv4/c128_cleanup.py (rename auto-detected)
  and export C128DraftCleanup from the new package __init__.
- Resolve the deepseek_v4_memory_pool import block to the new path while
  keeping the fused-cleanup construction and dispatch.
- Repoint the new unit test and benchmark to the new import path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@amd-bot amd-bot mentioned this pull request Jul 22, 2026
Open
BBuf added a commit to BBuf/sglang that referenced this pull request Jul 23, 2026
@BBuf @claude
[Kernel] Update kernels/README for the final RFC sgl-project#29630 la…
7f22c9a 
…yout

- Add the jit/ subpackage (shared JIT build/runtime infra) to the Layout.
- All 19 operator groups are populated; drop the stale "reserved placeholders /
  implementations not moved yet" wording (migration complete via sgl-project#31666/sgl-project#32015/
  sgl-project#32045/sgl-project#32072).
- Note sglang.jit_kernel removed; ops live at kernels.ops.<group>._jit_<op>;
  tests/benchmarks under test/registered/kernels/{ops,benchmark}/<group>/;
  shared test helpers in sglang.test.kernels.
- Fix the fused-op parity test path (now under ops/layernorm/).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
whybeyoung pushed a commit to stepinto/sglang that referenced this pull request Jul 27, 2026
ybyang
revert(functional): force CUDA per_token_group_quant to v2 kernel (PR s…
4f45d01 
…gl-project#30924)

A hard 'git revert 8bb0d8d' fails: later PRs (sgl-project#32072/sgl-project#32045/sgl-project#32015)
finalized the sglang.jit_kernel -> sglang.kernels namespace migration,
so the paths sgl-project#30924 wrote to no longer exist. Instead, short-circuit
the runtime dispatcher (_run_per_token_group_quant_8bit_kernel) to
always route the CUDA path to the deprecated-but-still-present v2 JIT
kernel (per_token_group_quant_8bit_v2), which achieves the same
behavioral revert without touching moved files.

Suspected root cause on Hopper (H20):
- PR sgl-project#30924 author's own note: 'flat load tiling fixed at 32B/lane
  (Hopper regression fix)' -- meaning the new kernel had a Hopper
  regression they tried to fix, but 66 non-bit-exact-vs-AOT cases
  remain outside CI coverage.
- Every activation quant on GLM 5.2 NVFP4 goes through this dispatcher;
  a tiny numerical drift in the new kernel compounds through the MoE
  and shows up as low MTP draft accept rate.

Scope of change:
- fp8_kernel.py _run_per_token_group_quant_8bit_kernel: CUDA + group_size
  in {16,32,64,128} -> v2 kernel (v2 does not support group_size=256, so
  fall through to the new kernel there).
- MUSA path unchanged (already AOT v2).
- deep_gemm._varlen_deep_gemm_silu_mul_quant bypasses this dispatcher
  and calls the new per_token_group_quant() directly; NOT covered by
  this revert. Only affects users of the EP-MoE fused varlen path
  (--enable-ep-moe with masked schedule); plain TP MoE decoders (the
  reproducer in PR sgl-project#32209) are fully covered.
Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026
@BBuf @claude @Zhylkaaa
[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups…
affe6e5 
… into kernels.ops (RFC sgl-project#29630) (sgl-project#32045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jinzhenfan pushed a commit to jinzhenfan/sglang that referenced this pull request Jul 29, 2026
@BBuf @claude @jinzhenfan
[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups…
d4d95e2 
… into kernels.ops (RFC sgl-project#29630) (sgl-project#32045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@hnyls2002 hnyls2002 mentioned this pull request Aug 18, 2026
Closed
5 tasks
OrangeRedeng pushed a commit to OrangeRedeng/sglang that referenced this pull request Aug 24, 2026
@BBuf @claude @OrangeRedeng
[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups…
d687a8f 
… into kernels.ops (RFC sgl-project#29630) (sgl-project#32045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
@BBuf @claude @jakki-amd
[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups…
3a8be9f 
… into kernels.ops (RFC sgl-project#29630) (sgl-project#32045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
@BBuf @claude
[Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups…
6a4e37a 
… into kernels.ops (RFC sgl-project#29630) (sgl-project#32045)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fzyzcjy fzyzcjy Awaiting requested review from fzyzcjy fzyzcjy is a code owner
@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner
@ch-wan ch-wan Awaiting requested review from ch-wan
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@Fridge003 Fridge003 Awaiting requested review from Fridge003 Fridge003 is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner
@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner
@hanming-lu hanming-lu Awaiting requested review from hanming-lu hanming-lu is a code owner
@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner
@hzh0425 hzh0425 Awaiting requested review from hzh0425 hzh0425 is a code owner
@alphabetc1 alphabetc1 Awaiting requested review from alphabetc1 alphabetc1 is a code owner
@lifuhuang lifuhuang Awaiting requested review from lifuhuang lifuhuang is a code owner
@yushengsu-thu yushengsu-thu Awaiting requested review from yushengsu-thu yushengsu-thu is a code owner
@jybsuper jybsuper Awaiting requested review from jybsuper jybsuper is a code owner
@mickqian mickqian Awaiting requested review from mickqian
@yuan-luo yuan-luo Awaiting requested review from yuan-luo yuan-luo is a code owner
@yhyang201 yhyang201 Awaiting requested review from yhyang201 yhyang201 is a code owner
@1am9trash 1am9trash Awaiting requested review from 1am9trash 1am9trash is a code owner
@hubertlu-tw hubertlu-tw Awaiting requested review from hubertlu-tw hubertlu-tw is a code owner
@kkHuang-amd kkHuang-amd Awaiting requested review from kkHuang-amd kkHuang-amd is a code owner
@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner
@YAMY1234 YAMY1234 Awaiting requested review from YAMY1234 YAMY1234 is a code owner
@rainj-me rainj-me Awaiting requested review from rainj-me rainj-me is a code owner
@Qiaolin-Yu Qiaolin-Yu Awaiting requested review from Qiaolin-Yu Qiaolin-Yu is a code owner
@hebiao064 hebiao064 Awaiting requested review from hebiao064 hebiao064 is a code owner
@Edwardf0t1 Edwardf0t1 Awaiting requested review from Edwardf0t1 Edwardf0t1 is a code owner
@xutizhou xutizhou Awaiting requested review from xutizhou xutizhou is a code owner
@ping1jing2 ping1jing2 Awaiting requested review from ping1jing2 ping1jing2 is a code owner
@yichiche yichiche Awaiting requested review from yichiche yichiche is a code owner
@AgainstEntropy AgainstEntropy Awaiting requested review from AgainstEntropy AgainstEntropy is a code owner
@yingluosanqian yingluosanqian Awaiting requested review from yingluosanqian yingluosanqian is a code owner
@DarkSharpness DarkSharpness Awaiting requested review from DarkSharpness DarkSharpness is a code owner
@celve celve Awaiting requested review from celve celve is a code owner
@HydraQYH HydraQYH Awaiting requested review from HydraQYH HydraQYH is a code owner

Assignees

No one assigned

Labels

amd apple-silicon blackwell SM100/SM120 bypass-fastfail bypass-maintenance deepseek dependencies Pull requests that update a dependency file diffusion SGLang Diffusion documentation Improvements or additions to documentation enable-retry Enable retry logic for CI tests jit-kernel lora Multi-modal multi-modal language model npu quant LLM Quantization run-ci run-ci-extra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@BBuf