[Kernel] RFC #29630 finale: retire sglang.jit_kernel into sglang.kernels - #32072
Merged
Merged
Conversation
…sglang.kernels
Structural finale of the jit_kernel -> kernels migration. After batch-1/2/3
moved every operator into sglang.kernels.ops.<group>, this removes the
sglang.jit_kernel package entirely.
- Move shared JIT build infra (csrc/, include/, __main__.py, benchmark/,
tests/, .clang-format) into sglang/kernels/jit/.
- KERNEL_PATH resolution: find_spec("sglang.jit_kernel") ->
find_spec("sglang.kernels.jit"); csrc/include now resolve under kernels/jit.
- Delete the last 8 batch-1 compat shims (activation, norm, dsv3_fused_a_gemm,
dsv3_router_gemm, set_mla_kv_buffer, per_tensor_quant_fp8,
per_token_group_quant, per_token_group_quant_8bit_v2) and rewrite all call
sites to sglang.kernels.ops.<group>._jit_<op>.
- Fix batch-2/3 dangling refs using the package-relative / top-level-tree
import forms (inkling_*, cutedsl_gdn/kda, dsa, rope, hicache, triton.* split,
flash_attention) under benchmark/ and sgl-kernel/.
- Packaging: package-data glob jit_kernel/**/* -> kernels/**/* (ships csrc,
include, and moved data dirs like trtllm_lora_temp/data).
- CI: change-filters, labeler, CODEOWNERS, MAINTAINER.md now point at
python/sglang/kernels/**; namespace-invariant test guards sglang.kernels.jit.
- python/sglang/jit_kernel/ no longer exists.
Validated: namespace import stays metadata-only (CLEAN); KERNEL_PATH resolves
to kernels/jit with csrc+include; all rewrite targets importable; 155 changed
files compile; pre-commit clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BBuf
requested review from
Qiaolin-Yu,
hnyls2002,
ispobock,
kpham-sgl,
merrymercy and
yizhang2077
as code owners
July 22, 2026 13:41
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
BBuf
requested review from
AgainstEntropy,
Alisehen,
AniZpZ,
DarkSharpness,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
HydraQYH,
Kangyan-Zhou,
OrangeRedeng,
Ying1123,
b8zhong,
bingxche,
celve,
ch-wan,
fzyzcjy,
mickqian,
ping1jing2,
yhyang201,
yichiche and
yuan-luo
as code owners
July 22, 2026 13:41
BBuf
added a commit
to BBuf/sglang
that referenced
this pull request
Jul 23, 2026
…yout - Add the jit/ subpackage (shared JIT build/runtime infra) to the Layout. - All 19 operator groups are populated; drop the stale "reserved placeholders / implementations not moved yet" wording (migration complete via sgl-project#31666/sgl-project#32015/ sgl-project#32045/sgl-project#32072). - Note sglang.jit_kernel removed; ops live at kernels.ops.<group>._jit_<op>; tests/benchmarks under test/registered/kernels/{ops,benchmark}/<group>/; shared test helpers in sglang.test.kernels. - Fix the fused-op parity test path (now under ops/layernorm/). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
whybeyoung
pushed a commit
to stepinto/sglang
that referenced
this pull request
Jul 27, 2026
…gl-project#30924) A hard 'git revert 8bb0d8d' fails: later PRs (sgl-project#32072/sgl-project#32045/sgl-project#32015) finalized the sglang.jit_kernel -> sglang.kernels namespace migration, so the paths sgl-project#30924 wrote to no longer exist. Instead, short-circuit the runtime dispatcher (_run_per_token_group_quant_8bit_kernel) to always route the CUDA path to the deprecated-but-still-present v2 JIT kernel (per_token_group_quant_8bit_v2), which achieves the same behavioral revert without touching moved files. Suspected root cause on Hopper (H20): - PR sgl-project#30924 author's own note: 'flat load tiling fixed at 32B/lane (Hopper regression fix)' -- meaning the new kernel had a Hopper regression they tried to fix, but 66 non-bit-exact-vs-AOT cases remain outside CI coverage. - Every activation quant on GLM 5.2 NVFP4 goes through this dispatcher; a tiny numerical drift in the new kernel compounds through the MoE and shows up as low MTP draft accept rate. Scope of change: - fp8_kernel.py _run_per_token_group_quant_8bit_kernel: CUDA + group_size in {16,32,64,128} -> v2 kernel (v2 does not support group_size=256, so fall through to the new kernel there). - MUSA path unchanged (already AOT v2). - deep_gemm._varlen_deep_gemm_silu_mul_quant bypasses this dispatcher and calls the new per_token_group_quant() directly; NOT covered by this revert. Only affects users of the EP-MoE fused varlen path (--enable-ep-moe with masked schedule); plain TP MoE decoders (the reproducer in PR sgl-project#32209) are fully covered.
chuyeh
added a commit
to chuyeh/sglang
that referenced
this pull request
Jul 29, 2026
pr-test-amd-rocm724.yml was branched from pr-test-amd-rocm720.yml before the kernel module reorganization, and being a separate file it does not inherit later template fixes. Carry over the two that diverged: - Point the jit_kernel and multimodal_gen path filters at sglang/kernels and test/registered/kernels (sgl-project#32072, sgl-project#32128). The old jit_kernel paths no longer exist, so those filters could never match and the jobs would be skipped in filtered mode without any error. - Split stage-c-test-large-8-gpu-amd-mi35x across 3 partitions instead of 2, matching sgl-project#24651. The suite grew, and two partitions risk exceeding the 60-minute timeout. Co-authored-by: Cursor <cursoragent@cursor.com>
Zhylkaaa
pushed a commit
to Zhylkaaa/sglang
that referenced
this pull request
Jul 29, 2026
…sglang.kernels (sgl-project#32072) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jinzhenfan
pushed a commit
to jinzhenfan/sglang
that referenced
this pull request
Jul 29, 2026
…sglang.kernels (sgl-project#32072) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
long10024070
added a commit
to long10024070/sglang
that referenced
this pull request
Aug 2, 2026
Upstream RFC sgl-project#29630 finale (sgl-project#32072) removed the legacy sglang.jit_kernel package; aiter_paged_mqa_logits now lives in sglang.kernels.ops.attention.dsa.paged_mqa_logits. The rebase carried the PR test file over with the old import path, breaking collection.
long10024070
added a commit
to long10024070/sglang
that referenced
this pull request
Aug 2, 2026
Upstream RFC sgl-project#29630 finale (sgl-project#32072) removed the legacy sglang.jit_kernel package; aiter_paged_mqa_logits now lives in sglang.kernels.ops.attention.dsa.paged_mqa_logits. The rebase carried the PR test file over with the old import path, breaking collection.
This was referenced Aug 10, 2026
hassellof
added a commit
to hassellof/sglang
that referenced
this pull request
Aug 12, 2026
Boot on sage (SM120, flashinfer autotune dummy forward) fails with `paged_mqa_metadata.cuh: CUDA error: invalid argument` at kernel launch: the upstream form (PR sgl-project#32072 retire jit_kernel -> kernels) sets MaxDynamicSharedMemorySize to 131KB unconditionally and picks use_smem purely from batch size, so a large batch requests dynamic smem that exceeds the CUDA opt-in per-block budget -> cudaLaunchKernel returns invalid argument. Restore the budget-capped launcher carried by our fix33 fork: query cudaDevAttrMaxSharedMemoryPerBlockOptin and the kernel's static smem, cap the requested dynamic smem, and only use shared memory when (batch+1)*4 fits the granted budget. Byte-identical to the proven fix33 image kernel. Prior art: sglang/flashinfer indexer smem "invalid argument" reports are widely attributed to requesting smem beyond the device opt-in limit; the documented community fix is exactly this clamp. Decision: keep-local (candidate to upstream) since PR sgl-project#32072 ships the un-clamped form.
5 tasks
Chronostasys
pushed a commit
to MindLab-Research/sglang
that referenced
this pull request
Sep 1, 2026
…ernel Rebased onto the sglang.kernels layout per RFC sgl-project#29630 (sgl-project#31666/sgl-project#32015/sgl-project#32072) now that sglang.jit_kernel is retired. No logic changes: the patch now lives in kernels/jit/csrc/kvcacheio/hicache.cuh and kernels/ops/kvcache/hicache.py.
This was referenced Sep 4, 2026
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
…sglang.kernels (sgl-project#32072) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
…sglang.kernels (sgl-project#32072) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Structural finale of the
sglang.jit_kernel→sglang.kernelsmigration (RFC #29630). After batch-1/2/3 (#31666, #32015, #32045) moved every operator intosglang.kernels.ops.<group>, this PR removes thesglang.jit_kernelpackage entirely.What changed
sglang/kernels/jit/:csrc/,include/,__main__.py,benchmark/,tests/,.clang-format(git-tracked as renames).KERNEL_PATHrefactor (kernels/jit/utils/compile.py):find_spec("sglang.jit_kernel")→find_spec("sglang.kernels.jit").csrc/+include/now resolve underkernels/jit/.sglang.kernels.ops.<group>._jit_<op>.from sglang.jit_kernel import X) or top-level-tree import forms and were missed by the earlier dotted-only rewrites:inkling_*,cutedsl_gdn/cutedsl_kda,dsa,rope,hicache, the splittriton.*submodules,flash_attention— acrossbenchmark/andsgl-kernel/.[tool.setuptools.package-data]globjit_kernel/**/*→kernels/**/*(ships csrc, include, and moved data dirs liketrtllm_lora_temp/data)._pr-test-check-changes,pr-test-amd,pr-test-amd-rocm720,pr-test-npu),labeler.yml,CODEOWNERS,MAINTAINER.mdnow point atpython/sglang/kernels/**. The namespace-invariant test now guardssglang.kernels.jit.python/sglang/jit_kernel/no longer exists.Validation (local, torch 2.11)
import sglang.kernels.opsstays metadata-only → CLEAN (nosgl_kernel/sglang.kernels.jit).KERNEL_PATHresolves tokernels/jitwithcsrc/+include/present.find_spec..pyfiles compile;pre-commitclean.Part of RFC #29630. This is the last migration PR — the
sglang.kernelsnamespace is now the single home for all kernels.CI States
Latest PR Test (Base): ⏳ Run #29967692037
Latest PR Test (Extra): ⏳ Run #29967691955