Repository navigation
[Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration - #45389
Merged
Merged
Conversation
…o low_latency_kernels Move sglang.srt.megakernels and sglang.srt.monokernels, with their unit tests, to sglang.kernels.low_latency_kernels; only import paths change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pinned sgl-nvidia-mega and sgl-amd-mono import sglang.srt.megakernels and sglang.srt.monokernels; alias the modules they import to the moved ones so the installed plugins keep loading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
KernelBackend.LOW_LATENCY records kernels in low_latency_kernels, with a forward_low_latency hook on BaseFusedOp; the namespace test pairs the directory with the backend as it does for kda_kernels and cake_kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ernel amd is the only monokernel backend, so the flag becomes a boolean beside --enable-nvidia-megakernel; --monokernel-backend amd stays as a deprecated spelling of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BBuf
requested review from
DarkSharpness,
Fridge003,
HaiShaw,
HydraQYH,
JustinTong0323,
Qiaolin-Yu,
Ying1123,
b8zhong,
celve,
hebiao064,
hnyls2002,
ishandhanani,
ispobock,
merrymercy,
mmangkad,
sogalin,
wisclmy0611,
yctseng0211,
yuan-luo and
zijiexia
as code owners
October 20, 2026 12:37
github-actions
Bot
added
documentation
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
# Conflicts: # docker/Dockerfile # docker/Dockerfile.cu134
Collaborator
Author
|
Reverted the contract rename: back to |
Collaborator
Author
|
/rerun-test -c test_dspark_draft_block.py test_dspark_step_inputs_once.py test_dspark_draft_graph.py test_startup_weight_load.py |
Collaborator
Author
|
/run-cpu-ci |
Contributor
|
🚀 🚀 ⛔
Please provide the full path, e.g. |
Contributor
|
🚀 Running |
5 tasks
5 tasks done
nvrohanv
added a commit
to nvrohanv/sgl-nvidia-mega
that referenced
this pull request
Oct 25, 2026
…arget verify and NextN draft on B300 TP8 Add sgl_nvidia_mega/glm5_3: the GLM-5.3 target spec for SGLang's megakernel layer (MTP verify steps of 1 to 4 requests, 4 to 16 rows, on 8 x B300 at TP 8), its step functions with stock paths adapted from SGLang's model code, and the CuTe DSL attention, small-M NVFP4 MoE and all-reduce hop kernels. With --enable-nvidia-megakernel every target slot runs (attention, moe, hops, qkv_a_small_m_shared, dnorm); SGLANG_NVIDIA_MEGAKERNEL_SLOTS is a test and bisect override. Needs SGLang's DSA binding view. Add the GlmMoeDsaForCausalLMNextN draft spec (slot draft, on by default) and its CuTe DSL kernels: the embedding gather and eh norms in one kernel, the column-sharded eh projection gathered by the target's all-reduce hop, the distributed top-1, and the draft layer's MLA prologue, in every captured draft graph. Needs embedding replication (with the flag, SGLang's megakernel layer sets SGLANG_ENABLE_EMBED_REPLICATION when it is unset), greedy draft acceptance and SGLang's EAGLE draft graph capture check; the contract test skips the check's marker, naming it, on an SGLang without it. At startup rank 0 logs the covered verify widths (1-4 requests) and the derived weight copies' memory per GPU, and warns when --max-running-requests is set above 8, where the megakernel helps little. Also the README section, a NOTICE paragraph, a CODEOWNERS line and the contract test's checks of the DSA binding view reads and the mirrored SGLang code; the GLM-5.3 tests are filed separately. The same specs serve GLM-5.2 NVFP4, which has GLM-5.3's architecture (GlmMoeDsaForCausalLM) and the same value of every config field the target and draft specs pin. The spec names and the plugin's log, refusal and rank agreement texts name both models, and the module path stays glm5_3. test/unit/test_glm5_3_glm5_2_config.py runs the published config of nvidia/GLM-5.2-NVFP4 (revision 53e0691e21895a3863a606dfd12910c69eba94ab) through SGLang's requirement check for both specs, and checks that a config that differs in any pinned value, or lacks one, is still refused. The package imports SGLang's megakernel interface from its location since sgl-project/sglang#45389 (sglang.kernels.plugin_kernels.nvidia, sglang.srt.kernel_plugins.nvidia) and falls back to sglang.srt.megakernels on an older SGLang, as deepseek_v41 does.
4 of 5 tasks
amd-cgilli
added a commit
to amd-cgilli/sgl-amd-mono
that referenced
this pull request
Oct 26, 2026
Drop unused helpers, parameters and constants and the provenance comments, trim comments that restated the code, and import SGLang's monokernel interface from its new location first (sgl-project/sglang#45389), as the other specs do. The kernels compile to the same IR.
5 tasks done
4 tasks done
jhinpan
pushed a commit
to sgl-project/sgl-amd-mono
that referenced
this pull request
Oct 27, 2026
…s wait deadline (#61) ## Motivation Since #23, a routed decode step starts with layer 0's launch, which takes the step's tokens and sums the embedding rows over the ranks. So no RCCL collective, which has no deadline, meets the ranks before the step's first wait on a peer. A captured graph runs a step's launches back to back, so any host work between steps lands in that first launch. Its waits have the same deadline as every other wait: half of `--watchdog-timeout`, at most 60 s. A cold SGLang JIT cache creates such host work. The first real decode builds `clamp_position` (`ForwardBatch.init_new` → `load_jit`) under an `fcntl.flock` in `~/.cache/sglang/jit`. When that directory is on NFSv3 with `local_lock=none`, a rank blocked on the lock wakes only on its next 30 s poll. The ranks then reach their first routed step 0, 30 or 60 s apart. Past 60 s, the early ranks' first wait on a peer times out and the guard stops every GPU: a false fail-stop. On main, 3 of 5 cold boots died this way during the server's own warmup request. The GPUs faulted at 0x0 in layer 0's launch (`glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1`). A host stall of more than 60 s between steps, on any one rank, does the same. ## Modifications - `spec.py`: the runtime that runs layer 0 gets `min(max(ctx.deadline_s, 0.5 * watchdog_timeout), guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ)` as its wait deadline. That runtime is the one with `key.tokens`, and layer 0 is always a step's first launch, since `_build` refuses a layer 0 that is not dense. Every other runtime keeps `ctx.deadline_s`. - The new deadline is half the watchdog, SGLang's own bound for a wait, without its 60 s cap. So a wait that never ends still stops every GPU before the watchdog fires. - It is at most 5,497 s, the longest whole second that a wait's 32-bit deadline check holds (`guard.MAX_DEADLINE_TICKS`, 5,497.56 s). So from `--watchdog-timeout 10994` on, and at `inf`, layer 0's deadline is 5,497 s, and `guard.deadline_ticks` takes it. Whole seconds keep the cap's conversion to ticks exact. | `--watchdog-timeout` | every other launch | layer 0's launch | | --- | --- | --- | | 300 s (SGLang's default) | 60 s | 150 s | | 600 s (the e2e test's) | 60 s | 300 s | | 20 s (the e2e fault test's) | 10 s | 10 s | | 11,100 s | 60 s | 5,497 s | | `inf` | 60 s | 5,497 s | - `_build` passes the cap to `_deadline`, so the deadline test runs on a CPU-only host: `common/guard.py` imports AITER, which needs a GPU to import. - The spec reads `get_device().watchdog_timeout` directly and declares `get_device` and `watchdog_timeout` in `SGLANG_READS`. SGLang computes `ctx.deadline_s` from the same read: `monokernels/runtime.py` on ab580b2e06, `kernel_plugins/amd/binding.py` after sgl-project/sglang#45389. - No kernel change. The deadline is a word of the launch's guard block, which the host writes at build (`FlashRuntime.set_deadline`). - `test/unit/test_glm53_flash_spec.py`: `test_a_step_s_first_launch_waits_up_to_half_the_watchdog` checks both deadlines at watchdogs of 300, 600, 20 and 11,100 s and `inf`. It takes the framework's deadline from SGLang's own `deadline_s`. - Second commit, test only, in the same file, promised in #27's review: `test_the_spec_needs_aiters_grouped_topk` re-implemented SGLang's env comparison, and `test_sglang_refuses_its_jit_grouped_topk` replaces it. - The new test calls SGLang's `monokernel_disable_reasons` with and without `SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK=1`. Setting the variable must add exactly that refusal. - Like #25, it imports from #45389's location first. What this does not change: - A real stall in layer 0's launch now stops after half the watchdog (150 s at the default) instead of 60 s. With `--watchdog-timeout inf`, SGLang's watchdog never fires, and such a stall stops after 5,497 s. - With `--disable-cuda-graph`, each launch is enqueued by its host, so host skew can reach any launch, and those launches keep 60 s. - The stall itself remains: this fix only removes the false stop. A SGLang JIT cache on local disk (`SGLANG_JIT_CACHE_DIR`) avoids the 30–60 s wait. ## Accuracy Tests No numerics change: there is no kernel or forward change. Setup: MI355X, `rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014` (`sha256:e4498f5d595f`), SGLang main ab580b2e06, `amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25`. Main is at da5faf9 and this PR at ac9f232. The boots run at TP4 with the e2e test's server flags, each a fresh server: its own warmup request makes the first routed step, then one 32-token greedy request follows. Rank 3's host stalls before its first routed decode replay, using the out-of-tree hook in the first comment below. **Around the guard's limit** (5 s stall on rank 3). Layer 0's deadline is 60 s on main at every watchdog here. | `--watchdog-timeout` | Layer 0's deadline with this PR | main | this PR | | --- | --- | --- | --- | | 10,900 s | 5,450 s | serves | serves | | 11,100 s | 5,497 s (the cap) | serves | serves | | `inf` | 5,497 s (the cap) | serves | serves | With `inf`, each scheduler's watchdog thread exits at start, on main as here (`time.sleep(inf)` raises `OverflowError` in SGLang's `watchdog.py`), so SGLang's watchdog never fires. **Host stall into the first routed step:** | Case | Watchdog | Layer 0's deadline (main / this PR) | main | this PR | | --- | --- | --- | --- | --- | | 90 s stall | 600 s | 60 s / 300 s | GPUs 0 to 2 fault at 0x0 in layer 0's launch 60.0 s after the stall starts; the server never comes up | the first routed step waits out the stall; the server is up 90.1 s after it starts, and the request completes | | 400 s stall: a peer that never arrives | 300 s | 60 s / 150 s | GPUs 0 to 2 fault at 0x0 in layer 0's launch at 60.1 s | GPUs 0 to 2 fault at 0x0 in layer 0's launch at 150.0 s, before SGLang's watchdog | **A stall past the first launch** uses the repo's fault injection: watchdog 300 s and `SGLANG_AMD_MONO_FAULT_INJECT=1:22:20`, with the e2e fault test's request (2 × 1,024 tokens). - Rank 1 stalls at guarded wait 22 once 20 steps have run. Layer 0's launch and the dense launches have at most 21 guarded waits, and the KDA and sparse-attention launches have 22 to 51, so only a later launch can stall. - All four GPUs fault at 0x0 in a KDA launch (`glm53_flash_layer_s2_tp4_k1_d0_q1_h1`) 60.2 s after the request. No tokens are returned, and the server exits. - So the later launches keep the 60 s deadline. **Cold SGLang JIT cache** (watchdog 600 s, the cache removed before each boot, `HOME` on NFSv3 with `local_lock=none`; main and this PR alternate, 5 boots each). "Entry skew" is the last rank's entry into its first routed decode replay, measured from the first rank's. It comes from a per-rank host timeline that is out of tree. | | Passed | Entry skew (s) | | --- | --- | --- | | main | 2 of 5 | failed: three boots with a rank still waiting on the lock when the GPUs stopped; passed: 30.2, 58.7 | | this PR | 5 of 5 | 58.2, 60.6, 60.6, 30.1, 60.3 | - On main, the GPUs stop 60.1, 60.0 and 60.0 s after the first rank enters, and the server never comes up. - With this PR, the early ranks' first step waits up to 60.6 s on the GPU, and the request completes. - One more boot of main is not counted: its server failed to open its device at start (`hipErrorLaunchFailure` in `current_device()`), before loading the model. Its rerun is the 30.2 s pass. **Other checks on this head:** - **E2E:** `test_glm53_flash_server.py` at TP4 with `HSA_DISABLE_COREDUMP_ON_EXCEPTION=1`, 5 of 5 pass: GSM8K 0.975, mean KL against prefill 0.012 to 0.033 (bound 0.06), 21 of 21 returned hidden-state rows give the sampled token, and the fault case, where both deadlines are 10 s. - **Code objects:** compile-only builds of every kind (FFN-only, KDA, sparse attention, dense, layer 0, tail), at every width and at TP 2, 4 and 8, preshuffled and row-major FP8 where they apply: 72 a tree. - In one build of each tree, 70 of 72 are byte-identical to main's. - The other two vary from build to build on main itself. Over 7 builds of main and 6 of this PR, the TP4 sparse-attention builds at 2 and 8 rows (both layouts) each give the same 2 or 3 variants on both trees, and every other build never varies. - Every build of both trees has the same VGPRs, SGPRs, spills, LDS and scratch as main's. - **Unit tests**, run with `--continue-on-collection-errors`. Two MiniMax test files fail at collection on a CPU-only host, on main as here. - CPU only, SGLang ab580b2e06: 261 passed (main 260). On both, 3 contract cases fail: deepseek_v41's two and minimax_m31's. - CPU only, SGLang main 39b54ae0ee: 262 passed (main 261). On both, the minimax and minimax_m31 contract cases fail. - With a GPU visible, SGLang ab580b2e06: 293 passed (main 292), with the same 3 contract failures. - Negative controls, CPU only: without the `min`, the deadline test fails at 11,100 s (5,550.0 against 5,497), and with that case removed, at `inf` (inf against 5,497). If `_deadline` returns the framework's deadline for every key, it fails at 300 s (60.0 against 150.0). Without the variable in `requires.env`, the grouped top-k test fails. ## Speed Tests and Profiling Not measured. The code objects are byte-identical to main's. At run time, the only difference is the value of one guard word in layer 0's runtime, and a wait reads that word only on its periodic check while it is still pending. ## Checklist - [x] `pre-commit run --all-files` passes. - [x] Tests under `test/` cover the change. - [x] The SGLang commit or PR this was tested against is named above (SGLang main ab580b2e06; the CPU unit tests also on 39b54ae0ee). - [x] Accuracy and speed results are attached when kernel or forward code changes (no kernel or forward change). --------- Co-authored-by: Hassan Eslami <hassan@marlowe.ai>
2 of 5 tasks
4 of 5 tasks
chuyeh
pushed a commit
to sgl-project/sgl-amd-mono
that referenced
this pull request
Oct 30, 2026
…nokernel (#29) ## Motivation Qwen3.8-Flash-Next (SGLang's Qwen4-Exp, `amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8`) decodes with its MTP head (NEXTN) at small batch sizes, where launch gaps dominate. This PR adds a FlyDSL monokernel for it to sgl-amd-mono, behind SGLang's monokernel interface (RFC #43949). `--monokernel-backend amd` selects it, and SGLang needs no model hooks. The server runs SGLang's AITER path (`SGLANG_USE_AITER=1`; the full launch command is in the README). - A verify step of 1-8 requests x 4 tokens (embedding, PLE, 48 GDN / QSA layers and the final mixer) runs in 13 persistent launches. - Each MTP draft step of 1-8 requests runs in one launch: a draft extend (R x 4 rows) or a draft decode step (R x 1 rows). Monokernel API v1 routes every runner of a speculative server by the target's verify key, which the NEXTN draft never runs. The draft spec therefore routes its own steps: a wrapper in front of the draft's `model.forward`. It can move to SGLang if a later monokernel API routes a draft's own step modes. The kernel itself (launch plan, stages, LDS and scratch layout, synchronization and fail-stop, weights, numerics) is described in [`sgl_amd_mono/qwen38_flash_next/README.md`](https://github.com/amd-cgilli/sgl-amd-mono/blob/qwen38-flash-next-monokernel/sgl_amd_mono/qwen38_flash_next/README.md). Below is a comparison of the before and after: <img width="1612" height="727" alt="b4" src="https://github.com/user-attachments/assets/85b581aa-d309-4fd5-8774-24fcb2d810dc" /> <img width="1272" height="669" alt="after" src="https://github.com/user-attachments/assets/2834498c-02a2-4546-b9c4-25d51567dfdb" /> ## Modifications - `sgl_amd_mono/qwen38_flash_next/`: - `spec.py`: the target spec, registered for `Qwen4ExpForConditionalGeneration` on gfx950, TP1, 1-8 requests x 4 tokens. It refuses unsupported configurations with reasons: - NEXTN/EAGLE with topk 1, 3 steps and 4 draft tokens only; - no ReplaySSM, no PP speculative scratch remapping; - bf16 KV, no HiSparse; - `SGLANG_USE_AITER`, the AITER/auto MoE runner, the fused shared expert, no paged experts or MegaMoE, no expert simulation or recording; - the Quark MXFP4 checkpoint; - `--attention-backend` triton or aiter; - plus per-layer config and weight-layout checks. - Image and video requests: a verify step is rotated by SGLang's MRoPE positions (`forward_batch.mrope_positions`, as the model's own forward does), and each request's sequence start comes from `seq_lens`. With images or video the two differ, and the QSA index tiles, ring slots and selection count from the sequence position. - QSA index-key ring: the launches index SGLang's QSA key ring by the width its pool allocates (`ring_width` in `gfx950/runner.py`: 4 or 8 slots a request). With today's 4-slot ring (one group), a verify row past the pending group overwrites one of its members, and a rejected row's key stays in the slot, so the group can later compress with it; SGLang's own verify has the same issue. sgl-project/sglang#47051 makes the ring two groups (8 slots), which keeps every pending member. The launches work with either layout. - `draft.py`: the MTP draft spec, registered for `Qwen4ExpForCausalLMMTP`. - `_build` installs a wrapper in front of the draft's `model.forward`. - The wrapper sends draft extend (`DRAFT_EXTEND_V2`, R x 4 rows) and draft decode (`DECODE`, R x 1 rows) steps of R <= 8 requests to the draft launch, and every other step to SGLang's forward. - The decision depends only on the step's shape, so a captured graph replays it. - It requires the draft's QSA attention backends (`"qsa"`), logs each draft step at its first launch, and refuses during capture a draft-shaped step the launch cannot read. - `check.py`: `SGLANG_AMD_MONO_CHECK=1` checks the target only: - at build, it compares the layer weights against the checkpoint; - on each eager verify step, it compares the logits and hidden states against SGLang's forward of the same step (`NUMERICS`). - `gfx950/`: - the kernel (`kernel.py`, the guarded wait in `device.py`), its layout, packing and config; - the layer views read from SGLang's params in place (`weights.py`); - the host-side runners (`runner.py`). The draft runner copies `input_ids` and new hidden states into buffers it owns, so its pointer tables stay valid under graph capture. - `README.md`: the kernel documentation. - The specs import SGLang's interface from `sglang.kernels.plugin_kernels.amd` and fall back to `sglang.srt.monokernels` (SGLang before sgl-project/sglang#45389), as the other specs do. - `sgl_amd_mono/__init__.py`: SPECS entries for the target and the MTP draft. - `README.md`: Supported Models, Usage, Structure, Testing. - Tests: - `test/unit/test_qwen38_flash_next_{spec,layout}.py`, CPU. These cover the specs, refusals, layout, the RoPE positions a routed verify passes, and the draft wrapper: - routing of draft extend and decode steps; - the fallback to SGLang's forward; - stacking under the runtime's forward switch; - the per-mode backend check; - the capture-time refusal; - the fixed-address input buffers. - `test/kernels/test_qwen38_flash_next_compile.py`, GPU. - `test/e2e/{qwen38_flash_next_server,test_qwen38_flash_next_server}.py`, GPU: - GSM8K at 1/4/16 requests, accept length; - the target and draft launch lines; - KL decode vs prefill; - 126k passkey; - check mode; - fault injection (target launches). ## Accuracy Tests 1 x MI355X, TP1, SGLang upstream main 2229345dff, NEXTN 3 steps / 4 draft tokens / topk 1, real acceptance. "Off" is the same server without `--monokernel-backend amd`. Measured before the rebase onto current main. The rebase and the cleanup leave the kernels' LLVM IR unchanged, and the MRoPE change reads the same sequence start for text requests. | Test | Monokernel on | Off | |---|---|---| | GSM8K, 400 q | 0.970 (accept length 3.52) | 0.975 (accept length 3.52) | | GSM8K 5-shot chat, 200 q, at 1 / 4 / 16 requests | 0.98 / 0.98 / 0.97 | 0.98 / 0.985 / 0.975 | | AIME25, 30 q x 4 seeds, pass@1 (thinking on, T 0.6, top-p 0.95) | 96.67% (116 of 120) | 98.33% (118 of 120) | | AIME25, 30 q x 4 seeds, pass@4 | 100% | 100% | | KL decode vs prefill, mean over fresh servers (bound 0.1) | 0.062 (5 runs, 0.041 to 0.085) | 0.059 (4 runs, 0.051 to 0.069) | | RULER NIAH at ~261K tokens (niah_single_1/2/3, niah_multikey_1, 25 each) | 100 | 100 | | GPQA-Diamond pass@1 (same settings) | 0.855 (sem .006) | 0.836 (sem .004) | | GPQA-Diamond majority@4 / pass@4 | 0.871 / 0.909 | 0.884 / 0.899 | <details> <summary>Unit + contract tests (CPU): 102 passed</summary> ```text date: 2026-10-29 09:50 UTC sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 sglang from: /sgl-workspace/sglang-main-latest/python/sglang versions: torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0 GPU: 1 x AMD Instinct MI355X (gfx950) $ python -m pytest -v -p no:cacheprovider test/unit/test_qwen38_flash_next_spec.py test/unit/test_qwen38_flash_next_layout.py test/unit/test_contract.py -k qwen38 ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python rootdir: /sgl-workspace/sgl-amd-mono configfile: pyproject.toml plugins: platformdirs-4.13.5, anyio-4.15.1 collecting ... collected 136 items / 34 deselected / 102 selected test/unit/test_qwen38_flash_next_spec.py::test_the_plugin_registers_the_target_and_the_draft PASSED [ 0%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_spec_declares_the_drafts_own_backend PASSED [ 1%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_keeps_the_forward_signature PASSED [ 2%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-1-4] PASSED [ 3%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DRAFT_EXTEND_V2-8-32] PASSED [ 4%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-1-1] PASSED [ 5%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[positional-DECODE-8-8] PASSED [ 6%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-1-4] PASSED [ 7%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DRAFT_EXTEND_V2-8-32] PASSED [ 8%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-1-1] PASSED [ 9%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_launches_the_draft_steps[kwarg-DECODE-8-8] PASSED [ 10%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests] PASSED [ 11%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-nine requests extend] PASSED [ 12%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-verify] PASSED [ 13%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[positional-extend] PASSED [ 14%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests] PASSED [ 15%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-nine requests extend] PASSED [ 16%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-verify] PASSED [ 17%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_wrapper_runs_the_forward_it_wraps[kwarg-extend] PASSED [ 18%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_with_input_embeds_is_refused PASSED [ 19%] test/unit/test_qwen38_flash_next_spec.py::test_the_runtimes_forward_wraps_the_draft_wrapper PASSED [ 20%] test/unit/test_qwen38_flash_next_spec.py::test_the_wrapper_checks_the_backend_once_a_step_mode PASSED [ 21%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[three rows a request] PASSED [ 22%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[fp32 hidden] PASSED [ 23%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[hidden of other rows] PASSED [ 24%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[strided hidden] PASSED [ 25%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_the_launch_cannot_read_is_refused[no hidden states] PASSED [ 26%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[qsa] PASSED [ 27%] test/unit/test_qwen38_flash_next_spec.py::test_a_draft_step_on_shared_sparse_indices_is_refused[hybrid] PASSED [ 28%] test/unit/test_qwen38_flash_next_spec.py::test_every_draft_call_site_calls_the_model_forward_attribute PASSED [ 29%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_writes_the_buffer_its_input_does_not_overlap PASSED [ 30%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_reads_new_hidden_states_from_a_fixed_address PASSED [ 31%] test/unit/test_qwen38_flash_next_spec.py::test_build_installs_the_wrapper PASSED [ 32%] test/unit/test_qwen38_flash_next_spec.py::test_the_draft_worker_reads_index_sharing_off_after_the_build PASSED [ 33%] test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_is_accepted PASSED [ 34%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[hidden size] PASSED [ 35%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[experts] PASSED [ 36%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[index budget] PASSED [ 37%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[layer pattern] PASSED [ 38%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[quantization] PASSED [ 39%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[ssm dtype] PASSED [ 40%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[context] PASSED [ 41%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_setting_is_one_reason[no flydsl] PASSED [ 42%] test/unit/test_qwen38_flash_next_spec.py::test_the_ple_layer_is_one_reason PASSED [ 43%] test/unit/test_qwen38_flash_next_spec.py::test_the_recipe_routes_verify_batches_of_one_to_eight PASSED [ 44%] test/unit/test_qwen38_flash_next_spec.py::test_the_requirements_are_declared_sglang_reads PASSED [ 45%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[two tokens] PASSED [ 46%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[nine requests] PASSED [ 47%] test/unit/test_qwen38_flash_next_spec.py::test_a_step_the_launches_do_not_take_runs_the_models_forward[decode] PASSED [ 48%] test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[mrope] PASSED [ 49%] test/unit/test_qwen38_flash_next_spec.py::test_a_routed_verify_rotates_by_the_positions_sglang_applies[text positions] PASSED [ 50%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_reaches_the_jit_cache_key PASSED [ 50%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[kind] PASSED [ 51%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[empty] PASSED [ 52%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[requests] PASSED [ 53%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[tokens] PASSED [ 54%] test/unit/test_qwen38_flash_next_spec.py::test_a_build_key_out_of_the_kernels_range_is_refused[first source] PASSED [ 55%] test/unit/test_qwen38_flash_next_spec.py::test_a_verify_is_thirteen_launches_of_the_48_layers PASSED [ 56%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[1] PASSED [ 57%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[2] PASSED [ 58%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[3] PASSED [ 59%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[4] PASSED [ 60%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[5] PASSED [ 61%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[6] PASSED [ 62%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[7] PASSED [ 63%] test/unit/test_qwen38_flash_next_spec.py::test_the_scratch_regions_do_not_overlap[8] PASSED [ 64%] test/unit/test_qwen38_flash_next_spec.py::test_the_layout_matches_the_spec PASSED [ 65%] test/unit/test_qwen38_flash_next_spec.py::test_the_views_accept_the_layers PASSED [ 66%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[no fused shared expert] PASSED [ 67%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[unshuffled experts] PASSED [ 68%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[interleaved gate and up] PASSED [ 69%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[2-byte scales] PASSED [ 70%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[dequantized projection] PASSED [ 71%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[wide b / a projection] PASSED [ 72%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[bf16 projection] PASSED [ 73%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[conv weight] PASSED [ 74%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[f32 norm] PASSED [ 75%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[index projection] PASSED [ 76%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[q norm] PASSED [ 77%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_view_is_refused[router] PASSED [ 78%] test/unit/test_qwen38_flash_next_spec.py::test_an_offloaded_ngram_table_is_refused PASSED [ 79%] test/unit/test_qwen38_flash_next_spec.py::test_the_model_is_accepted PASSED [ 80%] test/unit/test_qwen38_flash_next_spec.py::test_each_unsupported_model_is_one_reason PASSED [ 81%] test/unit/test_qwen38_flash_next_spec.py::test_the_mtp_draft_views_are_checked PASSED [ 82%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[2560-6144] PASSED [ 83%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[16384-2560] PASSED [ 84%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[128-2560] PASSED [ 85%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle16_inverts_aiter_shuffle[640-2560] PASSED [ 86%] test/unit/test_qwen38_flash_next_layout.py::test_moe_padding_matches_fusedmoe PASSED [ 87%] test/unit/test_qwen38_flash_next_layout.py::test_unshuffle_moe_expert_inverts_fusedmoe_layout PASSED [ 88%] test/unit/test_qwen38_flash_next_layout.py::test_check_moe_params_accepts_fusedmoe_shapes PASSED [ 89%] test/unit/test_qwen38_flash_next_layout.py::test_pack_bf16_tiles_16_rows_by_64_k PASSED [ 90%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 91%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 92%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 93%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 94%] test/unit/test_contract.py::test_every_sglang_import_is_declared[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [ 95%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/check.py] PASSED [ 96%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/draft.py] PASSED [ 97%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/runner.py] PASSED [ 98%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/gfx950/weights.py] PASSED [ 99%] test/unit/test_contract.py::test_sglang_provides_what_the_plugin_reads[sgl_amd_mono/qwen38_flash_next/spec.py] PASSED [100%] =============================== warnings summary =============================== <frozen importlib._bootstrap>:488 <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute <frozen importlib._bootstrap>:488 <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead. from distutils.dep_util import newer, newer_group test/unit/test_qwen38_flash_next_spec.py: 14 warnings /opt/venv/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`. warnings.warn( -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =============== 102 passed, 34 deselected, 18 warnings in 8.10s ================ exit code: 0 ``` </details> <details> <summary>Kernel tests: 7 passed</summary> ```text date: 2026-10-29 09:50 UTC sgl-amd-mono: 9bdc57a Read the GDN and PLE states through SGLang's KV layer router SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 sglang from: /sgl-workspace/sglang-main-latest/python/sglang versions: torch 2.11.0+rocm10.0.0, HIP 7.15.26333, flydsl 0.4.0 GPU: 1 x AMD Instinct MI355X (gfx950) $ python -m pytest -v -p no:cacheprovider test/kernels/test_qwen38_flash_next_launches.py ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.0.3, pluggy-1.7.0 -- /opt/venv/bin/python rootdir: /sgl-workspace/sgl-amd-mono configfile: pyproject.toml plugins: platformdirs-4.13.5, anyio-4.15.1 collecting ... collected 7 items test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=1] PASSED [ 14%] test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=8] PASSED [ 28%] test/kernels/test_qwen38_flash_next_launches.py::test_verify_step[R=7+1 pad] PASSED [ 42%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-extend] PASSED [ 57%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=1-decode] PASSED [ 71%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-extend] PASSED [ 85%] test/kernels/test_qwen38_flash_next_launches.py::test_draft_step[R=8-decode] PASSED [100%] =============================== warnings summary =============================== ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 ../../opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15 /opt/venv/lib/python3.12/site-packages/Cython/Distutils/old_build_ext.py:15: DeprecationWarning: dep_util is Deprecated. Use functions from setuptools instead. from distutils.dep_util import newer, newer_group -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================== 7 passed, 2 warnings in 24.13s ======================== exit code: 0 ``` </details> <details> <summary>Synthetic benchmark C1 8K input / 1K output</summary> ```text date: 2026-10-29 15:31 UTC sgl-amd-mono: 24c72df3930a4c6269df157565b83f1b194387ac Turn off crash dumps in the Qwen3.8-Flash-Next fault test SGLang: 7df65d970df80235ecda9862ea67fa210bc6d7b9 [Perf] [AMD] diffusion: Optimize the Wan VAE decode for gfx1250 (#46722) (sgl-project/sglang main) aiter: ee5f7a8225c09be211f2e3a7da51dbd7a37230d7 Add fused MoE sort + quant for hidden size 2560 GPU: 1 x AMD Instinct MI355X (gfx950) MTP acceptance: measured (0 SGLANG_SIMULATE_ACC_* in the server env) # server $ SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 AITER_FLYDSL_FORCE=1 SGLANG_MAMBA_SSM_DTYPE=bfloat16 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \ python -m sglang.launch_server --model-path amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8 --port 31060 --tp 1 --attention-backend triton --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --cuda-graph-max-bs-decode 8 --max-running-requests 16 --enable-amd-monokernel # client $ python -m sglang.bench_serving --backend sglang --port 31060 --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 --num-prompts 20 --max-concurrency 1 --warmup-requests 2 --seed 1 /sgl-workspace/sglang-main-latest/python/sglang/bench_serving.py:13: FutureWarning: `sglang.bench_serving` is deprecated and will be removed in a future release; use `sglang.benchmark.serving` instead (e.g. `python -m sglang.benchmark.serving`). warnings.warn( benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model=None, served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) Waiting up to 60s for http://0.0.0.0:31060/v1/models to become ready... Server ready in 0.0s. Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=31060, ready_check_timeout_sec=60, dataset_name='random', dataset_path='', dataset_offset=0, agentic_max_turns=None, speed_bench_category=None, speed_bench_output_len=512, model='amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8', served_model_name=None, tokenizer=None, num_prompts=20, sharegpt_output_len=None, sharegpt_context_len=None, random_input_len=8192, random_output_len=1024, random_range_ratio=1.0, image_count=1, image_resolution='1080p', random_image_count=False, image_format='jpeg', image_content='random', request_rate=inf, use_trace_timestamps=False, max_concurrency=1, output_file=None, output_details=False, goodput=None, print_requests=False, disable_tqdm=False, disable_stream=False, return_logprob=False, top_logprobs_num=0, token_ids_logprob=None, logprob_start_len=-1, return_routed_experts=False, cache_report=False, seed=1, disable_ignore_eos=False, temperature=0.0, top_p=1.0, extra_request_body=None, apply_chat_template=False, profile=False, plot_throughput=False, profile_activities=['CPU', 'GPU'], profile_start_step=None, profile_steps=None, profile_num_steps=None, profile_by_stage=False, profile_stages=None, profile_output_dir=None, profile_prefix=None, lora_name=None, lora_request_distribution='uniform', lora_zipf_alpha=1.5, prompt_suffix='', pd_separated=False, profile_prefill_url=None, profile_decode_url=None, flush_cache=False, flush_cache_timeout=60.0, warmup_requests=2, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256, gsp_range_ratio=1.0, gsp_fast_prepare=False, gsp_send_routing_key=False, gsp_num_turns=1, gsp_ordered=False, gsp_group_distribution='uniform', gsp_zipf_alpha=None, mooncake_slowdown_factor=1.0, mooncake_num_rounds=1, mooncake_workload='conversation', fake_prefill=False, tag=None, header=None, rust_client=False, rust_worker_threads=None) #Input tokens: 163840 #Output tokens: 20480 Starting warmup with 2 sequences... Warmup completed with 2 sequences. Starting main benchmark run... 0%| | 0/20 [00:00<?, ?it/s] 5%|▌ | 1/20 [00:02<00:38, 2.03s/it] 10%|█ | 2/20 [00:03<00:35, 1.95s/it] 15%|█▌ | 3/20 [00:05<00:31, 1.85s/it] 20%|██ | 4/20 [00:11<00:55, 3.46s/it] 25%|██▌ | 5/20 [00:13<00:42, 2.87s/it] 30%|███ | 6/20 [00:15<00:38, 2.73s/it] 35%|███▌ | 7/20 [00:17<00:31, 2.41s/it] 40%|████ | 8/20 [00:19<00:27, 2.33s/it] 45%|████▌ | 9/20 [00:21<00:24, 2.23s/it] 50%|█████ | 10/20 [00:23<00:21, 2.16s/it] 55%|█████▌ | 11/20 [00:25<00:19, 2.14s/it] 60%|██████ | 12/20 [00:27<00:16, 2.02s/it] 65%|██████▌ | 13/20 [00:29<00:13, 1.94s/it] 70%|███████ | 14/20 [00:31<00:12, 2.02s/it] 75%|███████▌ | 15/20 [00:33<00:09, 1.94s/it] 80%|████████ | 16/20 [00:35<00:08, 2.00s/it] 85%|████████▌ | 17/20 [00:37<00:05, 1.93s/it] 90%|█████████ | 18/20 [00:38<00:03, 1.88s/it] 95%|█████████▌| 19/20 [00:40<00:01, 1.85s/it] 100%|██████████| 20/20 [00:42<00:00, 1.83s/it] 100%|██████████| 20/20 [00:42<00:00, 2.13s/it] ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 1 Successful requests: 20 Benchmark duration (s): 42.57 Total input tokens: 163840 Total input text tokens: 163840 Total generated tokens: 20480 Total generated tokens (retokenized): 20422 Request throughput (req/s): 0.47 Input token throughput (tok/s): 3849.17 Output token throughput (tok/s): 481.15 Peak output token throughput (tok/s): 686.00 Peak concurrent requests: 2 Total token throughput (tok/s): 4330.31 Concurrency: 1.00 Accept length: 3.57 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 2127.22 Median E2E Latency (ms): 1851.48 P90 E2E Latency (ms): 2236.08 P95 E2E Latency (ms): 2638.79 P99 E2E Latency (ms): 5264.40 ---------------Time to First Token---------------- Mean TTFT (ms): 456.02 Median TTFT (ms): 245.55 P90 TTFT (ms): 291.58 P95 TTFT (ms): 720.12 P99 TTFT (ms): 3441.69 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 1.63 Median TPOT (ms): 1.53 P90 TPOT (ms): 1.87 P95 TPOT (ms): 1.93 P99 TPOT (ms): 2.12 ---------------Inter-Token Latency---------------- Mean ITL (ms): 1.63 Median ITL (ms): 1.46 P90 ITL (ms): 1.92 P95 ITL (ms): 2.88 P99 ITL (ms): 5.79 Max ITL (ms): 6.31 ================================================== # monokernel lines from the server log [2026-10-29 15:31:03] monokernel qwen38_flash_next_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 2436.7 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: final mixer packed (2 tensors, 12.8 MiB), gdn layer packed (360 tens [2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: SGLang's QSA MTP index sharing off (index_share_for_mtp_iteration True -> False), so every draft decode selects its own blocks [2026-10-29 15:31:09] monokernel qwen38_flash_next MTP draft: draft extend steps of up to 8 x 4 rows and draft decode steps of up to 8 x 1 are routed by the spec's wrapper of the draft's forward, and one it cannot launch raises; each key logs 'MTP draft runs' at its first launch [2026-10-29 15:31:09] monokernel qwen38_flash_next_mtp_draft_hip on gfx950: widths [4, 8, 12, 16, 20, 24, 28, 32] built in 0.0 s, 543.0 MiB; wait deadline 60 s (a failed wait stops every rank with a GPU fault at address 0x0); weight copies: mtp packed (9 tensors, 66.0 MiB) [2026-10-29 15:31:24] monokernel qwen38_flash_next_hip on gfx950 runs target_verify steps of batch size [1, 2, 3, 4, 5, 6, 7, 8] x 4 tokens (8 of 8 captured decode graphs); every other step runs the standard path [2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 7, 7) [2026-10-29 15:31:25] monokernel qwen38_flash_next MTP draft runs ('DECODE', 8, 8) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 4, 4) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 5, 5) [2026-10-29 15:31:26] monokernel qwen38_flash_next MTP draft runs ('DECODE', 6, 6) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 1, 1) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 2, 2) [2026-10-29 15:31:27] monokernel qwen38_flash_next MTP draft runs ('DECODE', 3, 3) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 5, 20) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 6, 24) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 7, 28) [2026-10-29 15:31:29] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 8, 32) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 2, 8) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 3, 12) [2026-10-29 15:31:30] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 4, 16) [2026-10-29 15:31:31] monokernel qwen38_flash_next MTP draft runs ('DRAFT_EXTEND_V2', 1, 4) ``` </details> ## Speed Tests and Profiling 1 x MI355X, TP1, the same server flags with and without `--monokernel-backend amd`, CUDA graphs up to 8 requests. Verify and draft are both on the monokernel. Client: ```bash python -m sglang.bench_serving --backend sglang --dataset-name random \ --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 \ --num-prompts <max(10 x conc, 20)> --max-concurrency <conc> --warmup-requests 2 --seed 1 ``` | C | Off: median TPOT / output tok/s | On: median TPOT / output tok/s | Speedup: TPOT [95% CI] / tok/s | Accept length off / on | |---|---|---|---|---| | 1 | 4.04 ms / 220 | 1.60 ms / 517 | **2.53x** [2.28, 2.84] / 2.35x | 3.36 / 3.38 | | 2 | 4.58 ms / 400 | 2.17 ms / 812 | **2.12x** [1.89, 2.26] / 2.03x | 3.39 / 3.43 | | 4 | 4.99 ms / 669 | 3.21 ms / 1063 | **1.56x** [1.51, 1.71] / 1.59x | 3.39 / 3.40 | | 8 | 6.22 ms / 1088 | 5.86 ms / 1180 | **1.06x** [1.02, 1.09] / 1.08x | 3.38 / 3.40 | ## Checklist - [x] `pre-commit run --all-files` passes. - [x] Tests under `test/` cover the change. - [x] The SGLang commit tested against: upstream main 2229345dff (accuracy and speed), f3f0f7b68c (C=1 re-check), f3f0f7b68c and af21e2b989 (unit tests). - [x] Accuracy and speed results. - [x] `.github/CODEOWNERS`: `**/*qwen38_flash_next*` is already in upstream's CODEOWNERS.
kevin-mii
added a commit
to sgl-project/sgl-amd-mono
that referenced
this pull request
Nov 4, 2026
…104) - kimi_k3: the standalone MoE launch hands the shared stages the timeline keys (tl_on, tls) the router and route stamps read; check_moe raised KeyError at trace. - minimax, minimax_m31: the spec types, features_in_use and attention_backends_of from where SGLang defines them (sgl-project/sglang#45389), the old paths as the fallback; srt/monokernels is a shim SGLang plans to drop. - glm53, glm53_flash, qwen38_flash_next, muse_spark: SGLANG_READS names ForwardMode's and CaptureHiddenMode's methods in forward_mode.py, forward_batch_info.py before. - test_glm53_flash_spec: the JIT grouped top-k test checks the spec's declared env only; re-running SGLang's guard there broke on its signature, and SGLang tests the guard itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yichiche
added a commit
to yichiche/sgl-amd-mono
that referenced
this pull request
Nov 5, 2026
… path SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in sgl-project/sglang#45389. Read it there first and fall back to sglang.srt.monokernels, as the other specs on main do, so the contract test passes on both SGLang versions. Co-authored-by: Cursor <cursoragent@cursor.com>
yichiche
added a commit
to yichiche/sgl-amd-mono
that referenced
this pull request
Nov 5, 2026
… path SGLang moved the monokernel spec module to sglang.kernels.plugin_kernels.amd in sgl-project/sglang#45389. Read it there first and fall back to sglang.srt.monokernels, as the other specs on main do, so the contract test passes on both SGLang versions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Some kernels are specialized for one model, configuration and platform, and replace a model's whole step. They come from external libraries that load as SGLang plugins:
Today these plugins serve small-batch decode and verify steps. Nothing ties them to low latency or to "mega" fusion; prefill or high-throughput specializations would plug in the same way.
SGLang's side of each integration lived in
srt/megakernelsandsrt/monokernels. This PR splits that code at the kernel/runtime boundary:kernels/plugin_kernels/<hardware>/, recorded by a newKernelBackend.PLUGIN(as withkda_kernels/andcake_kernels/)srt/, assrt/kernel_plugins/<hardware>/The kernels themselves stay in their libraries.
Modifications
srt/{megakernels,monokernels}/spec.py→kernels/plugin_kernels/{nvidia,amd}/spec.py.guards.py+runtime.py(and, on NVIDIA,binding.py) →srt/kernel_plugins/<hardware>/binding.py.test/registered/unit/kernel_plugins/{nvidia,amd}/. They now patch the onebindingmodule.sglang.srt.megakernelsandsglang.srt.monokernelsare removed.SGL_NVIDIA_MEGA_COMMITbecomes6fd7ac3in both CUDA Dockerfiles. Main pins3333e4c([Fix] docker: Pin sgl-nvidia-mega to 3333e4c so the megakernel starts on main #46030), an ancestor of6fd7ac3.sgl-amd-monopin inpyproject_other.tomlbecomes05e0bb0.KernelBackend.PLUGIN. Adds the enum value and aforward_pluginhook onBaseFusedOp, placed after KDA/CAKE inDEFAULT_PRIORITY. The namespace test pairsplugin_kernels/with this backend.--monokernel-backend amd→--enable-amd-monokernel. The flag becomes a boolean next to--enable-nvidia-megakernel.--monokernel-backend amdstays as a deprecated spelling: it warns and sets the new flag. Any other value is rejected.POSITIONAL_FIELD_ORDER.--configYAML files. Previously the merger accepted onlystore/store_trueactions, which rejected the deprecated alias and, through the shared dest, the new key too.--flag value.monokernel-backend: amd,enable-amd-monokernel: true/false, and both CLI spellings. Before the fix, the three YAML cases raiseValueError.handle_amd_monokernel.kernels/README.md,plugin_kernels/README.mdand the kernel-organization skill describe the split.No runtime behavior changes: the default path is untouched, and the megakernel and monokernel code is the same code at new paths.
Not covered: the MSA plugin. The MiniMax-M3.1 cookbook (#45924) serves the MSA plugin with
--monokernel-backend msa. On this PR that value is rejected, and the plugin'ssglang.srt.monokernelsimports fail. That plugin is moving off the monokernel name separately.Accuracy Tests
All runs: 4x GB300 (sm_103),
lmsysorg/sglang:dev-cu13, plugins at the pinned commits (sgl-nvidia-mega6fd7ac3, sgl-amd-mono05e0bb0).The PR content was tested at
7311b92f02, which has main6e667c4310merged. The head has main3ea7e210d2merged instead; that merge changed nothing in this PR except resolving the NVIDIA pin conflict.Tests and plugin loading
unit/kernels/**,unit/server_args/**,unit/model_loader/**, the moved tests,test_dspark_draft_block,test_runtime_contextand others.6e667c4310and this PR fail the same single case.sglang.srt.megakernelsis gone, and both plugins register from the new locations.MegakernelSpec, and all six AMD specs resolve ongfx950.test/unit.05e0bb0: 5 failed, 124 passed on both main and this PR, the same five. They are GLM-5.3 spec cases.6fd7ac3: the Kimi K3 tests fail on both.--continue-on-collection-errors, that is 22 failed, 162 passed, 1 error, all Kimi K3.kimi_k3importssglang.srt.megakernelswith no fallback. [Feat] [NVIDIA] megakernels: Expose hybrid runtime and DSpark verification views #45406 has to move those views tosrt/kernel_plugins/nvidia/binding.pyanyway, andkimi_k3should then import them new-path-first, as DeepSeek-V4.1 does.The real megakernel, end to end
DeepSeek-V4.1-Flash, served with the plugin README's command (TP4, DSpark block 5,
--enable-nvidia-megakerneland the three required env vars):b453996b7db453996b7d(48b0063ff6)NVIDIA megakernel: active(target and DSpark draft specs, 4 ranks)draft proposal engagedduring graph captureOn main
6e667c4310, the same command fails with or without this PR, identically:--max-running-requests 8 --cuda-graph-max-bs-decode 8. SGLang's DeepSeek-V4 indexer, reached throughview.low_ratio_sources, hitsassert A.shape[:-1] == As.shape[:-1]in the fp8wq_bGEMM.A partial bisect puts that regression between
bb9e9d5632(good) andb588aa39e9(bad, #45921). The head after the latest main merge was not run on GPU.Not run: monokernel serving on gfx950. The AMD side was checked statically and on CPU only.
Speed Tests and Profiling
N/A: no kernel or forward change.
Checklist
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #40090470519
Latest PR Test (Extra): ❌ Run #40090470482
Latest PR Test (AMD ROCm 10): ❌ Run #40090470545