[YAMY1234]

Depends on sgl-project/sglang#47200 (runtime optimizations) and sgl-project/sglang#47201 (provider binding). The two SGLang PRs are independently based on main, build and test without each other, and may land in either order; this plugin uses both. The local-key provider needs the sharded-verification runtime support from sgl-project/sglang#47200 and the contracts and binder from sgl-project/sglang#47201, while unsupported or absent capabilities retain the documented stock fallbacks.

When the second SGLang PR is rebased after the first lands, or when both are composed before merge, the later PR must resolve eagle_draft_cuda_graph_runner.py, eagle_draft_extend_cuda_graph_runner.py, eagle_megakernel.py, eagle_worker_v2.py, and test_eagle_binding.py by retaining the runtime PR's relay/static-buffer behavior and the binding PR's always-present facade.

The combined SGLang stack has 28 Boolean children of SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION: 27 come from the shared main base, and sgl-project/sglang#47200 adds SGLANG_TRTL3_EP_DEFER_FINALIZE as the 28th. sgl-project/sglang#47201 declares no child switch; its SGLANG_NVIDIA_MEGAKERNEL_SLOTS setting is an independent tuple selector.

Motivation

Keep Qwen3.5-specific kernels, shape policy, and numerical contracts in sgl-nvidia-mega, while SGLang exposes only model-independent runtime plumbing and data views. Every path in this PR is explicitly selected, enabled only for supported capabilities, and able to fall back to the original SGLang implementation.

Modifications

Context-parallel attention slots

What and why

qwen3_5/attention/, qwen3_5/draft/attention.py, and qwen3_5/sm103/attention/q35_cp_{partial_attn,lite,fused}.py provide skv2_cp, draft_q1_cp, and draft_skv2_cp for target verification, draft-loop Q1, and draft extend. Target and draft variants build independently and agree on the requested CP configuration across ranks before allocation or early return.

Compared with main

Main executes the stock attention, partial-reduction, and merge sequence. The selected slots fuse partial attention and merge work, removing separate host launches, intermediate writes, and repeated workspace setup for supported shapes.

Current placement

The Qwen3.5 CP kernels and selection policy live in plugin-owned slots, leaving only generic views and binding contracts in SGLang. This makes the CP implementation independently replaceable without adding model checks to the runtime PR.

Switch and rollback

The three slots are opt-in through SGLANG_NVIDIA_MEGAKERNEL_SLOTS. Dtype, layout, shape, topology, workspace, or cross-rank agreement rejection removes the slot before execution and preserves stock attention.

Size and evidence

The principal move is +7,172/-31 production lines and +217/-10 test lines. Unit coverage exercises slot construction, independent target/draft capability, cross-rank agreement, and stock fallback; kernel coverage compares fused outputs with the native attention path.

MoE and deferred draft finalization

What and why

qwen3_5/moe/route.py, sm103/moe/shared_act_quant.py, draft/finalize.py, and draft/parts.py keep routed-MoE, shared-expert BF16 activation/FP8 quantization, and deferred MTP finalization next to the kernels and layouts they require. If draft_moe cannot consume an expert layout, it is removed from the plan so SGLang MoE runs rather than raising at startup.

Compared with main

Main executes routed/shared-expert operations and draft finalization as separate SGLang operations. Supported slots fuse shared activation/quantization and defer finalization, removing intermediate tensor traffic and host launches.

Current placement

Verification and MoE selection live in the plugin, scalar ModelOpt shared-expert scales are accepted, and unsupported draft_moe layouts return to SGLang MoE. The plugin does not enable a BF16 twin or a low-precision overlay by default.

Switch and rollback

draft_moe, draft_fin, and MoE/shared-expert slots are individually requested. Expert-layout, scale, dtype, shape, or backend rejection executes the corresponding SGLang operation.

Size and evidence

The policy move is +518/-25 production lines and +862/-6 test lines. Scalar-scale support is +6/-4 production and +83/-15 tests. Coverage includes unavailable-draft-MoE fallback, scalar/tensor scales, routed expert layouts, and shared-expert output parity.

GDN and ReplaySSM slots

What and why

The Qwen3.5 GDN plan exposes front, core, convolution, push, and verify slots and keeps high-batch ReplaySSM verification in the fused path through 896 rows.

Compared with main

Main executes the stock GDN/ReplaySSM sequence as separate operations. The selected plugin kernels fuse compatible state, convolution, and commit work, reducing intermediate launches and copies.

Current policy

The production GDN core policy supports the fused path through 896 rows. The companion runtime PR owns its generic one-launch ReplaySSM fallback; this plugin owns Qwen3.5-specific kernels and shape policy.

Switch and rollback

GDN slots are individually opt-in. Unsupported dtype, layout, row count, state view, or backend declines to the SGLang implementation.

Size and evidence

The high-batch policy change is +2/-2 production and +2/-1 tests. The mirror/output-contract update described below adds +35/-3 production and +144/-1 tests across GDN, attention, and low-M paths. Tests cover the 224/896 boundary and stock fallback above the supported bound.

hop_mid

What and why

The tuned policy explicitly lists hop_mid alongside hop, selecting the middle-shape communication/normalization kernel when requested.

Compared with main

Main has no plugin hop kernel and executes its existing communication and normalization sequence. hop_mid selects the tuned fused kernel for its supported shape.

Current policy

The explicit production slot list includes hop_mid; it is not made an implicit shipped default.

Switch and rollback

Request hop_mid in SGLANG_NVIDIA_MEGAKERNEL_SLOTS. Omitting it returns to the existing hop or stock selection.

Size and evidence

The policy documentation change is +6/-0; it reuses the already-tested kernel body. Focused slot-plan tests verify that explicit policy includes hop_mid without enabling unrelated slots.

Route kernels and policy

What and why

Qwen3.5 route projection, top-k, route-GEMM, and shared-expert policy remain next to their plugin kernels, including EP-size and engine-width guards.

Compared with main

Main launches router projection, top-k, and shared-expert preparation as separate operations. Supported plugin slots fuse or specialize those stages and reduce launch and intermediate-memory overhead.

Current policy

Qwen-specific route policy lives in the plugin rather than SGLang. route_lin remains available as an experimental plugin slot but is not requested by the production slot list; this avoids silently adding a verify kernel that is not part of the validated production policy.

Switch and rollback

Route slots are explicit. EP size, engine width, dtype, layout, or shape rejection uses stock routing. route_lin is default-off.

Size and evidence

Direct route implementation and policy contribute +85/-24 production and +120/-13 direct kernel-test lines; shared policy/facade tests are counted with the MoE move above. Tests cover EP guards, engine widths, top-k output, and stock routing.

Sharded verification local-key provider

What and why

qwen3_5/runtime/hooks.py and runtime/executor.py attach a Qwen3.5 local-key provider for supported verification while leaving reduction, collectives, write-back, acceptance, forced acceptance, and stock sampling in SGLang.

Compared with main

Main gathers full vocabulary logits before sampling. For an eligible request, the provider produces compact rank-local keys so the dependent SGLang runtime can avoid that gather.

Current contract

The binder permits only local-key production, so the plugin cannot change collective ownership or acceptance policy.

Switch and rollback

sharded_gumbel is explicit. Seeded or unsupported sampling, penalties, watermarking, logprobs, or provider rejection use the stock sampler with unchanged RNG ordering.

Size and evidence

Provider attachment is +263/-1 production and +562/-2 tests. Tests cover padded vocabulary tails, greedy delegation, seeded no-RNG fallback, unseeded RNG ordering, and provider decline.

FP8 / ModelOpt output dtype contract

What and why

The plugin carries the layer's original BF16/FP16 activation dtype through its mirrors and uses it when materializing prequantized E4M3 ModelOpt/FlashInfer outputs.

Compared with main

Main returns the model layer's declared output dtype. The plugin uses the original layer dtype for _out_proj, so its output contract matches main rather than inheriting the GDN recurrence dtype.

Current contract

orig_dtype is present in all relevant mirrors, preserving the declared output dtype when recurrence state and layer output dtypes differ. Output dtypes outside explicit BF16/FP16 support are rejected.

Switch and rollback

This is a correctness contract on the FP8/ModelOpt slot, not an independent performance switch. Slot validation rejects unsupported dtypes and uses the native contract.

Size and evidence

The combined mirror/output fix is +35/-3 production and +144/-1 tests. Coverage checks BF16 and FP16 model outputs, recurrence/output dtype disagreement, mirror signatures, and unsupported dtype rejection.

TP2 / DEP8 topology filtering

What and why

The plan derives the actual attention tensor-parallel size and offers CP-only slots only when the attention backend built a compatible CP topology.

Compared with main

Main never attempts plugin CP construction. The Qwen3.5 plugin itself supports attention TP equal to model TP at TP4 and TP8; attention TP2 on model TP4 and DEP8 are DP-attention topologies and are rejected clearly by the plugin configuration gate at startup.

Current behavior

Before that top-level gate, the CP capability filter removes skv2_cp, draft_q1_cp, and draft_skv2_cp on an unsupported attention topology, returning those CP slots to stock and preventing incompatible CP construction. TP4 and TP8 retain their supported CP path. This CP-slot fallback is not a claim of end-to-end plugin support for DP attention.

Switch and rollback

This is an automatic capability check on skv2_cp, draft_q1_cp, and draft_skv2_cp; no user action is required.

Size and evidence

The topology filter is +81/-17 production and +163/-2 tests. Coverage models attention TP independently from model TP and checks TP2/DEP8 rejection plus supported TP4/TP8 construction.

cp_multi fused batch limit

What and why

The fused CP selector uses the validated target/draft limits of 32/32. The target route also stops offering fused CP above batch 32, so larger batches use the existing SKV2/stock route instead of constructing an unvalidated cp_multi plan.

Compared with main

Main always uses its native attention selection and therefore has no plugin batch-limit mismatch. The plugin eligibility check likewise returns unsupported batch sizes to the existing route.

Current policy

The validated 32/32 fused limit is enforced at both the selector and route-offer boundaries.

Switch and rollback

There is no new environment variable. The limit is part of the CP slot capability check. Batches above 32 do not offer target fused CP and automatically use the existing route.

Size and evidence

The selector change is +5/-5 production and +18/-0 tests. The route-offer check is +4/-1 production and +15/-0 tests. Boundary tests cover batches 32 and 33 and verify that the fallback route remains available.

Accuracy Tests

• The focused Qwen3.5 contract suite records 118 selected tests, with 62 unrelated tests deselected. It covers CP selection and topology, route guards, GDN bounds, MoE fallback, dtype contracts, local-key behavior, and stock fallback.
• A no-simulation serving run answered GSM10 10/10, and a long-context replay completed 53/53 requests.
• Layer-by-layer localization found no numerical difference before the ReplaySSM committed state. At that first difference, 233 of 5,898,240 BF16 elements differed, all within atol=1e-6, rtol=2^-7; this state is tolerance-equivalent, not bitwise-equivalent.
• In a real two-request TEP8 verify batch, all eight ranks compared 434,560 plugin-slot logits and 57,344 hidden-state values with the stock computation and found zero bitwise differences.
• DEP8 with only SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION enabled (runtime only; the Qwen3.5 plugin disabled) answered GSM10 10/10.
• The sharded sampled path is distribution-equivalent, not RNG-stream or bitwise equivalent. Seeded requests deliberately use the stock sampler.

GSM8K on the current head, no draft-acceptance simulation, seed 1234 (strict / flexible accuracy):

Topology	1319 questions, greedy	250 questions, greedy	250 questions, sampled T=1.0
TP8	0.978 / 0.977	0.988 / 0.988	0.972 / 0.972
TEP8	0.980 / 0.979	0.980 / 0.980	0.968 / 0.968

No empty responses in any run. TP8 also passed a 53-request long-context replay (three attempts, all 53/53, no CUDA errors).

Speed Tests and Profiling

Static decode uses identical requests and warmed, unprofiled rounds. P50/P90 TPS/user is derived from round latency; autotune selections were frozen for all measurements in both columns.

Topology	Batch	main P50/P90 TPS/user	this PR P50/P90 TPS/user	change
TP8	1	773.39 / 753.59	1561.443 / 1553.168	+101.9% / +106.1%
TP8	8	492.46 / 488.33	657.33 / 650.93	+33.5% / +33.3%
TP8	16	410.25 / 406.68	553.19 / 546.35	+34.8% / +34.3%
TP8	64	260.23 / 258.97	324.28 / 323.57	+24.6% / +24.9%
TEP8	32	359.99 / 357.98	431.472 / 428.328	+19.9% / +19.7%
TEP8	64	283.69 / 282.05	351.007 / 348.592	+23.7% / +23.6%
TEP8	128	201.43 / 200.51	258.757 / 257.732	+28.5% / +28.5%

AgentX trace replay, 3600 s window, TEP8 prefill/decode disaggregation at concurrency 272, same request set and the same fixed draft acceptance length in both columns:

Metric	main	this PR	change
TPS/user P50 / P90	85.48 / 81.31	129.37 / 112.92	+51.3% / +38.9%
TTFT P50 / P90			-80.1% / -60.0%
E2E latency P50 / P90			-47.2% / -36.3%
Total tok/s/GPU	68946	80774	+17.2%

TP8 decode with prefill/decode disaggregation at concurrency 16, same settings:

Metric	main	this PR	change
TPS/user P50 / P90	628.25 / 521.07	1163.77 / 850.48	+85.2% / +63.2%
TTFT P50 / P90			-1.9% / -33.1%
E2E latency P50 / P90			-31.8% / -43.4%
Total tok/s/GPU	8671	8950	+3.2%

TP8 decode with prefill/decode disaggregation at concurrency 128 (1 prefill + 4 decode nodes), same settings:

Metric	main	this PR	change
TPS/user P50 / P90	369.30 / 288.71	640.99 / 525.06	+73.6% / +81.9%
TTFT P50 / P90 (s)	0.820 / 3.458	0.860 / 3.488	+4.9% / +0.9%
E2E latency P50 / P90 (s)	2.542 / 7.892	1.900 / 6.135	-25.3% / -22.3%
Total tok/s/GPU	32933	33980	+3.2%

TEP8 prefill/decode disaggregation at concurrency 448, same settings:

Metric	main	this PR	change
TPS/user P50 / P90	85.72 / 81.26	113.30 / 106.15	+32.2% / +30.6%
TTFT P50 / P90 (s)	25.93 / 30.47	14.45 / 17.87	-44.3% / -41.3%
E2E latency P50 / P90 (s)	31.45 / 49.93	18.69 / 33.26	-40.6% / -33.4%
Total tok/s/GPU	66230	87126	+31.5%

All columns of all AgentX tables completed the full window with zero errored requests.

TP8 B1 was measured before the latest plugin commit, which does not affect the single-request path; all other rows were measured on the current head. The TEP8 rows were measured on the current combined tree.

Known limitations

• Corrected AgentX measurements have not completed.
• Rebasing one dependent SGLang PR after the other lands, or stacking them before merge, requires the documented five-file resolution.
• The existing Kimi K3 provider in this repository still imports sglang.srt.megakernels, which upstream SGLang removed in [Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration sglang#45389. Its unit tests therefore cannot be collected against current SGLang main. This is independent of the Qwen3.5 changes in this PR.

Checklist

• pre-commit run --all-files passes; focused formatting, lint, and compilation checks pass.
• Tests under test/ cover changed kernels, policies, topology checks, and stock fallbacks.
• The dependent SGLang runtime and binding PRs are named above.
• Add matching main static measurements, serving accuracy (GSM8K), and AgentX results.
• CODEOWNERS changes are not required; this extends the existing Qwen3.5 plugin.

🤖 Generated with Claude Code

Update (2026-11-02): rebased on main c396f2ab, eight new optimizations, one per commit

The 43 earlier commits are unchanged in content (rebased; a separate style commit 6b1f5df applies the repository's isort/ruff formatting so every commit passes pre-commit run --all-files). Eight optimizations are added, each in its own commit. Tests for the new commits were trimmed to the core cases: on/off numerical or bitwise equality against the SGLang path, fallback when unsupported, and topology selection.

Measured on Qwen3.5 (GB300). AgentX = trace replay, one 1200 s run per arm, TPS/user P50 / P90, run-to-run noise ≈2% (TP8) / ≈1.4% (TEP8) at P50. Static = fixed batch decode, step time.

#	Commit	Change	Production / tests	Measured effect
K1	ec40427	Wider GDN verify head layouts with per-kernel fallback; DP-attention padding; gdn_prefill_split_conv slot for PD prefill (TP1/PP4)	+540/−22 · +324/−34	Not measured separately (see K1–K5 total). PP4 prefill GSM8K 1319: 97.73% vs 97.95% off (−0.23 pp, within the 1.5 pp gate)
K2	67cbe30	Multi-request attention front for TP2/TP4 as well as TP8	+48/−29 · +79/−33	See K1–K5 total
K3	92950d6	Router-only slots independent of the expert engine; empty DP ranks join expert collectives	+23/−10 · +19/−9	See K1–K5 total
K4	a39fab2	BF16 draft on measured TP4 layouts	+33/−18 · +99/−17	See K1–K5 total
K5	1046f06	Explicit topology plans (aggregated / PD prefill / PD decode) for slot selection	+398/−78 · +173/−24	See K1–K5 total
K1–K5	—	Combined	+1042/−157 · +694/−117	AgentX TP8 PD c16 P50 +0.75% / P90 −2.4% and TEP8 PD c130 +0.03% / +1.0% (within noise on TP8/TEP8). Static TP2 B1/B8 +13.8% / +7.3%, TP4 B1/B8 ≈ +15% / +10% (together with sglang#47201's prefill view)
CP-bal	fef99c9	Fused CP verify assigns CTAs by KV length instead of per request; verify and Q1 limits raised to batch 64	+14/−10 · +5/−9	Plugin-only A/B: TEP8 PD c130 +9.6% / +58.2%; TP8 PD c128 +2.2% / +6.1%; TEP8 PD c272 +2.0% / +1.3%; TP8 PD c16 −1.6% / +0.3% (noise). GSM8K 1319: 1289 vs 1287
cap128	f037dfd	Keep verify batches 65–128 on the two-slot CP plan (Q1 stays 64)	+3/−2 · +4/−4	TEP8 PD c272 +117% / +60% (TTFT P90 −42%, E2E P90 −44%). Static TEP8 with 8k-token contexts: batch 128 is 4.5% slower (+795 µs verify); batches 1–64 within ±0.3%. A KV-length gate follow-up is in progress. GSM8K 1319: 1290 vs 1289
route_lin	4da6382	In the TP8 plans (aggregated and PD decode), compute router logits before the all-reduce when all dependencies are selected	+40/−7 · +46/−0	Static TP8 batch 1: −146 µs/step (−4.9%). AgentX TP8 PD c16: P50 +0.9% / P90 −0.5% (noise; batch-1 decode step 3206 → 3016 µs, batches 2–4 unchanged). Router logits RMS error vs FP64: 0.989× stock. GSM8K 1319: 97.42% vs 97.80% (−0.38 pp, within the gate)

New-commit totals (first eight): production +1099, tests +749 (before trimming 1652).

Validation: all eight new commits pass the full test/unit suite on two SGLang runtimes with no new failures relative to main; every commit passes pre-commit. GPU kernel tests that were reshaped during trimming were checked for syntax and format only.

Appended after the first update

#	Commit	Change	Production / tests	Measured effect
wide-bal	33aa1ec	TP8 plans that select the wide MoE engines also select the existing balanced row-block schedule (mkm_wide_bal; batch 19–73 verify); other topologies unchanged	+10/−4 · +15/−0	Static TP8 (CUDA graphs, same nodes): batch 32 step time −3.06%, batch 64 −2.35% (wide MoE kernel −7.4% / −5.3% per call); other batches unchanged within noise. Bitwise identical to the unbalanced schedule (16/16 cases); GSM8K 10/10
K6	b83045b	In the default plan, an unavailable slot falls back to stock with an INFO log instead of raising; an explicitly requested unavailable slot still raises with a message naming the slot, the condition and the fix	+271/−99 · +132/−0	Robustness only (no kernel change). Explicit recipe slot lists for TP2 / TP4 / TP8 / TEP8 / DEP8 / PD prefill raise nothing and select the same slots as before; build-time checks only, no per-step synchronization added

| hooks member | 13ef503 | Declare speculative_runtime_hooks = None on the DeepSeek-V4.1, Kimi-K3 and Qwen3.5 draft built classes, so SGLang can read it as a declared member (sglang#47201) | +3/−0 · 0 | No runtime change; should land with or before sglang#47201 |

| K7 | e7ed0d7 | Refusals that depend on the weights or runtime state (draft MoE weight layout, MTP depth, context length, LM-head / embedding sharding, missing kernel dependencies, multicast mapping, CP-lite head layout) log INFO and keep the stock path even for explicitly requested slots; only configuration contradictions (CP slot with --page-size ≠ 64, a per-request row count with no covered width) still raise when explicit. Two draft-exchange build refusals that previously escaped are now caught and fall back to stock | +29/−26 · +59/−11 | Fixes a startup failure introduced by K6 on real weights (explicit draft_moe with the FlashInfer TRT-LLM MoE layout). Explicit recipe slot lists for TP2 / TP4 / TP8 / TEP8 / DEP8 / PD prefill / PD decode raise nothing and select the same slots as before |
| fp4 MoE tactics | 4190d31 | Fix the TEP8 routed body's trtllm_fp4_block_scale_moe tactic per token bucket (FlashInfer 0.7.1, SM103, the routed experts' MoERunner signature); any other version, architecture, signature or bucket asks the autotuner as before, and the log names each width's source. SGLANG_NVIDIA_MEGAKERNEL_MOE_TACTICS switches it; unset, it follows SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION | +156/−5 · +56/−0 | Same-node back-to-back AgentX pairs, 1200 s each, all other ops tuned online in both arms: TEP8 PD c130 P50 +5.0% / P90 +1.5%; TEP8 PD c272 P50 +4.8% / P90 +4.5% and, in a repeat on other nodes with the order reversed, P50 +5.2% / P90 +2.0% (TTFT P90 −14%). TP8 points do not use the table |
| shared act quant | a1a418e | On TEP8 (TP8, attention DP 1, MoE EP > 1), when every layer's ordinary shared expert is eligible, fuse its SiLU-and-mul and static FP8 input quantization for down_proj into the plugin's existing shared_act_quant kernel, removing one BF16 activation write and read. The shared gate/up projection and gate stay stock; no new env, slot, kernel or weight copy. TP8, DEP8 and ineligible layer groups keep the stock path | +42/−6 · +75/−1 | Static decode, same nodes and frozen tuning in both arms, no profiler, AL 4.8: per-round time B64 −0.65% (mean) / −0.63% (median), B128 −0.28% (median; the mean change is within window noise). FP8 bytes match and BF16 outputs are bitwise identical to stock; GSM8K-10 10/10 |
| GDN 3-wave window | f60f990 | The GDN core also caps registers at 168 (three resident CTAs per SM) when the CTA count is above two waves but fits in three (2·SMs < CTAs ≤ 3·SMs; batch 39–57 at TEP8 on 152 SMs), reusing the existing 168-register variant; the rule above four waves is unchanged | +3/−3 · +19/−0 | Static decode, same nodes and frozen tuning in both arms, no profiler, AL 4.8: per-round time B40 −1.4% (mean) / −0.9% (median), B48 −1.0% / −1.1%, B54 −1.2% / −1.2%; B64 unchanged. GSM8K-10 10/10 |
| GDN mirror audit | 5edd8e9 | sgl-project/sglang#48004 reserves state-staging row 0 as a padding sink in GDNAttnBackend.forward_extend. The GDN prefill slot rejects strided conv/temporal pools before any write and keeps the native pool indices, and target verify never takes the new staging, so both mirrors accept the new digest; tests pin the fallback, the index handling and that verify never stages | +3/−2 · +219/−0 | No runtime or numeric change. Removes the two mirror-digest failures against main after sglang#48004 |

All seven commits pass pre-commit and the full test/unit suite with no new failures.

[YAMY1234]

All utests passed after sgl-project/sglang#47200 merge