Fix DSV4 DSpark shared expert loading - #33312
Merged
Merged
Conversation
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Collaborator
Author
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…utation, sgl-project#471/sgl-project#33407/sgl-project#33312/sgl-project#33098/sgl-project#33271 ports, sgl-project#481 PP fixes Eight commits, largest bundle this session: - sgl-project#479 (two commits): the GGUF-MoE offload coverage test and its refusal message named MXFP4/type 39 as THE example of a ggml type with no MoE kernel -- sgl-project#398 made that false (type 39 has a full native kernel set on a wheel that carries it). Traced the ACTIVE DSV4-Flash MXFP4 path end to end and confirmed it is native, not an untraced fallback; the refusal message now names the layer's own declared types instead of a stale hardcoded example, so it cannot go stale again the same way. docs/dev/NOTE_479_mxfp4_active_driver_path.md has the trace. - sgl-project#471: ports upstream sglang#32320 -- mask the SM120 SWA page-split to touched pages only, instead of the full page range. docs/dev/TICKET_471_masked_page_split.md. - sgl-project#33407 port: SM120 sparse-MLA buckets the topk width instead of using it unbucketed, falling back to Triton outside the bucketed range. - sgl-project#33312 port: DSpark draft head resolves and loads fused shared experts. - sgl-project#33098 port: DSpark draft fills the EP token-accounting fields. - sgl-project#481: three pipeline-parallel defects from the sgl-project#445 window fixed in server_args.py/resident_fraction.py/expert_stats.py. - sgl-project#33271 port: auto-round GPTQ MoE completes the MoeWNA16 delegation. FEATURE_CATALOG.md §8 conflict-risk as flagged, but no actual conflict -- this branch's base (a5eff26) is now several merges behind (sgl-project#520, sgl-project#523, design-453 landed since), yet none of those touched §8, so git's 3-way merge combined everything automatically. All 17 sections and all 22 previously-preserved additive paragraphs verified present. **Explicit verification requested and confirmed**: the two "ancient pre-existing" failures in tests/moe_offload/test_gguf_moe_offload.py (test_materialize_declines_uncovered_ggml_type, test_guard_still_refuses_an_unstaged_gguf_layer) that this session bisected back to before this session even began -- sgl-project#479's fix to expert_offload.py and the type-39 assumption in gguf.py directly addresses their root cause. Full file run: 20 passed, 0 failed, both confirmed GREEN now. One test-isolation quirk found and run to ground, not a code bug: the three flash_mla test files this branch adds/touches (test_flash_mla_backends.py, test_flash_mla_page_split_mask_471.py, test_flash_mla_sm120_topk_buckets.py) interfere with each other when run in the same pytest process -- a shared Triton-decorated function object resolves to JITFunction instead of the expected InterpretedFunction once another file in the same process has already imported flash_mla_sm120.py under real (non-interpret) conditions. Verified this is NOT a merge regression: reproduced identically on the unmerged branch tip in isolation (fresh archive, no other code involved), and every one of the 30 tests across the three files passes cleanly when its file is run alone. Branch-inherent test-ordering fragility, not in scope for this merge to fix. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - tests/moe_offload/test_gguf_moe_offload.py alone: 20 passed (was 18 passed/2 failed before this merge, per this session's own bisection). - The eight new/touched test files, run individually where the isolation quirk above requires it: all pass in full (dspark 442: 18; auto_round: 17; dspark shared-expert 33312: 21; gguf mxfp4 479: 18; pp defects 481: 26; flash_mla backends/topk-buckets/page-split-mask: 10/25/10 each alone). - test/registered/unit/quantization/ + unit/models/ + spec/dspark/: 28 failed both before and after (diffed by test ID, not count -- the pre-merge archive showed 35 because 7 of its failures were an artifact of an incomplete git-archive extraction missing sgl-kernel/, confirmed by re-checking those 7 IDs against the working tree directly). The remaining 28 are dspark_kernel_parity's CUDA-only subtests plus test_gguf_dequant_scratch_budget.py, both pre-named as pre-existing. - test/registered/unit/layers/ + tests/moe_offload/: 3 failed (the pre-named test_deterministic_fp8_gemm.py x2 + test_flashinfer_workspace_zeroing.py x1), 1168 passed -- confirms test_gguf_moe_offload.py's contribution to this count is now 0 failures. - test/registered/unit/distributed/ (full, server_args.py touched): 24 failed, byte-identical established baseline. - ruff check --select=F401,F821,UP037: 15 findings, all in expert_offload.py, all confirmed pre-existing (same set already verified at cc5a92e in the earlier sgl-project#494 merge) and outside this branch's two diff hunks (2112-2138, 2616-2648) -- auto_round.py and every other touched file clean. - codespell: clean on all 21 touched/added files.
mmangkad
force-pushed
the
mmangkad/fix-dsv4-dspark-shared-experts
branch
from
August 9, 2026 15:04
8cc1c91 to
7366fdc
Compare
Collaborator
Author
|
Reopening because #33889's per-runner fusion refactor reintroduced this issue. DSpark's standalone entry class was missing DSV4's fusion gate, so it used the wrong shared-expert layout and skipped the bundled weights. This updates the fix for the current per-runner design. |
Collaborator
Author
|
Adding |
ch-wan
approved these changes
Aug 10, 2026
Merged
3 tasks
ch-wan
added a commit
that referenced
this pull request
Aug 11, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
5 tasks
ch-wan
added a commit
that referenced
this pull request
Aug 11, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
vstone-w
pushed a commit
to ClownBin/sglang
that referenced
this pull request
Aug 12, 2026
ch-wan
added a commit
that referenced
this pull request
Aug 12, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
ch-wan
added a commit
that referenced
this pull request
Aug 13, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
ch-wan
added a commit
that referenced
this pull request
Aug 14, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
ch-wan
added a commit
that referenced
this pull request
Aug 15, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
ch-wan
added a commit
that referenced
this pull request
Aug 15, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
ch-wan
added a commit
that referenced
this pull request
Aug 15, 2026
Two problems in the same family as #33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with #33312 in place it passes.
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 16, 2026
Two problems in the same family as sgl-project#33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until sgl-project#33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with sgl-project#33312 in place it passes.
hanwlax
pushed a commit
to hanwlax/sglang
that referenced
this pull request
Aug 28, 2026
Two problems in the same family as sgl-project#33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until sgl-project#33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with sgl-project#33312 in place it passes.
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
Two problems in the same family as sgl-project#33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until sgl-project#33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with sgl-project#33312 in place it passes.
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
Atituiset
pushed a commit
to Atituiset/sglang
that referenced
this pull request
Sep 10, 2026
Two problems in the same family as sgl-project#33312 (a per-runner decision that one participant answered differently), one fixed here and one guarded. **The linear-attn kernel backends were process-wide.** `attn_backend_wrapper` rebuilt a module-level dict once per runner, from the handed record plus a local `prefill_default`. Two things follow, and both are wrong: - **A draft could not hold a different choice than its target.** Only the runner whose model is GDN gets the SM100 FlashInfer prefill default; the operator's explicit flag belongs to the launch. The full-attention backends already model this correctly -- the runner stamps `prefill_attention_backend_str` / `decode_attention_backend_str` and its backend objects are built from the stamp. Linear attn had no stamp at all. - **The second rebuild replaced the first one's choice.** The default was also recorded into the process-wide config, which the record does not see, so a runner rebuilding without a default of its own resolved `prefill` back to the base backend -- silently swapping the kernel the earlier runner selected. Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`. `resolve_linear_attn_backends(prefill_default=None)` returns a frozen `LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba` leaves; the wrapper stamps it as `runner.linear_attn_backends` before building the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it off the runner they are built for. Each already took `model_runner` and cached the result on itself, so the value now simply comes from the right place. A backend built outside that path has no stamp and raises on the attribute, the way the full-attention strings do -- no silent fallback to hide the wiring mistake. The recording goes away with it. A per-runner choice in the process-wide config has no meaning the second runner can read correctly: the leaf is how the gate asks "did the operator name a backend", so a recorded default reads back as an operator flag and the next runner declines its own. The leaf now keeps meaning what was asked for at launch, and the effective choice lives in the stamp (and in the log line the gate already emits). Precedence is unchanged: the resolver takes the default as an argument and an explicit `--linear-attn-prefill-backend` wins over it, with the gate declining early so it neither probes the device nor logs. **A draft entry class must answer the loader exactly when its target does.** The loader asks the entry class it instantiates for the shared-experts-fusion decision, and a draft is its own entry class. When the target family carries auto-disable conditions and the draft's class does not expose them, the loader installs one decision for each and the draft's weights are laid out for the wrong one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert tensors until sgl-project#33312 gave it the gate, costing accept length 5.60 -> 2.05. `test_fusion_gate_coverage.py` walks the same registry but asks whether an entry class *touches* the decision -- reads the flag, names a gated class. That catches a class once it already consumes the decision; it could not catch one that should consume it and does not, which is what the DSpark class looked like (it built the family's *layer* classes, so the flag reader lived in another module and its own source named no gated class). `test_draft_entry_hook_parity.py` asks the invariant directly: presence parity between a draft entry class and the target it is named after. Identity is deliberately not required -- the Qwen3.5 MTP delegates with adapted arguments (unwrapping `text_config`, using the MTP quantization config), which is right -- and weight-name maps are out of scope, since a draft's checkpoint has its own names. Reverse-verified against the original defect: with the DSpark gate removed the case names the pair and the side that is missing it; with sgl-project#33312 in place it passes.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#33889 moved shared-expert fusion to a per-runner decision installed from each model's entry class. The DSV4 target exposes that gate, but the standalone DSpark entry class did not, so the draft used the wrong layout and skipped its bundled shared-expert tensors.
This makes DSpark use the DSV4 gate and retain the resolved layout when it is built. Shared experts stay separate by default, while explicitly forced fusion remains supported and maps those weights into the fused expert slot.
Before the fix, every shared-expert weight and scale across all three DSpark stages was reported as unexpected:
With the fix:
All three stages now load with no unexpected-weight warnings. Focused coverage also verifies the DSpark gate, the retained fusion decision, and explicitly fused shared-expert loading.
Results
DSpark accept length went from
2.05to5.60and output throughput from237.33 tok/sto589.95 tok/s. GSM8K was96.59%before and97.12%after.Before the fix
With the fix
CI States
Latest PR Test (Base): ✅ Run #31320326827
Latest PR Test (Extra): ❌ Run #31320326742