[weireweire]

Motivation

DecodeCudaGraphRunner._init_replay_attn_metadata only routes the attention-metadata prep through the metadata glue graph when raw_bs == bs. Every other replay runs the eager init_forward_metadata_out_graph path, which is dozens of small host-dispatched ops. Two common cases hit that path:

• The DSpark draft runner is dp-local, so it replays at the capture bucket of its own batch size. Any rank whose batch is not exactly a capture size takes the eager path.
• Target verify replays padded to the DP max take it as well.

On DSV4.1-Flash GB300 disagg DEP16 (c2884, 40 running per rank), the draft G.attn_meta costs 0.56-0.62 ms on padded ranks vs 0.06 ms glued. That cost sits on the host path between the DP agreement and the draft graph launch.

Related PRs

One of four DSV4.1 attn-DP decode host-path changes that cut the host work between the DP agreement and the draft graph launch. They are independent in code and were measured together: #47676 (padded metadata glue), #47677 (persistent DP-MoE counts), #47678 (bulk decode-prep fill) and #47685 (KV page top-up, measured on top of the other three).

Modifications

The glue route now also takes padded replays (0 < raw_bs < bs). There is no separate switch: it engages wherever the backend declared the DSV4 draft or target-verify glue (the audited host-free prep) and the runner is not in ragged verify mode, and falls back to the eager prep otherwise. Non-DSV4 backends, the generic SGLANG_ENABLE_METADATA_GLUE_GRAPH glue, TBO and LoRA keep the padding-free route.

Behavior change: the DSpark draft glue is on by default (static ragged verify mode), so padded draft replays now take the glue graph without any flag. The target-verify glue still follows SGLANG_OPT_DSV41_TARGET_VERIFY_METADATA_GLUE. The DEP16 / DEP32 validation below ran without SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION on decode, i.e. in exactly this default configuration; the TP4 c1 runs had it on.

Why the padded prep is the same as the eager padded prep: both read the same req_pool_indices / seq_lens slots, whose tails fill_from resets to the registry pad values (0 and seq_len_fill_value). The glue view reads the padded out_cache_loc slot (num_tokens is passed for every glued replay). fill_from zeroed that slot's tail, and the eager prep zero-pads the raw tensor to the same length. The declared prep reads nothing else that depends on raw_bs, so the captured prep for key bs serves padded and unpadded replays alike. IDLE replays stay eager. The contract comment in metadata_glue_graph.py is updated to cover the padded route.

All runs: GB300, DSV4.1-Flash, on a 2026-10-30 nightly.

Accuracy Tests

• In-server bitwise check (diagnostic overlay, not part of this PR): for the first 40 captured padded draft replays per rank, the check snapshots the glued metadata, runs the eager padded prep on the same inputs, and compares every metadata tensor (page_table, raw_out_loc, seq_lens_casual, positions_casual, swa_page_indices, swa_topk_lengths). Result on DEP16: 640 checks (16 ranks, raw_bs 2-13 on bs=14), 0 mismatches.
• The padded target-verify route is covered by the same argument, but none of the runs below exercised it: it needs SGLANG_OPT_DSV41_TARGET_VERIFY_METADATA_GLUE, which was off in the DEP runs, and TP4 c1 never pads.
• GSM8K, 1P1D DEP16 with real DSpark acceptance, together with the two sibling host changes ([Perf] [NVIDIA] dspark: Stage draft DP-MoE token counts through persistent buffers #47677, [Perf] [NVIDIA] dflash: Fill decode-prep length buffers in bulk #47678): 96.82% (1319 examples, 0% truncated or errors) vs 96.89% without them. Accept length 3.50 vs 3.49-3.51.
• TP4 c1 agg (mega flag on, all three host changes): T=1 GSM8K 97.33% vs 97.33%, long-context needle retrieval at 20K-160K tokens 48/48 vs 48/48 (greedy) and 24/24 vs 24/24 (T=1), accept length 3.591 vs 3.598.

Speed Tests and Profiling

	DEP16 nsys, before	DEP16 nsys, after
draft G.attn_meta, median over ranks	170 us	58 us
draft G.attn_meta, worst rank	620 us	64 us

The following numbers are combined with the two sibling host changes (#47677 DSpark persistent DP-MoE counts, #47678 DFlash bulk decode prep). Same cluster and recipe, but the two arms ran on different days:

• DEP32 c5768 (10P1D): decode step p50 13.9 ms vs 15.2 ms. This is the robust number.
• Same runs: steady-window per-GPU throughput (prompt + completion tokens, mean of minutes 8-20 of the aiperf run, same x0.96 normalisation for both arms) 660.6k vs 664.4k tok/s/GPU at p90 interactivity 219.8 vs 200.8 tok/s/user (+9.5%). Same-day controls of this recipe drift by about 3% on interactivity, so read the interactivity gain as indicative, not exact.
• DEP16 c2884: 720.1k vs 719.0k tok/s/GPU (same steady-window metric and normalisation), step 13.73 vs 13.80 ms (no regression).
• TP4 c1 agg: step p50 3.514 vs 3.501 ms, p90 interactivity 968.1 vs 963.2 tok/s/user (neutral).

A MegaMoE NVLink barrier timeout after the benchmark abort also happens without these changes, so it is not caused by them. Serving-phase results are unaffected.

Checklist

• Format with pre-commit
• GPU validation on cluster (above)
• Unit tests: test_dsv41_draft_metadata_glue.py, test_dsv41_target_verify_metadata_glue.py and test_decode_cuda_graph_runner.py (existing tests updated for the num_tokens argument, a padded-slot case, and the padded / IDLE routing cases)

---

CI States

Latest PR Test (Base): ✅ Run #41787902058
Latest PR Test (Extra): ❌ Run #41787902112
Latest PR Test (AMD ROCm 10): ❌ Run #41787902029

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[nvpohanh]

[by Claude Code] I've started babysitting the NVIDIA CI of this PR. I'll check it every 30 minutes until all NV pipelines pass. If a relevant job fails, I may rerun it, merge the latest main into this branch, push a small test-only fix, fix the PR title if the title check fails, or post a note here.

Anyone can stop me at any time: just post a comment asking me to stop (for example, stop babysitting). I'll stop at my next check, within 30 minutes, and take no further action on this PR.

[nvpohanh]

/rerun-failed-ci

[nvpohanh]

[by Claude Code] All NV pipelines have passed.