[GLM-5.3] Restore opt-in DeepGEMM schedules outside verify graphs

[Fridge003]

Motivation

Restore SGLANG_EXPERIMENTAL_DSA_INGRAPH_VERIFY_METADATA_DG_OUT_OF_GRAPH=1, removed by #38071. It keeps verify metadata refresh in the CUDA graph while rebuilding the full and pooled DeepGEMM schedules eagerly before replay.

Depends on PR #38852, the in-graph metadata restoration. For GLM-5.3-Flash, enable all three restored flags. The new flag defaults to false.

#38845 is merged into main. This PR has been rebased onto the updated #38852 branch so its diff contains only the DeepGEMM option. After #38852 merges, rebase and retarget this PR to main.

Modifications

• Put the residual schedule builder in the dedicated dsa_metadata_deepgemm.py module.
• Build both schedules from the current raw sequence lengths before replay; captured metadata still contains the preceding replay's values at that point.
• Materialize constant offsets lazily outside capture and copy schedules into their existing destination buffers.
• Skip both schedule builders during capture when this option is enabled; preserve the default KPool write-plan update behavior.
• Add a GPU regression that poisons old metadata, verifies both schedules before graph replay, and checks the final metadata and stable buffer addresses afterward.

Accuracy Tests

Rebased head: c164ff266e610b521aa1fbb54c49c5e744e589ba. This commit was replayed onto the updated #38852 without conflicts; its functional patch is unchanged. Changed-file pre-commit checks and git diff --check pass. The B200 correctness and benchmark results below are the preserved pre-rebase measurements; GPU tests were not rerun for this rebase.

On baizhou-dev-b200 (4× B200), commit bd4adbb010462fec7f861973652b27bb536369c4 with all three restored flags enabled:

SGLANG_EXPERIMENTAL_DSA_KPOOL_METADATA_FUSION=1 \
SGLANG_EXPERIMENTAL_DSA_INGRAPH_VERIFY_METADATA=1 \
SGLANG_EXPERIMENTAL_DSA_INGRAPH_VERIFY_METADATA_DG_OUT_OF_GRAPH=1 PYTHONPATH=python \
  python test/registered/e2e/models/test_glm53_flash_b200.py -v
# Ran 4 tests: OK (693.815 s)

The test file, 500 GSM8K examples, 20 shots and 0.930 floor are unchanged.

Recipe	This PR GSM8K	Result
DFlash2	0.938	Pass
HighThroughput	0.936	Pass
LowLatency	0.934	Pass

The LowLatency speed/acceptance test also passes: 348.25 tok/s, 4.422 accept length (gates: >250 tok/s and >4.0). Real model capture logs confirm dg_in_graph=False on all TP ranks.

Cumulative focused GPU regressions pass 9/9, including the new stale-schedule regression:

PYTHONPATH=python python test/registered/kernel/attention/test_dsa_kpool_metadata_fusion.py
PYTHONPATH=python python test/registered/kernel/attention/test_dsa_metadata_replay.py
PYTHONPATH=python python test/registered/kernel/attention/test_dsa_ingraph_metadata.py
PYTHONPATH=python python test/registered/kernel/attention/test_dsa_ingraph_deepgemm_residual.py

This full run passes all four checks. The stack's earlier failures are retained: main LowLatency GSM8K scored 0.928, then 0.920 in one controlled repeat; the parent in-graph PR scored 0.926 in its full run, then 0.936 in a separately recorded LowLatency repeat. The 0.928, 0.920 and 0.926 scores fail the unchanged 0.930 floor. This PR remains draft because its parent is draft with an unresolved accuracy gate; this passing run does not establish the cause of the earlier failures.

The additional real-GPU metadata diagnostic also passed 16 configurations × 17 rounds = 272 graph replays with the residual enabled: bs=1/4/16/48, draft lengths 2/4/6/8, changing sequence lengths across pool/page/top-k boundaries through 100001 tokens, and remapped requests. It interleaves live graphs and checks ordinary versus optimized metadata and stable addresses. The parent in-graph path passed the same 272-replay diagnostic. The script, exact heads/flags and logs are in the evidence link below.

Speed Tests and Profiling

Same four B200 GPUs on baizhou-dev-b200, TP4/EP4, CUDA 13.0, driver 580.173.02, PyTorch 2.13.0+cu130, Triton 3.7.1, DeepGEMM 0.1.7 and FlashInfer 0.6.18. The fixed main baseline is 9a2f17f41d185614a17f740c006c4aca84c016bf.

Model revisions: zai-org/GLM-5.3-Flash@eb9eb208eb0d988989d07a6a12d0fdeb5f52574a and incoai/GLM-5.3-Flash-DFlash2@bf582e4eacc1810f76656d1811693ff6c6737d2a. Every weight shard was SHA256-verified.

Serving benchmarks use the test file's LowLatency EAGLE recipe, with radix caching disabled consistently for measurement. bs means --max-concurrency; each run sends 4 * bs requests under that concurrency limit. Inputs are exactly 8192 token IDs, outputs exactly 1024 tokens, seed 42, temperature 0, one warmup request and a cache flush. EOS is ignored by the current CLI default. Each cell has three runs; throughput is the median of the three run-level rates, and latency is the median of the three run means; throughput ranges are retained. All measured requests completed with the expected token counts.

bs	Main tok/s	Parent PR tok/s	This PR tok/s (min–max)	Speedup vs parent	Speedup vs main
1	511.93	523.69	535.00 (534.41–539.50)	1.022×	1.045×
4	1319.90	1354.75	1379.02 (1371.43–1384.11)	1.018×	1.045×
16	2100.45	2159.02	2182.21 (2171.33–2186.44)	1.011×	1.039×

Latency medians for this PR:

bs	Mean TTFT (ms)	Mean TPOT (ms)	Mean E2E (ms)
1	259.491	1.606	1902.127
4	452.261	2.418	2919.749
16	2024.732	5.312	7456.763

Reproduce from this PR's checkout (other restored flags unset):

export PYTHONPATH=python
export SGLANG_EXPERIMENTAL_DSA_KPOOL_METADATA_FUSION=1
export SGLANG_EXPERIMENTAL_DSA_INGRAPH_VERIFY_METADATA=1
export SGLANG_EXPERIMENTAL_DSA_INGRAPH_VERIFY_METADATA_DG_OUT_OF_GRAPH=1

python -m sglang.launch_server --model-path zai-org/GLM-5.3-Flash --tp-size 4 --ep-size 4 --dsa-prefill-backend trtllm --dsa-decode-backend trtllm --kv-cache-dtype fp8_e4m3 --moe-runner-backend deep_gemm --reasoning-parser glm45 --tool-call-parser glm47 --speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 --speculative-adaptive --disable-radix-cache --host 127.0.0.1 --port 31080

In another shell after the server is ready:

export PYTHONPATH=python
for rep in 1 2 3; do
  for bs in 1 4 16; do
    python -m sglang.bench_serving --backend sglang \
      --base-url http://127.0.0.1:31080 --model zai-org/GLM-5.3-Flash \
      --dataset-name random --num-prompts "$((4 * bs))" --max-concurrency "$bs" \
      --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1 \
      --seed 42 --tokenize-prompt --warmup-requests 1 --flush-cache \
      --output-file "bs${bs}-rep${rep}.jsonl"
  done
done

Raw run-level measurements, accuracy records and a standalone recomputation script are available together.

Checklist

• Changed-file pre-commit checks pass.
• GPU regression tests are registered in base-b-kernel-unit.
• Feature behavior and activation are documented above.
• Full-model correctness and serving benchmark results are recorded.

---

CI States

Latest PR Test (Base): ❌ Run #34724484159
Latest PR Test (Extra): ❌ Run #34724484350
Latest PR Test (AMD ROCm 10): ❌ Run #34724484225

[Fridge003]

This optimization is not valid