Skip to content

[Cherry-pick to release/v0.5.15] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay (#30839) - #31083

Merged
Fridge003 merged 2 commits into
release/v0.5.15from
glm52-pd-v0.5.15post1
Jul 14, 2026
Merged

Fridge003 merged 2 commits into
release/v0.5.15from
glm52-pd-v0.5.15post1

Conversation

@kpham-sgl

@kpham-sgl kpham-sgl commented Jul 13, 2026 •

Copy link
Copy Markdown
Collaborator

Backport of #30839 to release/v0.5.15.

Structured as two commits to stay conflict-free against the release branch:

  1. Reapply [Spec] Anchor GLM-5.2 MTP IndexShare topk on the draft-extend step #29787 — reverts the release-branch revert Revert "[Spec] Anchor GLM-5.2 MTP IndexShare topk on the draft-extend… #30842, reinstating the GLM-5.2 MTP IndexShare draft-extend anchor that this fix builds on.
  2. Squashed backport of [bug-fix] Stabilize GLM-5.2 MTP IndexShare across PD and CUDA graph replay #30839 — the stabilization work itself.

Motivation

This PR makes GLM-5.2 MTP IndexShare reliable across PD disaggregation, context parallelism, overlap scheduling, batch splitting, and CUDA graph replay. It preserves the correct draft-extend DSA seed across draft steps, avoiding redundant indexer computation without reusing stale or misaligned indices. This improves proposal consistency, acceptance length stability, and inference performance.

This is a follow-up to #29787. The optimization remains gated to EAGLE with speculative_eagle_topk == 1, index_share_for_mtp_iteration=true, and a valid index_topk.

Modifications

  • Transfer per-request DSA top-k seeds from prefill nodes to decode nodes in PD disaggregation, with explicit invalid sentinels and retry-state cleanup.
  • Unify eager and CUDA-graph seed capture through LogitsProcessorOutput, including correct last-token extraction after CP gather/reordering.
  • Preserve seed alignment across context parallelism and Two-Batch Overlap splitting.
  • Keep reusable seed state across copied forward batches and graph capture/replay.
  • Track seed validity in FutureMap to prevent stale overlap/PD state, and fall back to eager recomputation when no valid seed is available.
  • Refresh paged-MQA schedules, top-k-v2 plans, and per-step metadata during multi-step DSA graph replay.
  • Resolve DSA context-parallel aliases from the effective attention backend.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #29295651002
Latest PR Test (Extra): ❌ Run #29295650791

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

…eplay

Backport of #30839 to release/v0.5.15 (squashed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kpham-sgl
kpham-sgl force-pushed the glm52-pd-v0.5.15post1 branch from 5e80b91 to 27dd90d Compare July 14, 2026 00:20
@kpham-sgl

kpham-sgl commented Jul 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Internal nightly run #0e8ae04b has passed GSM8K. Accept length of PD Disagg also looks healthy > 5

@kpham-sgl
kpham-sgl marked this pull request as ready for review July 14, 2026 00:43
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@Jiminator

Jiminator commented Jul 14, 2026 •

Copy link
Copy Markdown
Collaborator

Validated with GLM-5.2-NVFP4 (agentic multi-turn workload, ISL ≈ 80k, EAGLE 5/1/6) on 4×GB300 and 8×B300: no perf regression (within run-to-run noise vs release without this change), acceptance length unchanged, and no degenerate output across all generated turns.

4×GB300

Concurrency TP=4 tok/s/gpu TP=4 tok/s/user TEP=4 tok/s/gpu TEP=4 tok/s/user
1 15,452 * 455 17,728 439
2 25,062 265 25,359 258
4 32,074 156 32,239 146
8 37,886 78 38,006 80
16 40,921 39 42,089 41

8×B300

Concurrency TP=8 tok/s/gpu TP=8 tok/s/user TEP=8 tok/s/gpu TEP=8 tok/s/user
1 10,393 * 524 11,263 505
2 16,108 364 16,252 341
4 21,173 239 20,822 195
8 24,957 108 25,771 110
16 27,309 59 27,319 60

* depressed by one-time JIT compile on the cold first step; interactivity unaffected.

@Fridge003
Fridge003 merged commit 1bee78d into release/v0.5.15 Jul 14, 2026
71 of 80 checks passed
@Fridge003
Fridge003 deleted the glm52-pd-v0.5.15post1 branch July 14, 2026 02:37
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
…oss PD and CUDA graph replay (sgl-project#30839) (sgl-project#31083)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants