MTP verify-dedicated interior arena: both CUDA-graph slots replay (Primary + Verify)

[justinchuby]

Summary

Gives the fixed M=k+1 speculative-verify forward its OWN executor sibling with a private interior device-buffer arena, unblocking verify CUDA-graph capture+replay, then closes Blocker B so the M=1 Primary decode graph also replays. Both cuda_graph slots now replay on one executor — the campaign's core proof — GPU-validated on the real qwen38-27b-int4 hybrid (GDN+GQA) MTP artifact.

Branches off 39885ffc5 (per-slot executor host capture state, #1654).

What changed

• Verify-dedicated sibling (build_verify_sibling, Session::verify_exec + verify-sibling run/capture/replay/reset methods): a structural clone of the main graph on the Verify slot, sharing only immutable weights/EP + the persistent external KV/recurrent-state bindings. Its interior [1,M] scratch is independent of the M=1 decode's [1,1] scratch, so the interleaved base decode can no longer resize/clobber the M=2 verify graph (the old Slice [1,2] vs [1,1] decline → eager fallback).
• Race fix (CUDA-700): the sibling PINS its StepScoped workspace (it only ever runs the fixed M=K shape), so its captured graph's baked scratch pointer is never freed back to the shared EP arena between replays. Reset the verify sibling's graph only at the generation boundary.
• Blocker B: re-advance accepted tokens one at a time (M=1, state-equivalent, byte-identical) instead of a batched M=num_accepted forward; and enable retain_decode_graph_across_spec when verify-capture arms so the contents-only KV roll-back no longer invalidates the Primary graph.

No hardcoded dims (M derived from num_speculative_tokens+1). Greedy path inert (no verify_width, never hits a non-zero rewind → byte-identical).

Validation (H200, k=1, 83.3% accept, median of 5, ORT 1.28 cuda13, int4 block-32)

Both slots replay, fallbacks=0, invalidations→~0 (every one of 5 runs):

cuda_graph:        captures=4 replays=504 fallbacks=0 invalidations=3
cuda_graph_verify: captures=4 replays=184 fallbacks=0 invalidations=0

• Token-identical across all 5 runs (deterministic), no NaN, coherent text.
• MTP throughput: median 20.63 tok/s (up from 15.6 pre-fix).
• engine lib 579/0 (native-backend), ep-cuda graph::tests 8/8, recurrent-commit + native_verify_logits_require_restored_recurrent_state byte-identity oracles pass.

Honest perf finding (not a speedup)

Graphing the verify was necessary but not sufficient. Approach-A (snapshot→restore→re-advance) redundantly re-runs the accepted tokens on commit, and MTP runs verify(M=2) + per-token re-advance forwards per step for ~2.67 tokens/step — strictly more device work per accepted token than greedy's single graphed M=1 forward on this launch-bound decode. So MTP (20.63 tok/s) still trails the campaign-established greedy baseline (~56 tok/s; corroborated here by Primary replays=504 exactly matching the greedy run). The greedy baseline artifact is out-of-sandbox this session.

Next lever: move to approach-B — thread num_accepted through the verify to SELECT the committed recurrent state from per-position post-states instead of re-scanning, removing the redundant re-advance forwards. The two-slot graphed executor here is the prerequisite.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.65%. Comparing base (72403a2) to head (96e271e).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1658       +/-   ##
===========================================
+ Coverage   81.02%   82.65%    +1.62%     
===========================================
  Files         384       12      -372     
  Lines      180358     5477   -174881     
  Branches   180358     5477   -174881     
===========================================
- Hits       146142     4527   -141615     
+ Misses      29271      757    -28514     
+ Partials     4945      193     -4752     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (ø)
cli-ort-windows	82.11% <ø> (ø)
mlas	?
offline	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 372 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.