[merceod]

Third pass over the host side, on top of #380. Smaller than the first two rounds: the remaining per-step Python is
spread thin, so this one is a handful of targeted changes behind knobs, plus one fix the cross-model check turned up,
each gated the same way as before (greedy parity vs HF at N=1/8/32, fixed-8, the stop-sequence check).

• Sampler loop-back masters only for a node that reads them. worker: device loop-back, rows-only steps, leaner API delivery (round 2) #380's device loop-back had the sampler keep a slot master of
  every request's last token for every node, and on an eager decode step the write behind it builds its slot index with
  torch.tensor(..., device=cuda): a synchronous host-to-device copy per step. Whisper's decoder samples eagerly and
  reads its tokens on the host, so it paid a stream sync per decode step for nothing (Qwen3.5's captured decode never
  did). A submodule now declares reads_device_loopback (the Qwen3.5 LLM node does) and the engine enables the
  sampler's masters for that node alone. Whisper numbers below.
• KV decode plan off the previous step's plan (MSTAR_KV_PLAN_CACHE, default off). When a captured decode step has
  the same rows as the last one and only that step's commit touched the streams, the KV manager advances lengths by
  one, refreshes the rows that crossed a page boundary and builds FlashInfer's index arrays with numpy instead of
  walking every stream. Anything else that touches a stream (resets, removes, reloads, offloads, rollbacks, releases,
  forks, a dropped pre-plan, a commit of a cached row in another batch) invalidates it; another request's prefill does
  not. =2 runs both plans every step and logs any difference; the gate ran that way.
  Numbers: neutral on 0.8B at c32/c64/c128 (three fresh instances per side on one H200: 8921 / 13024 / 15912 vs 8911 /
  13036 / 15504 tok/s for the best clean pair), and the kv plan phase itself does not move (the batch changes every step or
  two under that load, so there is little for a cache to reuse). Kept as an opt-in with the verify mode.
• Page-locked ring for the index arrays (MSTAR_KV_PINNED_INDPTRS, default off: measured neutral). FlashInfer's plan copies them
  with non_blocking=True, which from pageable memory is a staged copy, and it makes the indices copy blocking
  outright; the decode wrapper now copies the indices straight into its captured buffer from the ring.
  Numbers: neutral to slightly negative (12551 / 15273 vs 12366 / 14903 tok/s at c64 / c128 on one pair; the event
  bookkeeping per take costs about what the staged copies did), so it is off.
• In-graph decode inputs, in two knobs (MSTAR_INGRAPH_DECODE_TOKENS, MSTAR_INGRAPH_DECODE_ROPE, both off). Round
  2 reverted this because outputs broke; the cause was the piecewise capture calling the region with the slot's
  static inputs and no step, so the in-graph reads were never recorded. Fixed, the tokens half gates clean; the rope
  half lowers batch-vs-alone agreement and both together trip a device assert at bs 64, so they stay off.
• Lazy prefix-chain extension (MSTAR_KV_CHAIN_LAZY_STEPS, default 1 = as before). Measured a wash; kept as a knob.
• Capture passes its step to the region (PiecewiseCudaGraphRunner._call_inputs).

Not in this PR, written up in the report: speculation reuse (analysed), the stop check in Rust, the TP2 c8
follower waits (located, trace in notes/tp_c8_trace.md), startup compile time (measured: CuTe DSL kernels compile in
every process, no file cache in nvidia-cutlass-dsl 4.8).

Cross-model check (klone H200, main vs #368 vs #380 vs this branch, each server alone unless noted; full tables in the
round-3 report):

• Cosmos3-Edge: t2i 640x640 median 0.59 / 0.90 / 0.55 / 0.53 s (vLLM-Omni 0.28: 2.15 s); t2v 832x480 39.0 / 39.2 / 19.7 /
  19.0 s (vLLM-Omni 20.5 s). The 2x on video came with worker: device loop-back, rows-only steps, leaner API delivery (round 2) #380 and holds here.
• Kokoro TTS, 200 requests per point, three repeats: c32 23.0-24.5 / 23.5-23.7 / 23.2-23.5 / 22.8-23.5 req/s. Equal.
• Qwen3-TTS, same protocol: c32 3.06-3.09 / 3.02-3.07 / 3.04-3.09 / 3.04-3.14 req/s. Equal; vLLM-Omni's speech route does
  5.6 req/s at c32 on 26 requests, a gap that predates these branches.
• Whisper large: the one regression. Each tree alone, 64 requests: c16 8.2 / 6.7 / 6.3 / 5.2 req/s, c4 7.6 / 7.2 / 7.3 /
  5.5; a two-instance re-check of main vs this branch on a quiet GPU gives c1 5.4 vs 4.7, c4 5.6-6.1 vs 4.7-5.5, c16 6.1-7.2
  vs 4.9-5.2. Whisper is a large audio prefill plus a few dozen output tokens, so per-request fixed cost dominates and the
  branches added one: the sampler's eager-step master write (above). With the fix, fresh instances alternating with main:
  main 5.5 / 4.0 / 4.3 then 5.6 / 6.2 / 7.1 req/s at c1 / c4 / c16, this branch 5.3 / 5.4 / 7.2 then 5.2 / 5.7 / 6.1; the
  c16 loss is gone and so is the c1 first-token tail (p90 114-142 ms vs 286-345 before, main 104). Gate on the fixed
  tree clean.
• BAGEL t2i (12 requests): c1 JCT 16.8 / 17.1 / 16.7 / 16.7 s, c4 65.3 / 66.2 / 64.9 / 63.6 s. GPU-bound, equal.
• Wan2.2 TI2V-5B t2v 832x480x33f: c1 0.253 / 0.245 / 0.245 / 0.248 req/s (vLLM-Omni 0.26). GPU-bound, equal.
• Qwen3-Omni: not run on this stack (the model pins flash_attention_2 and there is no flash-attn wheel for torch 2.12/cu13
  on klone); Orpheus: gated checkpoint.

Qwen3.5 head-to-head with #380 (same GPU, fresh server instances): equal at every concurrency on 0.8B and 4B, and on
9B TP2 c8 with long cells. Against the other systems on the same GPUs, same day (output tok/s at c8 / c32 / c64 / c128,
256-token requests, best of two fresh instances for mstar):

model	mstar (this branch)	vLLM 0.31.0	SGLang 0.5.21
0.8B	3277 / 8791 / 13122 / 16237	2338 / 6370 / 9426 / 11759	3020 / 8892 / 14130 / 14484
4B	1928 / 5553 / 8663 / 11474	1799 / 5289 / 8553 / 11526	1990 / 5839 / 8991 / 11786
9B TP2	1914 / 5822 / 9523 / 12758	1870 / 5366 / 8971 / 12709	2062 / 6286 / 10127 / 13489
27B TP4	1142 / 3653 / 5964 / 8586	1012 / 3141 / 5680 / 8458	1103 / 3505 / 5934 / 8681

So ahead of vLLM at every size (40 % on 0.8B, 1-16 % on the others) and within -7..+12 % of SGLang. One thing worth knowing for anyone benchmarking this: a server instance comes up in one of two states
that last its whole life, where the worker's main thread loses either ~0.06 or ~0.16 ms per step between iterations
(about 7 % of 0.8B tok/s at c32, nothing at c64+). It is not tied to the branch (main, #368, #380 and this branch all
show both states), so compare state-matched instances or several instances per side; tools/phase_idle.py in the
perf notes measures it.

[merceod]

Re-merged on top of #380's re-merge (69abae2) and checked on H200s on 2026-10-12: 0.8B gate clean (parity 17 / 17 / 16 of 64 at N=1/8/32, fixed-8, stop-sequence check, prefill padding, long prompts, abort storm, bad requests all as before), 0.8B cells 8947 / 12995 / 15731 tok/s at c32 / c64 / c128 (the clean-instance range of the round-3 table), 9B TP2 gate clean with c8 / c32 at 2029 / 5637. Whisper on this head vs main, each alone on its own node, same day: 5.78 / 4.39 / 3.26 vs 4.03 / 4.12 / 3.20 req/s at c1 / c4 / c16, so the fix holds after the re-merge (c1 first-token p90 102 ms vs main's 427 that day). Both trees ran well under their 10-09 numbers at c16 that afternoon, with the response landing in one piece, which looks like the cluster on the eve of its maintenance window rather than either tree; worth a quiet-day re-check.

[vasilevklart]

Default path and the Whisper fix look good (tested on h200s) Needs a rebase onto #380

• INGRAPH_DECODE_TOKENS crashes at c128 because the master gets reallocated under the captured graph
• INGRAPH_DECODE_ROPE gives wrong answers at c=1
• KV_PLAN_CACHE barely runs with pre-plan on (promoting the pre-plan calls clear_preplan which bumps the epoch), and the =2 check skips decode_offsets
• the two new test files aren't in cpu-core.txt, so CI never runs them

LGTM otherwise

[stephen-dwq]

INGRAPH_DECODE_TOKENS=1 has captured graphs reading freed memory because of reallocation of last_token.master after it has been captured in a graph. It must therefore not grow after anyone captures it.

We can resolve by either removing the INGRAPH_DECODE_TOKENS option or by pinning a really big last_token.master and raising an exception if we ever have to assign past the really big limit.

[stephen-dwq]

lgtm