[NSagan271]

Stacked on #1127 (resident decode planning). Two host-time cuts on resident decode steps that #933/#966 had and #1127 doesn't, a resident row per KV pool, and Qwen3-TTS and Whisper on resident tables. #933/#966 stay as drafts for reference.

Attention page arrays built in the graph

On Hopper, FlashInfer's paged decode reads an indptr, a flat page-index list and last-page lengths, not a block table, so a resident step still built the page lists on the host (the KV plan) and copied them (the attention plan).

• FlashInferDecodeWrapper.gather_resident (mstar/engine/resources/attn/wrappers.py): one kernel per resident capture fills the wrapper's three arrays from Resident decode planning: decode rows on device tables #1127's block table and the lengths the KV gather already reads, through the step's slots.
• It is recorded only when the wrapper's plan takes the direct planner path (schedules_alone); the (bucket, pool) then joins ResidentPlanner.lengths_only, next to host_plan_free.
• On such a bucket the KV plan returns host lengths only, from the tables' host mirror (KVResidentTables.decode_indptrs), and the attention plan runs FlashInfer's planner alone (plan_schedule). A promoted step reuses its pre-plan's lengths.
• The fmha (Blackwell) path is unchanged.

Sampler scalars in one copy

gather_static staged each scalar buffer (temperature, top-k, top-p, seed, penalty, and any optional ones) with its own H2D, ~14 µs of host time apiece. The per-step rows are now views of one byte block per graph slot (StaticRows), staged and copied once. This one doesn't depend on #1127.

A resident row per KV pool

#1127's planner keeps one stream per row: one block table (_seal refuses a second), one label per step, one ResidentWrites.kv_len, and one block table, lengths buffer and attention mode per capture. Whisper's decoder has two KV pools, its self-attention cache and its cross attention context, so it couldn't run resident.

• Each of these is keyed by resource key (mstar/engine/resources/resident.py): every KV pool binds its own block table and lengths, positions read the length of the pool their PositionConfig.kv_cache names, and an attention reads its own pool's table. bind_resident takes the resource's key.
• A pool whose stream a step declares at span 0 is read only: the capture gathers its lengths and leaves its table, the host mirror doesn't advance, and the pool keeps its host plan. Whisper's context is the case: written by the prefill, read by every decode step.
• Adjacent runs of a rewritten row go in one copy, so a row of one pool is still one copy, as on Resident decode planning: decode rows on device tables #1127.
• Cross attention over a context in its own pool runs on resident steps, packing its queries off the step's rows; a context in the query's own pool still refuses.
• This is a proposal for Resident decode planning: decode rows on device tables #1127; if Resident decode planning: decode rows on device tables #1127 takes its own version, this commit drops.

Models

• Qwen3-TTS: the Talker's talker_decode walk opts in (resident_walks).
• Whisper: the decoder's decode walk opts in.

Results

2× H200; one GPU per server. Rust frontend and SHM arena for Qwen3.5. Each table is ABBA, back to back: #1127's head (863cf87), this branch, this branch, #1127's head.

Qwen3.5-0.8B, prefix cache off, --ignore-eos, 256–512 output tokens, text tok/s:

	#1127	this	Δ
c=16	9,269 / 9,108	9,317 / 9,415	+1.9%
c=128	27,445 / 27,599	28,647 / 28,294	+3.4%

Phase timing (benchmark/worker_phases), µs per iteration, before the per-pool commit (64085a9):

	#1127, c=16	this, c=16	#1127, c=128	this, c=128
plan thread (pre_plan)	481	322	836	527
↳ KV	92	39	218	35
↳ attention	131	68	79	45
↳ sampler	37	17	112	49
GPU thread total	1,006	826	2,410	2,115
GPU thread waiting on the plan	226	116	401	171
iteration	1,797	1,660	4,607	4,336

Qwen3-TTS 0.6B (benchmark.tts_speech_bench, 128 sentences, 5 repeats per concurrency), audio-s/s:

	#1127	this	Δ
c=32	169.9 / 167.4	169.6 / 168.4	+0.2%
c=64	236.7 / 239.7	239.8 / 238.8	+0.5%

Whisper large-v3-turbo (benchmark.asr_eval, LibriSpeech test-clean, 640 requests at c=64, four passes per server; the first is warmup), req/s, mean of passes 2–4. The transcripts are greedy and score the same WER, so req/s compares like for like:

	#1127	this	Δ
c=64	101.98 / 102.91	105.43 / 105.30	+2.8%

Each pass is ~6 s; the twelve measured passes don't overlap (#1127 101.6–103.5, this 104.7–106.3).

Output checks:

• Qwen3.5 (greedy, saved outputs compared per request): this against Resident decode planning: decode rows on device tables #1127 is 12–19% identical with a 38–43% median common prefix, the same range as Resident decode planning: decode rows on device tables #1127 against itself (14–15%, 40–41%).
• Qwen3-TTS WER (benchmark.tts_speech_wer): 0.93–1.00% for this, 0.71–1.50% for Resident decode planning: decode rows on device tables #1127.
• Whisper WER: 8.07–8.10% for this, 8.04–8.10% for Resident decode planning: decode rows on device tables #1127.
• 0 failed requests in every run.

Tests

• ruff check .: clean.
• New in test/modular/test_resident_planning.py:
  • the KV plan's host lengths match the full host plan of the same step, padding rows included (CPU);
  • the kernel's page arrays match the host indptr and the block table's pages (CUDA, ran on H200);
  • two pools bind rows of their own;
  • a pool the step only reads keeps its row, and its host plan, against a host-planned reference, including beside a lengths-only pool;
  • a rewritten row is one copy per pool;
  • cross attention's resident query packing matches the host plan's.
• test/modular/test_resident_planning.py, test/modular/test_sampler_*.py, test/modular/test_sampling_*.py, test/sampling_test, and the cross attention, Whisper and Qwen3-TTS modular tests: pass on H200. test_sampler_static_gather_skip.py now counts the one packed gather.
• Modular tests matching kv/attention/sampler/resident/decode/capture/position/recurrent/whisper/tts/qwen3_5 on H200: every failure also fails on Resident decode planning: decode rows on device tables #1127's head (test_client_error_text, test_gpu_mla_attention shared-prefix capture, test_kv_offload OOM report, test_audio_watermark, test_bagel_cfg_positions, test_node_fp32_precision_gpu, test_qwen3_omni_encoders), except test_ragged_attention.py::test_piecewise_replay_matches_reference[one_empty], which failed only inside that run and passes alone on both.

AI assistance was used for this PR.

🤖 Generated with Claude Code