Qwen3-TTS: stacked-head Talker, incremental codec, up to 128 rows a step by vasilevklart · Pull Request #978 · mstar-project/mstar

vasilevklart · 2026-10-30T15:07:50Z

Qwen3-TTS on @dghosef's stacked-head design, on main's worker. It is stacked on #977 (three small engine changes). The first two commits are Joseph's, from his qwen3tts/stacked-head branch.

Talker: the CodePredictor's depth loop runs on Triton kernels (components/code_predictor_kernels.py) inside the decode capture, and prefill rides decode steps. Captures go to 128 rows a step (was 64). Stops stay on main's check_stop_batched, which keeps the EOS frame out of the codec.
cp_linear fix: with its norm taken from x, the row sums were wrong once a launch had more programs than SMs (20 of 20 runs at 64 x 8192). They now come from a pass of their own.
Codec: each frame is decoded once, from per-stream state (conv tails, sliding-window KV) in a codec_state recurrent pool; streamed audio equals one whole decode. Dilated convs run as undilated ones over interleaved subsequences: decode step 358 -> 4.6 ms in bf16 (8 rows x 25 frames, H100).
Config (qwen3tts_1p7b.yaml only):
- bf16 codec
- (4, 8, 16) first chunks
- 512 codec slots
- knobs: gc_gen0_threshold: 10000: at 128 rows the worker's ~8 ms full collections ate most of a stream's 10 ms pause budget.
Other deployments keep the defaults.

Performance, 1 x H100, 2,048 KV pages, on main 28949fe43: measured before this branch was rebased onto 160b4bc6d, and not re-measured since. A streaming SLO: Poisson arrivals, 60 s a rate. A request meets it with its first audio within 200 ms and under 10 ms of pauses; capacity is the highest rate at which >= 90% of requests meet it. Two arrival seeds are shown as seed 0 / seed 1; per-rate rows are seed 0.

	main	this PR
Seed-TTS sentences
capacity, req/s	20 / 20	96 / 88
audio-s/s within the SLO, at capacity	92 / 91	453 / 430
first audio p50 / p90 / p99 at capacity	76 / 106 / 135 ms	109 / 146 / 221 ms
at 24 req/s: requests in SLO; first audio p50 / p90 / p99	53%; 104 / 177 / 256 ms	100%; 59 / 67 / 76 ms
~20 s passages
capacity, req/s	6 / 6	32 / 32
audio-s/s within the SLO, at capacity	117 / 121	634 / 647
first audio p50 / p90 / p99 at capacity	55 / 74 / 89 ms	109 / 130 / 167 ms
at 8 req/s: requests in SLO; first audio p50 / p90 / p99	88%; 70 / 99 / 672 ms	100%; 61 / 70 / 80 ms
Light load
first audio p50 / p90 / p99 at 1 req/s	21 / 36 / 50 ms	40 / 47 / 52 ms
first audio p50 / p90 / p99 at 4 req/s	37 / 45 / 57 ms	48 / 55 / 61 ms
Quality
WER (Whisper-large-v3, 3 x 128 sentences)	23 / 4,446 words	25 / 4,446 words

At light load the first audio comes about 20 ms later than on main. The first chunk is 4 frames (320 ms of audio), so a client has that much buffered before the next chunk is due under load.

The node was shared with other jobs. In a run with less load beside it, the same model code held 120 / 112 and 36 / 36 req/s, with 555 / 500 and 714 / 719 audio-s/s within the SLO. So read capacity as about ±20%.

What raises it further (open PRs this one doesn't need):

Host overhead 4/N: Rust backend for the micro-scheduler #851 + Host overhead 5/N: cheaper resource planning, admit and commit #852 (host overhead 4/N and 5/N: the Rust micro-scheduler; cheaper planning, admit and commit). With this model at 128 rows, side by side on one node (seed 0):
- Seed-TTS capacity: 88 -> 112 req/s.
- Closed loop at c=128: 547 -> 580 audio-s/s on Seed-TTS, 643 -> 671 on passages.
Merged with them, 256 rows a step held 112 / 128 req/s on Seed-TTS and 44 / 44 on passages. On main's worker today, 256 rows measured no better than 128 (80 / 96 and 32 / 36 in one run), so this PR stays at 128.
Performance 4/7: shared API server, resource and worker changes #969 (shared API server, resource and worker changes: lazy host copies, a one-launch codec-state slot reset, fused sampling, socket waits instead of 1 ms sleeps). Measured as the integration branch that holds them, against Host overhead 4/N: Rust backend for the micro-scheduler #851-Host overhead 7/N: stream buffers hold items by handle; tensor store in the Rust bookkeeper #861, with this model, in separate jobs: +5% / +8% closed-loop audio-s/s at c=128 (Seed-TTS / passages), and first audio p50 6 ms lower at 72 req/s.
Host overhead 6/N: declared stop policies, checked in the graph runtime #857 (stop policies) does nothing for this model until a StopPolicy can withhold the EOS frame (below).

Tests on f7717945e, main 160b4bc6d in the same jobs:

ruff check . clean. CPU Core (CI's env, OMP_NUM_THREADS=2): 9,092 passed, the same 5 failures as main by name. The TP lockstep simulation shard (11), test_cpu_selection.py (15) and rust-transport pytest (481) pass, as on main.
1 x H100: the Qwen3-TTS, kernel, lease, row-output, chunk-schedule and 0.6B / 1.7B real-weights files: 187 passed / 0 failed. Main: 155 passed, 3 stale real-weights failures, which this PR fixes.
New tests:
- cp_linear with more programs than SMs.
- Streamed codec audio against a whole decode, within 1 LSB, on the real 0.6B decoder too.

Follow-ups:

Raise the row cap to 256 once Host overhead 4/N: Rust backend for the micro-scheduler #851 / Host overhead 5/N: cheaper resource planning, admit and commit #852 land, and re-measure.
The TokenStop -> StopPolicy bridge after Host overhead 6/N: declared stop policies, checked in the graph runtime #857: a StopPolicy.withheld, so the EOS-frame withhold rides the runtime stop check.
Mixed walks.

AI assistance (Claude) was used for this PR.

Back to [4, 8, 16], as upstream mstar-project#978 ships. A 1-frame first chunk let the next chunk arrive after the client had played it under load; a 4-frame one costs about 20 ms of first-audio latency on an idle server. The [2, 4, 8, 16] ramp measured TTFA p50 63-78 ms vs 84-95 ms at c=64 on an H100, with no stalls either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The fused add+norm writes in place, so starting the stream at input_embeds wrote into the captured graph's static input. A packed replay never restages the rows past its real ones, so whatever an earlier, longer step left there picked up every layer's output on each replay. Clone over an out-of-place first add+norm: same cost on the captured step (+6/+6/+1 us against +6/+8/+2 at bs 1/8/32 on 3600-4500 us), and it's what #978 does.

The two-token first pass and the fused add+norms were only measured on Blackwell, and they move numerics and speed, so blackwell_path_measured keeps older GPUs on a plain add and one token per pass. A Hopper row lifts it. The q/k norm + RoPE + KV write kernel and channels-last go the other way. The LM already runs that kernel ungated on Hopper, so the code predictor drops its capability check and matches the Talker, and #978 runs the codec convs channels-last on every GPU.

merceod

The incremental codec is a nice change, and the 128-row cp_linear tiling reads right (rows never mix, padded tiles stay finite). The blocker is that main moved under this branch. #859 landed its own code_predictor_kernels.py with ResidualCodePredictor and DepthKV, and Qwen3-Omni imports them, so the merge is a re-port of the depth loop, not a rebase. Two defaults in qwen3tts_1p7b.yaml also change the audio of the shipped deployment on H100-only, pre-rebase evidence. Details below, happy to re-review once it is on top of main.

mstar/model/components/code_predictor_kernels.py (add/add with main): main's file holds ResidualCodePredictor and a DepthKV that owns its cache, and qwen3_omni_model.py and test_lora_targets.py import them. Either keep main's for Omni or port Omni onto these kernels, and run the Omni and LoRA-target tests on the merge commit.
mstar/model/components/decoder_layer.py:103: main's run_layers copies hidden_states before it starts the stream and Qwen3 dense and Qwen3.5 count on that. If this version wins the merge their replay inputs get updated in place. Keep main's and drop the clone in talker.py:129, so it is one copy a step.
mstar/model/qwen3_tts/submodules.py:1364: main's codec opts into split_batches_by_capture_key so each chunk width batches on its own, and docs/models.rst:581 here says the same, but this branch does not opt in. A 4-frame first chunk next to full chunks gets decoded at 25 frames. Keep main's opt-in. The numbers in the description were taken without it.
configs/qwen3tts_1p7b.yaml:10 (bf16 codec): this changes the default 1.7B audio and the only evidence is WER. Can you add speaker similarity and an SNR or mel distance against the fp32 whole decode on real weights? Every parity test runs fp32. Related, incremental_codec.py:213 clamps the waveform in bf16 and only casts for PCM16, so each sample has an 8-bit mantissa. The final conv in fp32 is cheap and worth checking.
configs/qwen3tts_1p7b.yaml:15 ([4, 8, 16] ramp): by your own table idle first audio goes from 21 to 40 ms p50. With the incremental codec a 1-frame first chunk is just as exact, so I'd ship [1, 3, 8, 16] and document [4, 8, 16] for high load, or keep it and say why.
configs/qwen3tts_1p7b.yaml:29 (512 slots): "the backlog never drained" sounds like an engine gap rather than sizing. The pool reserves at readiness on one rank and a refused admit only holds the blamed row, so a stuck backlog means something skips that path. Worth finding out or filing. Also 512 counts the sink, so it is 511 streams.
mstar/model/qwen3_tts/submodules.py:1140: dynamic=False compiles once per batch bucket and width, 56 compiles on the 1.7B and 84 on the 0.6B Base, right at the default MSTAR_RECOMPILE_LIMIT. A dynamic batch dim gets it to one per width. What is cold start now vs main?
Perf claims: one H100 on a shared node against a main that still had the 32-row cap and power-of-two codec buckets, before #859. The gain over today's main is unmeasured. H200 closed-loop c=1/8/32 against current main, with codec dtype and ramp stated, would settle it.
Nits: talker.py:164 and docs/models.rst:606 say Omni attends through cp_attend, which is not true at this base or on main. submodules.py:1062 overrides can_use_accelerator_graphs, which nothing in the engine calls anymore. submodules.py:890 can drop step_walk from check_stop_batched in the merge, main did.

The shared ResidualCodePredictor (Qwen3-Omni's and Qwen3-TTS's) ran each depth step's layers as compiled projections around a q/k norm + RoPE + KV-write prologue, a Triton attention and FlashInfer's add + norm. At decode widths each of those is a few microseconds of launch around a ~1 us job, 16 x 5 times a Talker step. - cp_linear: a projection with the RMSNorm before it folded into its weight (a row scale from partial sums of squares its producer recorded) and the residual add, SwiGLU or bias in its epilogue. - cp_attend: q/k norm, RoPE, the cache write and attention over the depth positions so far in one launch. It replaces cp_qk_norm_rope_kv_write and cp_attention. - consolidate_stacked_weights folds the norms into the projections and, where the predictor projects its inputs (Qwen3-TTS 1.7B), keeps every codec embedding table a later step reads projected, so that step's input is a row lookup that cp_embed_accumulate returns. Both Talkers call depth_loop as before, and each keeps its embedding sum. The layers, heads and input projection are read only through their weights now, so the LoRA walk refuses adapters on them.

The codec decoded overlapping windows: every chunk after the first repeated up to 72 frames of context through the transformer, and the conv stack ran over the new frames plus its receptive field. Every module of the decoder is causal, so IncrementalDecoder decodes each frame once from its stream's state (each causal conv's last inputs, the transformer's sliding-window keys and values, the frame count), kept per stream in the codec_state recurrent pool. A stream's audio matches one whole decode within 1 LSB of PCM16. Chunks carry new frames only; a clone's first chunk still leads with the reference tail, decoded to warm the state and trimmed from the audio. The Talker's prefills and decode are one combined walk, talker_mixed: up to four new prompts ride a decode step instead of a prefill-only step that every decode row waits behind. Prefill embeddings no request changes (special tokens, the role, the codec tags) are built once. The codec keeps splitting batches by chunk width, and the Talker keeps TokenStop. The can_use_accelerator_graphs overrides, which nothing calls any more, go. Co-authored-by: Kirill Vasilev <vasilevklart@gmail.com>

… a pass of its own A step's rows were one cp_linear tile, so the depth loop took at most 64 rows. Past 64 rows the grid now also tiles the rows, 64 to a program, in the same launch, and the row-partials scratch holds 128 rows. At 64 rows and fewer the launches are the ones before. With no producer partials, a tensor-core cp_linear took its rows' sums of squares from the pipelined loads that also feed tl.dot. Once a launch had more programs than the GPU has SMs, those sums came out wrong, and not the same way twice: on an H100, 64 rows by 8192 outputs in 32-wide tiles was wrong in 20 of 20 runs, by up to 12% of the output's range. The sums now come from a first pass over x. The depth loop's own launches stay at 128 programs or fewer, so it did not hit this. Depth loop per frame at the 1.7B's shapes, one H100, graph replay: rows ms/frame us/row 8 2.688 336 64 2.913 45.5 96 3.559 37.1 128 3.765 29.4

…sequences cuDNN runs some bfloat16 channels-last dilated convs through its conv2d_grouped_direct_kernel: on an H100, 8 rows at 384 channels and dilation 9 take 134 ms (0.16 ms undilated), and at 768 channels and dilation 3 or 9, 108 ms, with or without cudnn.benchmark. Those are the decoder's residual-unit convs, so a bfloat16 codec ran far slower than real time. A dilation-d conv is d independent undilated convs over the inputs of each residue mod d, so it now runs as one undilated conv on those subsequences batched. The same products and sums: the streamed audio still matches one whole decode within 1 LSB. Serving decode step, replayed from its CUDA graph (8 rows x 25 frames, one H100): bfloat16 358 -> 4.6 ms, float32 10.4 -> 9.2 ms.

…ch bucket torch.compile(dynamic=False) compiled the incremental decode once per chunk width and batch bucket: 56 compiles on the 1.7B deployment and 84, the default MSTAR_RECOMPILE_LIMIT, on the 0.6B Base. _decode now marks the batch dimension of its inputs dynamic, so a width's buckets share a compile.

The largest decode capture bucket is the cap on rows per Talker step. Near capacity more than 64 streams are live, so the rest waited their turn until their playback paused. Decode and mixed steps now also capture 96 and 128 rows, which cp_linear's row tiles allow.

Both real-weights modules asserted the codec decoder's parameter count as the checkpoint's 114,323,137. Its SnakeBeta alpha and beta (23,232) load into FusedSnakeBeta's buffers, so the module holds 114,299,905; these fail on main too.

configs/qwen3tts_1p7b.yaml only; other deployments keep the defaults. - codec_dtype: bfloat16. - codec_chunk_schedule: [4, 8, 16]. With a 1-frame first chunk the next chunk arrived after the client had played it under load. - codec_state max_slots 512, 511 streams (one slot is the sink). Past the default 256, a new stream waits at readiness for a slot. - gc_gen0_threshold: 10000. At 128 rows a step keeps more young objects alive than CPython's 700-object gen-0 threshold, and the worker's ~8 ms full collections ate most of a stream's 10 ms pause budget.

* docs/models.rst: the Omni audio encoder's eager and sub-second cases as the code has them (R2.1) A step also packs, and a clip also replays, only within the attention plan's `audio_attn.max_segments_per_request` windows, so a clip under 8192 rows but over that ceiling runs the layers eagerly too. A clip under 1 s encoded alone differs from HF's in its last conv row, which the windowed attention mixes into every embedding of the clip (one window), not only in its last rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Head sharding: one per-resource switch, shard_over_sp; NodeSubmodule.splits_heads goes Every resource now shards its heads over all TP x SP ranks of the node's lockstep group (EngineResourceInfo.head_shards, a property), and a ragged resource that each SP rank runs whole says so with RaggedAttentionConfig(shard_over_sp=False). The node-level splits_heads flag, the engine's per-node fold and its agreement check, and their docs and tests are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni Thinker prefill capture: padding rows take the broadcast positions (R2.2) The `prefill_text` capture template, which `prefill_text`, `prefill_audio` and the mixed walk replay, still carried `custom_pos_ids`, and a replay's padding rows are built from it, so every padded text-only prefill or mixed step built its MRoPE positions row by row (a cast per real row and a cat) instead of the one broadcast view. The template is now a text row, as the decode one is since R1.14; an audio row brings its own positions. GPU-unverified: prefill capture and padded text/mixed replays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * GLM-5.2 prefill ladder through ARNodeSubmodule's override; the base declares the ladder (R2.3) `model_kwargs.prefill_token_buckets` had two mechanisms: Qwen3.5 and Qwen3-Omni pass it to `ARNodeSubmodule.__init__`, which refuses an empty or repeated ladder, while GLM-5.2 read its config's copy at capture with `or`, so `[]` silently meant the default and a repeat was captured twice. GLM-5.2 now passes it to the same constructor and its capture reads `self.PREFILL_TOKEN_BUCKETS`; the `or` fallback is gone. `ARNodeSubmodule` declares `PREFILL_TOKEN_BUCKETS`, the attribute the override replaces and the capture configs read. Test: test_glm52_prefill::test_a_bad_prefill_ladder_is_refused ([] and [64, 64]) did not raise before and raises ValueError now; the module's helper builds the submodule through its constructor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: the frontend takes the step's clips as a list; Omni and Qwen3-ASR stop padding them (R2.4) Both encoder submodules zero-padded every clip of a step to the longest and stacked them, and the frontend sliced each clip straight back out. Omni's features arrive in fp32, so a long clip batched with short ones allocated B x 128 x T_max x 4 B per step (about 0.5 GB for one 10 min clip beside fifteen 2 s clips) for nothing. The frontend now takes the clips as they are, `(num_mel_bins, frames)` each, and reads each one's length off its shape; the padded stack, the `feature_lens` argument, Qwen3-ASR's `num_frames` kwarg and its 2-D `forward` special case go. The convolution input is built from the same frames in the same order, so the embeddings are unchanged. GPU-unverified: the transient memory saving (no CPU-observable change). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: one checkpoint loader, `AuTEncoder.from_checkpoint`, for Qwen3-Omni and Qwen3-ASR (R2.5) The two models loaded the shared tower with the same six steps (build on meta, cast, `to_empty`, stream `thinker.audio_tower.*`, `load_weights`, `eval`), copied line for line. The sequence is now a classmethod on the encoder, which owns the parameter layout, and both models call it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT frontend: pad the last convolution launch to a multiple of 32 chunks, not a power of two (R2.6) The power-of-two rule could nearly double a launch: 257 chunks ran as 500 and 300 as 500, about 4 GFLOP per empty chunk, and Qwen3-ASR pays it too since the frontend is shared. A multiple of 32 (8 for a launch of at most 8 chunks) wastes at most 31 chunks at the cost of about 17 conv shapes instead of 7, each a one-time cuDNN plan. GPU-unverified: the audio encoder's phase time on Omni (5 x 60 s clips) and Qwen3-ASR (8 x 37 s), power of two vs multiple of 32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: code2wav never looks up the MASK id; unmask knobs must be finite (R1.1) A canvas that still held MASK cells went into the codec decode, whose codebooks end one id before the MASK id: on the GPU that lookup is a device-side assert that kills the worker and every request on it. The host check that used to refuse such a canvas went with the move of the output shaping to the API side. The decode now reads code 0 in those cells, and the NaN poison still makes postprocess refuse the waveform, so only that request fails. process_prompt now checks the unmask loop's float knobs are finite numbers (one parser, also used where the conductor reads them): an infinite reveal score ties every cell, which is how a client knob could leave MASK cells. The refusal message names both causes of a non-finite waveform instead of labelling a codec NaN as masked cells. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: bound the output-shaping durations, take 0/1 for postprocess_output (R1.12) output_shaping checked the shaping kwargs' types, not their ranges: pad_duration 1e5 passed process_prompt and then had the API server allocate tens of GB of padding after the whole unmask loop, and NaN or inf failed only after generation. Both durations must now be finite and between 0 and 10 seconds. postprocess_output takes 0 and 1 again, as JSON clients send booleans (the base read it through bool(); the move to postprocess had made them a 400). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: a decode row reads its text token by its loop iteration (R1.4) Decode prepare_inputs read the streamed text condition at a per-request counter and advanced it. prepare_inputs runs before admit, and a refused admit prepares the step again, so each refusal skipped one text token (a dropped sub-word, possibly an early EOS, no error). Mixed steps made this reachable for decode rows: a step carrying a new prompt's pages is refused under KV pressure, and every decode row in it was re-prepared. The index is now the step's own iteration of talker_decode_loop (InputMetadata.dynamic_loop_iter_counts, incremented for a speculated step), which a refused admit leaves alone; the counter is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: pack a step only up to the largest captured token count Qwen3-Omni's HF audio encoder split a step into calls of at most 30,000 mel frames. On the AuT node nothing bounded a step, so eight long clips ran as one eager forward over every clip's tokens. can_batch now packs a step only while it fits the largest capture bucket (4096 tokens); past it the clips run one by one, each replayed when it fits a bucket, so an eager step holds one clip's activations. The same node serves Qwen3-ASR, whose steps up to 4096 tokens are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: count tokens by the checkpoint's own chunk, for both models tokens_for_frames counted 13 tokens per 100-frame chunk whatever the config's n_window, while the frontend cuts chunks of 2 * n_window frames and Qwen3-Omni's prompt layout counted with transformers' helper at the checkpoint's n_window. It now counts each full chunk_frames chunk by the convolutions' arithmetic, and the layout calls it, so the prompt, the encoder's split and its frontend share one count. Unchanged for the released checkpoints (n_window 50). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ragged attention: refuse a per-request ceiling on a resource sized per step A resource with max_segments_per_step sizes its plan from that alone, so a deployment override of max_segments_per_request was accepted and then read by nothing. apply_yaml_overrides now refuses it, naming the per-step key. Its docstring counts the ceilings it takes correctly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: the engine hands a forward only the regions its step leased The case called SubmoduleManagement.runners_for directly, so reverting the engine's call to it went unnoticed. It now drives Engine._drive_step and reads the runners the forward receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3: take cudnn_benchmark from fp32_precision, drop the wan_vae re-export The scope moved to mstar.engine.fp32_precision and wan_vae kept a re-export for Cosmos3's two decode paths. They import it from its owner now, the re-export goes, and its test moves beside the other fp32 precision tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni: drop audio_seqlens, which the audio encoder no longer reads The AuT node takes a clip's frame count off its unpadded log-mel features, so the audio_seqlens tensor rode to the encoder unread. The prompt layout, its one reader, counts the same frames off the features' shape, so the tensor, its graph input and its routing go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder node: move it beside the encoder, one loader and spec for both models Qwen3-Omni imported its audio encoder node from the qwen3_asr package and repeated its loader and attention spec line for line. The node is now AuTEncoderSubmodule in mstar/model/components/aut_encoder_submodule.py, with AUT_ATTN, aut_attention_spec and load_aut_encoder_submodule, and both models call them. It takes a LogMelSpectrogram instead of Qwen3-ASR's config. Behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: a Qwen3-Omni audio step over 4096 tokens runs its clips one by one Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor kernels: fit cp_linear's tiles to the device's shared memory (R1.2) cp_linear's tiles were tuned on an H200 (227 KB of shared memory a block). At 3 to 64 rows the SwiGLU tile buffers x and both weights per pipeline stage, 40 to 48 KB, so four stages needed up to 192 KB: Triton refuses to compile that on 99 KB GPUs (sm_86, sm_89: A10G, A40, L4, L40S, RTX 3090/4090) and the 17-32 row tile on an A100 (163 KB), and the Qwen3-TTS depth loop runs only through cp_linear. The tile choice moves into _cp_linear_tile, which keeps the H200 choice where it fits and otherwise keeps fewer stages, then shorter K steps, until the stages fit the device's max_shared_mem. Every H200 tile is unchanged. The stale harness names in the old comment are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: on Base, batch each codec chunk width apart (R1.3) Codec steps batch chunks of any width, each padded to the widest. An in-context clone's first chunk leads with the reference tail, 73 frames wide by default, so every row batched with it paid a 73-frame decode, about three of a full chunk. The width-mixing A/B that motivated the mix ran CustomVoice, which has no clone chunk. Base, the one variant that clones, opts back into split_batches_by_capture_key, as the codec did before mixing; the other variants keep mixing widths. The docs said the scheduler batched every width apart, which it no longer did; they now say which variant does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B config: default pools, bf16 codec scoped to H200 (R1.10, R1.16, M.2, M.4, M.6, part of R1.11) configs/qwen3tts_1p7b.yaml, the default for `mstar serve qwen3_tts_1p7b`, allocated 4096 Talker KV pages (~56 GiB) and 512 codec slots at startup: it could not start on a 40-48 GB GPU. It now keeps the default pools; the overload-sized ones stay in the YAML as a commented override for GPUs with 80 GB or more, with their measurement dated before admission started queueing requests for KV pages and codec slots. The bf16 codec claim is scoped to what was measured: an H200. The B300 result in the docs was main's windowed codec, and the docs said "measure before switching" next to a config that had already switched; the YAML and docs now say to serve float32 on other GPUs until it is measured. The codec_dtype comment no longer claims float32 is bit-exact (the parity test allows one PCM16 step), and the comments say RMSNorm and RoPE are computed in fp32 from bf16 weights and frequencies rather than kept in fp32: the bf16 gate measured these numerics, so they stay. The 1.7B YAML headers said only the checkpoint differed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: OmniVoice split and Qwen3-TTS codec text match the code (R1.11) - OmniVoice's split config and its docs said code2wav's worker does the silence trim and post-processing; that runs on the API side now, and the split's measured gain was taken before the move, which they now say. - The Qwen3-TTS benchmark said the codec filters EOS frames; the Talker withholds them from the codec stream. - A Qwen3-TTS stream's audio matches a whole-utterance decode to within one PCM16 step, as its parity test checks, rather than equalling it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: Base takes four new prompts into a mixed step, as the other variants do (R1.19) max_rows_in_combined split MIXED_MAX_PREFILL_ROWS over the two prefill walks when the checkpoint supports reference audio, against a step that mixes both. No variant runs both: Base requires one reference clip, so every Base prompt runs the clone prefill, and the others never do. Base was capped at two new prompts a step where the docs say four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Recurrent state pool: refuse a max_slots override that leaves no slot (R1.20) RecurrentStateConfig refused max_slots below 2 (with the sink) when it was built, but the YAML override set it afterwards unchecked: a deployment's `codec_state: {max_slots: 1}` left no slot past the sink, and every request waited in admission forever. The override now runs the same check, which also covers a sinkless pool of zero slots. The Qwen3-TTS docs say what N slots hold: N - 1 streams, the sink taking one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: refuse codec chunk knobs that are not frame counts (R1.17) codec_chunk_schedule and codec_chunk_frames went through int(): [2.5, 8] ran as (2, 8), 12.5 as 12, [true] as 1, and a scalar schedule failed with a bare TypeError while the model was built. apply_codec_chunking now takes only positive integers and raises ValueError otherwise, as Qwen3-Omni's same-named knobs do. An empty ramp, refused before because the clone width read its first entry, starts at the steady chunk, as on Qwen3-Omni. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT frontend: pad the last convolution launch to the smaller of a power of two and a multiple of 32 (R3.1) The multiple-of-32 rule (R2.6) ran a last launch of 9-16 chunks as 32, where the power-of-two rule had run it as 16, so a single 9-16 s Qwen3-ASR clip convolved 2-3.5x its chunks. Taking the smaller of the two roundings (still at least 8, at most conv_chunksize) pads no launch more than either rule did: 9-16 -> 16, 17-64 as before, 65-96 -> 96, and at most 31 empty chunks, for 18 conv shapes instead of 17. GPU-unverified: the encoder's time on Qwen3-ASR at c=1 with 10 s and 14 s clips, multiple of 32 vs this rule (CUDA events around AuTEncoder.frontend, or worker_phases on the encoder node). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ARNodeSubmodule: narrow the PREFILL_TOKEN_BUCKETS comment to what holds (R3.2) The comment said every submodule that captures prefill names its ladder PREFILL_TOKEN_BUCKETS. BAGEL's LLM (PREFILL_TEXT_TOKEN_BUCKETS), the Qwen3-Omni Talker (TALKER_PREFILL_TOKEN_BUCKETS) and Cosmos3's DiT (an instance ladder with an env-var override) don't, and an override passed to one of them would be ignored. The comment now states the obligation instead: a submodule that takes `prefill_token_buckets` must read its capture ladder off `self.PREFILL_TOKEN_BUCKETS`, as Qwen3.5, the Omni Thinker and GLM-5.2 do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: fold the depth weights without autograd; drop dead hooks, caches and keys (R1.15) fold_depth_weights multiplied each projection weight by its norm weight with grad enabled, so every folded weight kept a grad_fn and its fp32 intermediates alive: about 0.34 GB a Talker on the 1.7B. It now runs under no_grad. Also gone, none with a reader: - the lm_head_weight buffer, a third copy of the LM heads beside the modules and the folded heads (its comment still cited the piecewise loop); the heads are stacked only to fold them; - both submodules' can_use_accelerator_graphs overrides (the engine never calls it); - the Talker's per-tuple id tensor cache, which only fed _fixed_embed's builds, already cached per key; - the Codec's codec_chunk_frames/codec_chunk_schedule step_metadata. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor kernels: cp_embed_accumulate says how many partials it wrote (R1.22) cp_linear's norm reads n_part row partials that the producer wrote, and the Qwen3-TTS depth loop worked out how many cp_embed_accumulate wrote by hand (embed width // EMBED_BLOCK); a wrong count normalizes silently wrong. cp_embed_accumulate now returns the count with the input, and refuses a batch wider than the partials' rows instead of relying on cp_linear's check running first. Its docstring no longer cites a constant that doesn't exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec, code predictor kernels: ValueError for limits a caller can hit (R1.23) The incremental decoder raised TypeError for a codec whose conv geometry it can't decode, and cp_linear checked its row, width and partials limits (tied to this checkpoint's shapes) with bare asserts. Both now raise ValueError with the limit in the message; the stride checks stay asserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: drop a clamp that never applies (M.5) prepare_inputs clamped a chunk's context items to its frame count, but StreamBuffer.pop_chunk already caps context_items at the chunk's items, and every item is one frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: cover cp_linear's two-row mode; drop Qwen3-TTS codec duplicates and pins (R1.24) - cp_linear's two-row tile was never run; the parity test now covers it. - The codec's empty-final-flush test, and the empty-chunk half of its padding test, repeated cases of the EOS-withholding stream test (num_frames 4 and 0); the state-pool test re-checked the block layout that CodecSubmodule already refuses to build without. - The codec graph-config test pinned CAPTURE_BATCH_SIZES and asserted can_use_accelerator_graphs, a hook the engine never calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: time and check the served decode, nothing else (R1.14) The bench's "base" arm called the window decode that no checkout in the stack has (_trim, a three-argument _decode, codec_window_contexts), its "nwc" arm timed a 180-line prototype of the decoder, its "ncw" arm (the served decoder) was labelled the other way round, and a SlotPool plus a whole-step compile stood in for the state pool. The base arm, the prototype and SlotPool are gone. ServedStep builds the codec's state pool from the model's own codec_state declaration and drives each step as the engine does (declare, admit, plan, the codec's own _decode with its compiled decode, commit); timing and kernels replay a planned step per chunk width, and parity streams real codes through it in the model's chunk ramp against a whole float32 decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: run the Qwen3-TTS codec parity on the served CUDA path; fix the real-weights codec test (R1.18) The streamed-codec parity test ran only eager fp32 on the CPU, while the server runs the decode compiled (Inductor) over channels-last cuDNN convs. It now also runs on CUDA, where CodecSubmodule compiles its decode, with TF32 off so both decodes are fp32. The real-weights codec test called _decode with the signature two codecs ago. It now runs a served step (the model's declared state pool, admitted and planned) and checks the bf16 codec against the float32 one on the same codes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Name the TP x SP head-sharding test for the kinds it covers; reflow a docs paragraph Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: say a step holds up to 4096 tokens; import its loader at the top The token budget applies to Qwen3-ASR as well as Qwen3-Omni, so the ASR docs and the node's docstrings now say a larger step runs its clips one by one. Both models import load_aut_encoder_submodule beside aut_attention_spec, from the module they already load at the top. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 9B DP2 config: give the GDN pool prefix snapshot slots (RM.1) The dp2 config was written before #877 and has no `gdn_state.prefix_slots`, so the recurrent pool can't serve a cached prefix and the engine keeps the LLM node's KV prefix cache closed with it. Each replica runs at world size 1, so it takes the same 128 snapshot slots as qwen3_5_9b.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests, docs: run the Qwen3-TTS codec parity with TF32 off for matmuls too; scope its claim (R2.1) Importing mstar.engine sets float32_matmul_precision('high'), so turning off only cuDNN's TF32 left the codec's GEMM-lowered transposed and kernel-1 convs in TF32 on the CUDA cases. Both decodes now run under fp32_precision_scope(Fp32Precision("highest", cudnn_allow_tf32=False)). The one-PCM16-step claim in the model's comment and docs/models.rst is scoped to float32 with TF32 off (as on CPU); the served CUDA codec runs its GEMMs in TF32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: drop --tf32, which could not turn TF32 off (R2.5) The flag was applied before the mstar imports set float32_matmul_precision ('high'), so every run used TF32 matmuls, as the served decode does, and a run without it was mislabelled as plain float32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: drop the cp_linear tile test that restates the shared-memory formula (R2.3) It asserted _cp_linear_tile's own stage_bytes expression, so it could not catch a formula that undercounts what Triton allocates. The real check is test_cp_linear_matches_the_unfused_ops compiling on a 99 KB (sm_86/sm_89) GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: drop Talker asserts on can_use_accelerator_graphs, a hook no engine path calls (R2.4) R1.15 deleted the Talker's override, so these three asserts only tested the base class's default; their comment described the deleted behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: say that Base's four-prompt mixed cap can spill long clones to eager (R2.2) The cap counts prompts, not tokens, so four clones with references of about 18 s or more outgrow the 1024-token top mixed bucket and that step runs eager. A token cap would be a new scheduler concept for a rare case; the comment states the limit instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice, code predictor kernels: say what the knob check and the partials limit cover (R2.6) unmask_knobs' finite-number check stops non-finite knobs only; finite extremes that overflow the reveal scores still fail in postprocess, so the docstring and the test comment no longer claim it stops that case. cp_embed_accumulate's ValueError names the partials-count limit as well as the row limit, since either trips it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan 2.2: keep the dit's heads whole over SP with shard_over_sp=False The node-level splits_heads flag is gone; the dit's one ragged attention declares the same thing on the resource. Under SP each rank still runs every head (the ranks split CFG branches), and TP still splits them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: bring precision and bucket comments in line with R2 fixes (R3.1-R3.3) R3.1: the mixed-step class comment no longer says the prompt cap always keeps a step in the captured buckets; it points at max_rows_in_combined, where the long-reference eager spill is described. R3.2: the served fp32 codec runs its cuDNN convs in TF32 as well as its GEMMs; the CUDA test case is the served code path at fp32, not served precision. R3.3: codec_bench parity compares against a whole-sequence decode at the served precision (TF32), not a float32 or reference decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: run a guided window as its own walk on one node too A single-node (or LLM-disaggregated) deployment ran a guided request as one image_gen walk and read each step's branch count off the loop counter against a CFGWindow in step metadata, with a guard for a counter that disagreed with the seeded latents. The CFG-parallel layouts already ran the window as its own walk with a fixed count. Every layout now does: a guided request runs image_gen_head (if the window starts late), then image_gen_guided2 or image_gen_guided3 on LLM alone, then image_gen (or image_decode when the window reaches the last step). Each step runs the same branches as before. DenoiseWalk gains `split` (a node per branch or not), ImageGenPhase loses its `branches`, and the per-step count, its guard, the CFG_BRANCHES kwarg and the cfg_window / image_gen_branches step metadata are deleted. The disaggregated configs and the docs' examples list the new walks for the image LLM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: fix stale codec parity and mixed-step wording R1.1, R1.2 (tail review): the codec_bench parity reference is always a float32 decode (TF32 GEMMs and convs), not "the served precision" under a bf16 config; and the model doc now notes that four long-reference clones can outgrow the top mixed token bucket and run eager, as the class comment already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say the api_switch_interval_s numbers predate #849's output-path rewrite The row still said the knob would be re-measured once #849 lands. It has landed and changed the message-thread and data-worker path the knob tunes, so the numbers are from before that change and the re-measure is still pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the head-sharding paragraph from the submodule attributes section It was there to document the removed splits_heads attribute; what is left repeats the RaggedAttentionConfig row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the TP x SP head-sharding test that main's sizing already passes Head sizing now equals main's, so the six cases pinned nothing this branch changes. The shard_over_sp=False test stays; the helper loses its recurrent branch, which only the deleted test used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: hold guidance KV only for the branches a request's window runs A request is guided when its CFG window has a step, not when a scale is above 1: a window with no step (a cfg_interval that excludes every timestep) runs no guidance, so it no longer prefills or decodes cfg_img or forks cfg_text. `_requires_cfg` is gone; process_prompt resolves the window before the layout and the conductor reads it from the stamped window. The layout reserves a branch label's denoise scratch only where the window runs that branch: cfg_img's only with image guidance, and neither branch's for an unguided request, which no layout denoises on them any more. A guided text-to-image request no longer holds a latent-sized cfg_img scratch it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the shard_over_sp sizing test, which main's sizing already passes Main already sizes a shard_over_sp=False ragged attention over the TP group, so the test pinned nothing this branch changes. The file's coverage goes back to main's plus the manager registry test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: comments state the contract instead of the code they replaced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: give the measured output shift of guided text-to-image Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_kv_admission: name the CFG walk BAGEL defines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3 and LingBot decode through wan_vae.FastDecoder and decode_dtype wan_vae.FastDecoder and decode_dtype had one user, Wan2.2, while Cosmos3's two decoder nodes and LingBot-World kept their own copies of the cuDNN >= 9.16 dtype gate, of the channels-last + DecoderStep setup and of the autocast-off decode_causal call. All of them now use the shared helpers, and the copies (_vae_decode_dtype, _fast_step, the streaming node's _step, LingBot's inline gate and _decode_step) are deleted. Numerics and switches are unchanged: Cosmos3 keeps COSMOS3_VAE_DECODE_FP32, its compile switch, its cast before the layout switch, the streaming step's non-deterministic compile (built only with the warm cache), its latents per call and cudnn_benchmark scope; LingBot keeps no override (decode_dtype's variable is now optional), and bf16 and channels-last only off reference numerics. Every node keeps the layout it set up, so FastDecoder's per-call layout check is a no-op for them. FastDecoder's autocast-off scope now follows the latents' device instead of naming cuda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan 2.2: drop the FFN GELU epilogue, which has no measurement of its own MLP(gelu_epilogue=) and Wan22DiT's fuse_ffn_gelu ran the FFN GELU in the up-projection's cuBLASLt epilogue through the private torch._addmm_activation. Nothing measured it apart from the ragged attention change it shipped with, so it goes, as unmeasured performance paths do: MLP is back to its plain form and the DiT builds without the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3 text encoder: refuse a row without text_len instead of never replaying The shared encoder node picks its captured (masked) graph from each row's text_len step metadata. A model that reused the node without setting it got every batch run eager, with no error. cg_key_info now raises ValueError for such a row; FLUX.2-klein and Z-Image set it at intake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: an explicit null is the default, and only integers pass intake image_output_format let png_compress_level: null through, and encode_image then failed on int(None) after the denoise and decode. encode_image now reads null as the default level, as it already did for output_compression. Intake also refused nothing that int() accepted, so True passed as 1 and 50.9 was truncated to 50; it now takes only integers (not bools) in range, as BAGEL's width and height check does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2: say the B=1 guard routes eager steps only, and drop "as before" comments A step that leases the AC rollout capture replays forward_batched at any batch size, B=1 included, whatever can_batch returns, so can_batch's B=1 guard covers eager steps only; its comment now says so. Two comments described the code by comparison with an earlier version ("as before the capture", "same as before") and now state what the code does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan_vae.decode_dtype: say that an override of 0 forces bf16 The override variable is two-way, as Wan2.2's and Cosmos3's were before it was shared (and as the Wan fast-decode test uses it): set, it forces fp32, or bf16 when it is 0/false/no/off. The docstring read as if 0 meant "not forced". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Higgs Audio: declare text_output so inline decode tokens skip the per-row postprocess Higgs's decode token rides inline, but without a declared text_output Model.postprocess_inline built a tensor and called postprocess for every row on every step. Declaring TextOutput(self._detokenizer), with no stop ids as before, gives it the memoized id-to-bytes path Qwen3-ASR and Whisper use; postprocess hands text to the base class. Bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: say the API_SWITCH_INTERVAL_S gain predates #849 docs/serving.rst says the number was measured before #849 and awaits a re-measure; the comment pointing at it now says the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: unambiguous float32 reference wording (tail R2 P3) The parenthetical read as if the server runs float32; the shipped config runs a bfloat16 codec. Say the reference matches a float32-codec server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: build its instance table from WorkerGraph.replica_ranks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: serve the whole numbers main served Intake took only Python ints, so a whole-number float (90.0) or a number in a JSON string ("90"), which main read through int() and served, became a 400. It now accepts any whole number int() reads in range, and still refuses bools, fractions such as 50.9, infinities and out-of-range values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: build only the window walks a deployment can run A guided window runs on the branch nodes on CFG-parallel deployments and on LLM alone elsewhere, never both, yet CFG-parallel builds still placed the unsplit image_gen_guided2/3 walks and listed them in every request's needed-label map. LLM_GRAPH_WALKS is now keyed by the layout, and the walk graphs and the label map both read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say why a split LLM still publishes both branch labels The comment dated from when a remote image_gen walk's branch count varied per step. Each walk now fixes its count; what keeps cfg_img published on a split deployment is a remote decode reading it and the single-node denoise walks naming their labels per request rather than per walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL and Cosmos3: drop loop-counter leftovers from comments and names denoise_loop's docstring still named the step's loop counter, which BAGEL no longer reads; declare_step's walk-branch narrowing kept a requires_cfg guard that active_labels already implies, and its guided-row set shadowed the module's guided(). Cosmos3's decode-dtype comment now gives both directions of its override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output: check only the chosen format's compression knob Intake refused a request whose other-format knob was out of range (a JPEG request with png_compress_level 10, a PNG request with output_compression 150), which main served because encode_image never reads it. Check only the knob the format applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say LLM-disaggregated, not split, in the branch-label comment "Split" means CFG-parallel in the neighbouring code (DenoiseWalk.split, LLM_GRAPH_WALKS). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: derive the loop-back hooks from device_loopback = None The stand-in stubbed the deleted _device_loopback and restated the three hooks DeviceLoopback now derives; clearing the declaration gives the same answers through the base class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: drop the stale "emit" from the stand-in's comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge-up fixes: DeviceLoopback in the withhold check, chunk-sink stand-in poll, postprocess_s on inline failure - _refuse_unroutable_withholds read TokenStop.loopback_signal, which main moved to ARNodeSubmodule.device_loopback; it now reads the signal from the submodule's DeviceLoopback, and its test declares one. - test_chunk_sink: its request profile stub lacked output_postprocess_s, so every pass raised before relaying a chunk; it uses a real RequestProfile, its transport's poll ends the pass (the loop no longer sleeps), and an error in the pass is raised instead of retried. - An inline value whose batched postprocess_inline call failed is charged its share of that call as well as its own retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check: the loop-back signal is exempt only on its own walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check test: the loop-back exemption is scoped to its walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-ASR/Omni AuT node: pin the replay's windows and packed-clip parity on the shared node Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: pin the unprojected depth path, drop the unused DepthAttention.forward The depth-loop parity test now runs at both Talker widths: 2048 through the input projection (Qwen3-TTS 1.7B) and 1024 without one (Qwen3-Omni, Qwen3-TTS 0.6B), where each step's input is the raw embedding and the first norm reads cp_embed_accumulate's per-chunk partials. The cp_embed_accumulate test checks those partials for both the raw and the projected input. DepthAttention.forward and its attend reference were the unfused layer, and nothing runs them: run_depth_layers reads the weights and attends through the predictor's DepthKV. The class stays as the layers' parameter container that takes none of the node's resources, which the binding test now checks directly. run_depth_layers' eager note keeps only the verified reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: say why the multi-token path stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: keep the compiled depth step free of per-position guards depth_step took the LM head index as an int and passed it to the disabled run_depth_layers, so under a static-int compile Dynamo compiled depth_step once per head (guards "head is None", "head == 0", ...): one frame per depth position, up to the recompile limit. depth_step itself is now the eager boundary, with run_depth_layers folded into it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: name the row cap a multi-token opening would need raised Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: name every eager reader of the depth ints; count the test's groups Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: compile the incremental decode through compile_regions The codec compiled its own decode with torch.compile(dynamic=False), which recompiled once per captured shape (4 chunk widths x 14 batch buckets = 55 compiles) and skipped the engine's inductor overrides and recompile accounting. On an empty cache that was ~16 of the ~19 minutes of a 1.7B cold start. The decode is now a compile region: the engine compiles it once, with dynamic shapes, before capture. H100, slice-6 head (empty Inductor/Triton cache, 26 CPUs): cold start 1119 -> 444 s, warm 540 -> 333 s. Closed loop audio-s/s c=1/16/32/64, 2 rounds each: push4a 18.4-18.5 / 137-145 / 209-212 / 281-284 -> 18.4 / 139-146 / 208-212 / 289-301; push4c (server on 2 CPUs) 18.6/159/237/370 -> 18.4-18.5 / 160 / 237-251 / 371-379. Pinning the chunk width static (one compile per width, batch dynamic) measured no faster and started in 765 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: test and bench compile the decode as the engine does The codec's decode is compiled by the engine's compile_regions pass, so a CodecSubmodule built directly ran it eagerly. The per-node body of Engine._compile_regions becomes compile_node_regions, which the engine loop, the streamed-codec test's CUDA case and codec_bench's load_codec all call. The compile_regions docstring and models.rst now say what dynamic=True gives: a few graphs (batch 1 and width 1 specialise) for every chunk width and batch bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Codec and compile_node_regions docstrings: the measured cold start, no dynamic=None aside Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: start the codec stream with a 2-frame chunk The deployment config's ramp began with a 4-frame chunk, so a stream's first audio waited for four Talker steps. [2, 4, 8, 16] sends it after two, and the 160 ms first chunk still covers the time until the next one arrives under load. [1, 2, 5, 16] is documented as the lower-latency opt-in. H100 (push4a and push4c), closed loop, c=64: TTFA p50 84-95 -> 63-78 ms (c=1 33 -> 25 ms), no playback stalls either way, audio-s/s 379-424 vs 363-416 over 4-6 rounds each (same within noise). Open-loop SLO capacity (TTFA <= 200 ms, stall < 10 ms): Seed-TTS 80 @90/99 and passages 28 @90 either way. WER (Whisper large-v3, 128 sentences): 2.09% -> 1.73%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * qwen3tts_1p7b config: the header names the 2-frame start; date the TTFA numbers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: the bf16 codec is measured on H100 too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B bf16 codec: state what the H100 runs measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS bf16 codec: tie each H100 WER to its chunk schedule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attention: select_last_hidden is one eager boundary, so prefill buckets don't recompile its frame The disabled `qo_indptr_buf` call broke the graph inside `select_last_hidden`, leaving a resume frame that holds both the bucket's static token count (`hidden`) and its row count (`qo_indptr`). Captured forwards compile with dynamic=False, so that frame took one cache entry per (tokens, rows) bucket: Qwen3.5's 9 token buckets by 7 prefill/mixed row counts made 63, and the 13-step prefill ladder made 91, past the recompile limit of 84. Disabling the whole method moves the break to its call site, where the caller already breaks; no compiled frame holds the pair. Same for the MLA and XPU managers, which share the shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL takes its prefill's last rows through select_last_hidden; test docstrings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * GDN prefill run: state the zeroed-tail precondition the mixed wrapper also meets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: key the backbone's step outputs by the engine's row ids The packed backbone keyed its outputs by CanvasItem.request_id, which prepare_inputs now fills from fwd_info.request_id, the wire request id. The engine looks each row's outputs up by its worker-local handle, found none, and every OmniVoice request failed in postprocess with KeyError 'c_logits'. forward_batched now keys rows by engine_inputs.request_ids, as the other batched submodules do, and CanvasItem carries no request id at all, so the canvas can't be keyed by the wrong one again. The unread request_id kwarg goes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: keep the 8192 Thinker ladder off the 80 GB default qwen3omni_2gpu.yaml, the CLI default and the config sized for 80 GB GPUs, ran the Thinker's prefill ladder to 8192. On an H100 its 4096-8192 captures don't fit beside the 1,024-page KV pool (59 of 86 Thinker buckets captured), so those steps run eagerly and a deployment that requires accelerator graphs refuses to start. The default goes back to the model's 128-2048 ladder, as on the base. The 8192 ladder, measured on 2xH200, stays in qwen3omni_2gpu_large_kv.yaml, which is already the H200 variant; its header, the default's, the submodule comment and the Qwen3-Omni notes say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B configs: back to the base's codec, chunk ramp and pools (owned by #978) Kirill's #978 sets these himself (the [4, 8, 16] ramp, the bf16 codec, the codec_state slots, the GC knob and their comments), so this PR leaves both 1.7B configs as its base has them. This reverts the config hunks of fce78c050, f63eb7994, aba669bac, 9ef1d1e66, 08c8b2350, e4450ea53 and 5428cca16, including the [2, 4, 8, 16] first-chunk schedule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the incremental codec, mixed walk and fused depth loop (owned by #978) Kirill's #978 makes the same Qwen3-TTS changes (from the same stacked-head work): the incremental codec with its codec_state pool and chunk knobs, the combined prefill+decode walk and its captures, the fixed prefill embeddings, and a fused depth loop (cp_linear with folded norms, cp_attend, fold_depth_weights) beside the codec's own compile. This PR leaves all of it to him: Qwen3-TTS, the shared code predictor kernels and Qwen3-Omni's two call sites go back to the base's code, and so do their tests, the real-weights codec test, the Qwen3-TTS and Qwen3-Omni code-predictor paragraphs in docs/models.rst, and the codec bench, which timed the removed decoder. That reverts fce78c050 and the fixes made to its code since: 59d1b3854, f5b0422ff, 2f6333a90, e59639f21, 251d13e34, add26fbc7, f7887b71a, 4d9f54584, e6b2a2fef, a2331d014, ce5693c3d, 1158293d0, 8ebf63986, 613160070, b13ab78e4, 9a588f978, 26ef71b6e, 15413c499, 5fea4023e, ab3350e18, 4f935c1f6, a8c537e7d, c4ed93f0c, 88b5b68de, 5bfeeac60, a5687bdd7, the code-predictor half of 4791cb3ca and the Qwen3-TTS halves of f63eb7994, 7906d0902 and 13b35b17a. Kept: db8b241c7, a decode row reads its text token by its loop iteration rather than a counter a re-prepared step advances twice. The counter is the base's (and main's) code, which #978 leaves as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: drop compile_node_regions, which only the removed codec test and bench called 13b35b17a and 7a8be7397 lifted Engine._compile_regions' body into a module-level compile_node_regions so the Qwen3-TTS codec test and bench could compile the decode as the engine does. Both are gone with the incremental codec (owned by #978), so _compile_regions takes its body back, as on the base. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: don't claim the remaining captures fit 80 GB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin video decode threads to the CPUs the plan leaves under pinning A case with a pinned CPU fails if video_decode_threads goes back to usable_cpus(); the MSTAR_STEP_THREAD_CORE row names FFmpeg's per-decode threads among what the pins shrink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ragged attention: a wrapper skips replanning the layout it last planned The dense manager kept the (cu_q, cu_k, causal) each of its wrappers was last planned with and skipped an unchanged plan; the ragged self- and cross-attention managers replanned every step, so a denoise loop paid FlashInfer's plan (a kernel and a stream sync) once per step for the same segments. The skip now lives in the wrapper both use (_RaggedPrefillBase._plan): each wrapper is one (label, slot) or one (bucket, slot, label), so the schedule it keeps is the one the step reading it planned. The dense manager's own table goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the attention and CFG-parallel timings as before #963 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: leave the idle wait to its own change The conductor's 1 ms nap becomes a wait on the poll in a change of its own, which waits until the next timed check instead of a fixed 1 ms. This branch's capped poll, its failed-pass back-off, the conductor stand-ins' poll_for_messages and their test go, so those files are main's again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server tests: the failed-wait test pins its back-off; stand-in waits as main has them The message loop always passes a computed timeout, so the 1 ms defaults left by the fixed poll go: the chunk-sink stand-in takes main's again and the failed-wait stand-in takes it as required. That test now asserts the 10 ms back-off, without which a persistent wait error would spin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Preprocess pool: at most the CPUs the plan leaves unpinned Every other API-server pool is sized from the CPU plan; the preprocess pool took its declared or configured size uncapped, so under step_thread_core a model declaring 8 workers could run them, each with its own FFmpeg threads, on fewer free CPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: drain routes before an input only when it prepares here The drain keeps a call's held items from waiting behind a preprocess that may take seconds. An input prepared on the pool only stores and sends when it is handed on, and each pass already drains routes at its top, so it no longer takes a drain of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: say what a RecurrentStateSpec max_slots override must leave Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: compile the block loop, so a step past the captured buckets runs compiled disable_torch_compile left the block loop uncompiled whenever it did not replay: a long video (over 16384 patches) ran the blocks' fp32 rope, norms and activations as eager kernels, slower than the whole-forward compile it replaced. The block loop is now a compile region like the embed and the merger, dynamic in the patch count; it compiles in its capture's warm-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop the api_switch_interval_s knob Its gain was measured before #849 and #939 changed the message-thread and data-worker path, and the re-measure it was waiting on never ran. The serving-config key, its validation, docs rows and test go; workers keep main's MSTAR_PY_SWITCH_INTERVAL_SEC unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the GIL switch-interval knob's model defaults (the knob is removed in #858) Reverts #859's additions on top of #858's api_switch_interval_s: the Model.API_SWITCH_INTERVAL_S class default, Qwen3.5's 0.001 override, the model argument to entrypoint._set_switch_interval, the doc wording and the test fakes that carried the attribute. Every file #858 owns is back to #858's text, so #858's removal of the knob merges in cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: the TP2 publish test's KV managers hold one rank's head KVManager shards its config in build, so the test's direct constructor shards it the same way and each rank moves its own shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop warmup_requests' concurrency option No config sets it, so its parsing, docs and tests covered a dead path. Each entry still goes out once per data-parallel replica, now as warmup-<spec>.<copy>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: a request too large for a replay gets an encoder step of its own Up to 8 requests share an encoder step, and a step that cannot replay as one runs them back to back (can_batch). For requests that are each past the captured buckets (long videos) that only held every request's embeddings, and so their prefills, until the last one was encoded: under load the prefills bunched into waves (TTFT +30-50% on 60 s video at c=16/32, decode almost never interleaved with a prefill). The vision node now splits its steps by capture key, a request too large for a replay being a group of its own, and the engine lets a node whose only capture is a piecewise region split that way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: Canary's decode cap leaves the prefill's token in the budget #794's cap-versus-stop-check agreement tests for Canary and Canary-Qwen assumed a stop check that counts max_tokens; TokenStop leaves the budget to the loop's cap, so they pin the cap instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: last_prefill_flag for a prompt prefilled over several passes; GLM-5.2 test comment names the loop cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision split: review cleanups - Engine.capture_group asks Engine.splits_by_capture_key; the module helper existed only for a test fake, which now builds a real Engine. - The split docstrings say a piecewise region is not tied to a walk, and that on a piecewise-only node the key only groups the scheduler's rows. - Stale 'eager' comments on the vision tower: past the top bucket the block loop runs compiled, uncaptured. - docs/models.rst: drop the one-boot measurement of a different overlay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chatterbox, Higgs Audio tests: a stop id at prefill ends the partition #858 moved TokenStop's prefill_walks onto END_PARTITION; the step closing a reply that ended at prefill no longer exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Orpheus test: the stop check's fake info names its walk TokenStop now reads each stopping row's walk to choose between its loop and END_PARTITION. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 9B DP2 config: the replica pick follows #920's router The conductor now weighs a predicted prefix hit beside the live requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: the vision split's capture-only scope, Qwen3.5 configs' start-up compile, Qwen3-Omni's encoder header - The Qwen3.5 vision split applies while the block loop is captured, and requests that fit alone but not together still run one after another in one step; cg_key_info's and capture_group's docstrings say a piecewise-only node's key is a scheduler group, not a bucket. - The Qwen3.5 configs' warmup comments: the vision regions compile at start-up. - Qwen3-Omni's submodule header takes main's line for its audio encoder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the timings by the DiT step they measured, not a PR number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: an earlier version of the DiT block, not an earlier DiT step "DiT step" reads as a denoising step in this section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: a capped deployment holds cuDNN's capture search to its share capture_cudnn_benchmark's search runs in capture sizing since the sized warm-ups took the node's scope, and it tries workspaces as large as the GPU has free: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized, and an over-cap try is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the preprocess pool's CPU cap does not count a prepare's FFmpeg threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chunk policy: leave the longer primed first chunk to #977 #977 makes the same change (ScheduledLeftContextChunkPolicy.prime past left_context, and its test), and its consumer, the incremental Qwen3-TTS codec, is no longer in this stack: every primer here passes at most left_context (Qwen3-TTS's lead is min(ref_frames, left_context_frames)). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ViT patch embed: the conv timing in its comment is Qwen3.5's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: _search_cap's docstring separates a skipped search try from a failed sizing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B DP2 config: say the vision regions compile at start-up Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: capture_cudnn_benchmark's search runs in capture sizing, held to a capped deployment's share Sizing runs a node's warm-up forwards; the capture after it warms up with no forward, so the scope around warmup_and_capture covered no conv on CUDA and Kokoro's graphs kept cuDNN's heuristic picks. The scope now wraps sizing. The search tries workspaces as large as the GPU has free, and the allocator keeps them until sizing's empty_cache: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized and restored to its previous value after; an over-cap search try is skipped, and the node's own sizing fails the boot when its eager step does not fit in the share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: capture the decode buckets uncompiled, as on main Compiling the seven decode buckets took a cold start from 180 s to 1372 s on H100, with no measured throughput to show for it. The 64-row bucket and the packed prefill captures stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: TF32 codec decode is the model's default The shipped config set codec_exact_fp32: false while the model defaulted to the exact decode, so other Fish configs decoded differently. The default now lives in the model; codec_exact_fp32: true still opts into the exact decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: open the depth loop with position 0 alone Both Talkers opened with one-token passes on main; the two-token opening changed greedy codes for both, and a one-token pass keeps every pass at bs rows. The opening is now main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the 24- and 48-row decode buckets They took the 1.7B's cold start from 204 s to 307 s on H100, and a step at 64 rows costs about what it does at 32, so padding into 32 and 64 is nearly free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * run_layers: the residual stream starts from its own copy The stream was the caller's hidden_states, overwritten in place from layer 0, so a caller had to know to pass a copy (the Qwen3-TTS Talker did for its packed replay's static input). One [tokens, hidden] copy per forward, made inside the forward so a capture records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice silence test: pydub is the oracle, installed with dev The comparison imported the reference package, which CI does not install, so it always skipped there. It now restates the reference's trim over pydub.silence (raw-sample segments, no ffmpeg), with pydub in the dev extras, and covers the rates and edge cases the self-referential trailing trim test stood in for; that test is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: each Talker keeps its own embedding-sum dtype The shared depth loop summed a frame's codec embeddings in fp32 for both Talkers. Qwen3-TTS accumulated in the model dtype, rounding after every add, so its greedy codes could differ at near-ties. ResidualCodePredictor takes embed_sum_dtype from its Talker: None (the embeddings' dtype) for Qwen3-TTS, fp32 for Qwen3-Omni, whose fp32 sum is rounded once to the model dtype as before. The accumulate kernel adds in fp32 and rounds to the sum's dtype, as torch's add_ does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: Qwen3-TTS's embedding sum, the cuDNN search in capture sizing; audioop-lts for pydub on 3.13 - Qwen3-TTS's per-add rounding is the sum this Talker had in M*; the reference sums in fp32 and rounds once. - adding_models.rst and Kokoro say the cuDNN search runs in capture sizing. - pydub 0.25.1 imports audioop, which Python 3.13 removed: the dev extras add audioop-lts there, so the silence oracle does not skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStop: withholding outputs and the runtime check refuse each other The graph runtime's stop check never calls check_stop, which is what withholds a stop step's outputs, so a TokenStop with both would drop the withhold silently. It is a frozen dataclass now, and that combination raises at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_knobs: take main's version back now that ingraph_decode_rope stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 CFG-parallel: the scaffold keeps ragged_for, docs say a rank runs its branch over the batch Since the dit moved onto the scaffold's denoise loop, DenoiseLoopSubmodule is RaggedAttentionMixin's only user, so the mixin folds back into it as on main. The declare_step test keeps only what the per-rank segment test does not cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: quote the guided text-to-image PSNR and the configuration of each number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan22 extra: install flashinfer-python for the default ragged attention Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2: leave CFG-parallel to the cfg_size axis Drop wan22_cfg_parallel_2gpu.yaml, my_rows and the dit's sp-group plumbing, which used the SP slot to mean a CFG split. The cfg_size axis carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Z-Image-Turbo: capture batch sizes 1, 2, 4, 8; every size up to 8 is a commented opt-in Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2-AC test: a rollout request's stamped clip shape keys the encoder capture Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B configs: keep main's 2176 bucket for a mixed step The 9B ladders replace the default, so they lost the slack bucket that lets a mixed step of four 512-token prompts and its decode rows replay at 2176 rather than pad to 4096. Adding it keeps the compile count at 82, strictly under Dynamo's limit of 84; the 9B ladders keep 3072-7168 in place of main's 4352 and 8704. The mixed-step bucket and the recompile limit are now checked on every ladder a Qwen3.5 config ships, not only the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sampler RNG offset test: both captured kernels, each against a draw it can match At V=64 the captured step takes fused_sample, whose Triton RNG never agrees with the eager step's FlashInfer draw, so the test passed whether or not the two steps shared an offset. The eager case now draws its captured step through FlashInfer too, and a second case runs two captured steps on fused_sample, which reads the same offset buffer the replay advances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: refuse a bad preprocess_workers before the thread starts The thread reads preprocess_workers as it starts, so a value it cannot take ("auto") ended it while the server came up, and every request then waited for its timeout; 0 or a negative quietly meant inline, and true meant 1. PreprocessWorker now refuses anything but a positive integer, on the caller's thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStepNode: copy only the outputs the per-request stop check reads The per-request fallback aliased text_inputs as new_token with a dict splat, which reads every output name, so the lazy host outputs copied all of them to the host. A ChainMap over the outputs reads only what the check reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check every shipped prefill ladder, not only Qwen3.5's The ladder tests read only configs/qwen3_5*.yaml, so the Qwen3-Omni large_kv config's ladder was never checked. Every config that sets model_kwargs.prefill_token_buckets is now built into its own model's prefill capture; the Qwen3.5-specific checks keep to Qwen3.5 configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: pin the config-to-model plumbing, drop the Qwen3.5 repeat Every shipped ladder's model must take prefill_token_buckets as its own argument (it would otherwise vanish into **kwargs); the Thinker's capture check stays for Qwen3-Omni, whose ladder no other test builds. Co-Authored-By: Claude Opus 5…

…r does Since #725 the worker ingests a request once and RecurrentStatePool's ingest_request starts the request over, so the tests' codec driver, which ingested every row on every step, dropped each stream's slot without freeing it and filled the pool after a few steps. It now ingests a stream on its first step and ends it with remove.

One conflict, docs/models.rst: the Qwen3-Omni numerics bullet is this branch's, and main's new Thinker prefill-ladder bullet follows it.

@90

* Qwen3.5 9B DP2 config: give the GDN pool prefix snapshot slots (RM.1) The dp2 config was written before #877 and has no `gdn_state.prefix_slots`, so the recurrent pool can't serve a cached prefix and the engine keeps the LLM node's KV prefix cache closed with it. Each replica runs at world size 1, so it takes the same 128 snapshot slots as qwen3_5_9b.yaml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests, docs: run the Qwen3-TTS codec parity with TF32 off for matmuls too; scope its claim (R2.1) Importing mstar.engine sets float32_matmul_precision('high'), so turning off only cuDNN's TF32 left the codec's GEMM-lowered transposed and kernel-1 convs in TF32 on the CUDA cases. Both decodes now run under fp32_precision_scope(Fp32Precision("highest", cudnn_allow_tf32=False)). The one-PCM16-step claim in the model's comment and docs/models.rst is scoped to float32 with TF32 off (as on CPU); the served CUDA codec runs its GEMMs in TF32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: drop --tf32, which could not turn TF32 off (R2.5) The flag was applied before the mstar imports set float32_matmul_precision ('high'), so every run used TF32 matmuls, as the served decode does, and a run without it was mislabelled as plain float32. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: drop the cp_linear tile test that restates the shared-memory formula (R2.3) It asserted _cp_linear_tile's own stage_bytes expression, so it could not catch a formula that undercounts what Triton allocates. The real check is test_cp_linear_matches_the_unfused_ops compiling on a 99 KB (sm_86/sm_89) GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: drop Talker asserts on can_use_accelerator_graphs, a hook no engine path calls (R2.4) R1.15 deleted the Talker's override, so these three asserts only tested the base class's default; their comment described the deleted behaviour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: say that Base's four-prompt mixed cap can spill long clones to eager (R2.2) The cap counts prompts, not tokens, so four clones with references of about 18 s or more outgrow the 1024-token top mixed bucket and that step runs eager. A token cap would be a new scheduler concept for a rare case; the comment states the limit instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice, code predictor kernels: say what the knob check and the partials limit cover (R2.6) unmask_knobs' finite-number check stops non-finite knobs only; finite extremes that overflow the reveal scores still fail in postprocess, so the docstring and the test comment no longer claim it stops that case. cp_embed_accumulate's ValueError names the partials-count limit as well as the row limit, since either trips it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan 2.2: keep the dit's heads whole over SP with shard_over_sp=False The node-level splits_heads flag is gone; the dit's one ragged attention declares the same thing on the resource. Under SP each rank still runs every head (the ranks split CFG branches), and TP still splits them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: bring precision and bucket comments in line with R2 fixes (R3.1-R3.3) R3.1: the mixed-step class comment no longer says the prompt cap always keeps a step in the captured buckets; it points at max_rows_in_combined, where the long-reference eager spill is described. R3.2: the served fp32 codec runs its cuDNN convs in TF32 as well as its GEMMs; the CUDA test case is the served code path at fp32, not served precision. R3.3: codec_bench parity compares against a whole-sequence decode at the served precision (TF32), not a float32 or reference decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compile_utils: docstring covers the VAE decode side and group_norm (pre-push review P3-1) The module docstring named only the flow transformers and decode_utils as callers; group_norm also asks kept_exact for the VAE's fused GroupNorm+SiLU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: run a guided window as its own walk on one node too A single-node (or LLM-disaggregated) deployment ran a guided request as one image_gen walk and read each step's branch count off the loop counter against a CFGWindow in step metadata, with a guard for a counter that disagreed with the seeded latents. The CFG-parallel layouts already ran the window as its own walk with a fixed count. Every layout now does: a guided request runs image_gen_head (if the window starts late), then image_gen_guided2 or image_gen_guided3 on LLM alone, then image_gen (or image_decode when the window reaches the last step). Each step runs the same branches as before. DenoiseWalk gains `split` (a node per branch or not), ImageGenPhase loses its `branches`, and the per-step count, its guard, the CFG_BRANCHES kwarg and the cfg_window / image_gen_branches step metadata are deleted. The disaggregated configs and the docs' examples list the new walks for the image LLM. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: fix stale codec parity and mixed-step wording R1.1, R1.2 (tail review): the codec_bench parity reference is always a float32 decode (TF32 GEMMs and convs), not "the served precision" under a bf16 config; and the model doc now notes that four long-reference clones can outgrow the top mixed token bucket and run eager, as the class comment already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Say the api_switch_interval_s numbers predate #849's output-path rewrite The row still said the knob would be re-measured once #849 lands. It has landed and changed the message-thread and data-worker path the knob tunes, so the numbers are from before that change and the re-measure is still pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the head-sharding paragraph from the submodule attributes section It was there to document the removed splits_heads attribute; what is left repeats the RaggedAttentionConfig row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the TP x SP head-sharding test that main's sizing already passes Head sizing now equals main's, so the six cases pinned nothing this branch changes. The shard_over_sp=False test stays; the helper loses its recurrent branch, which only the deleted test used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: hold guidance KV only for the branches a request's window runs A request is guided when its CFG window has a step, not when a scale is above 1: a window with no step (a cfg_interval that excludes every timestep) runs no guidance, so it no longer prefills or decodes cfg_img or forks cfg_text. `_requires_cfg` is gone; process_prompt resolves the window before the layout and the conductor reads it from the stamped window. The layout reserves a branch label's denoise scratch only where the window runs that branch: cfg_img's only with image guidance, and neither branch's for an unguided request, which no layout denoises on them any more. A guided text-to-image request no longer holds a latent-sized cfg_img scratch it never writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the shard_over_sp sizing test, which main's sizing already passes Main already sizes a shard_over_sp=False ragged attention over the TP group, so the test pinned nothing this branch changes. The file's coverage goes back to main's plus the manager registry test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: comments state the contract instead of the code they replaced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: give the measured output shift of guided text-to-image Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_kv_admission: name the CFG walk BAGEL defines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3 and LingBot decode through wan_vae.FastDecoder and decode_dtype wan_vae.FastDecoder and decode_dtype had one user, Wan2.2, while Cosmos3's two decoder nodes and LingBot-World kept their own copies of the cuDNN >= 9.16 dtype gate, of the channels-last + DecoderStep setup and of the autocast-off decode_causal call. All of them now use the shared helpers, and the copies (_vae_decode_dtype, _fast_step, the streaming node's _step, LingBot's inline gate and _decode_step) are deleted. Numerics and switches are unchanged: Cosmos3 keeps COSMOS3_VAE_DECODE_FP32, its compile switch, its cast before the layout switch, the streaming step's non-deterministic compile (built only with the warm cache), its latents per call and cudnn_benchmark scope; LingBot keeps no override (decode_dtype's variable is now optional), and bf16 and channels-last only off reference numerics. Every node keeps the layout it set up, so FastDecoder's per-call layout check is a no-op for them. FastDecoder's autocast-off scope now follows the latents' device instead of naming cuda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan 2.2: drop the FFN GELU epilogue, which has no measurement of its own MLP(gelu_epilogue=) and Wan22DiT's fuse_ffn_gelu ran the FFN GELU in the up-projection's cuBLASLt epilogue through the private torch._addmm_activation. Nothing measured it apart from the ragged attention change it shipped with, so it goes, as unmeasured performance paths do: MLP is back to its plain form and the DiT builds without the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3 text encoder: refuse a row without text_len instead of never replaying The shared encoder node picks its captured (masked) graph from each row's text_len step metadata. A model that reused the node without setting it got every batch run eager, with no error. cg_key_info now raises ValueError for such a row; FLUX.2-klein and Z-Image set it at intake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: an explicit null is the default, and only integers pass intake image_output_format let png_compress_level: null through, and encode_image then failed on int(None) after the denoise and decode. encode_image now reads null as the default level, as it already did for output_compression. Intake also refused nothing that int() accepted, so True passed as 1 and 50.9 was truncated to 50; it now takes only integers (not bools) in range, as BAGEL's width and height check does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2: say the B=1 guard routes eager steps only, and drop "as before" comments A step that leases the AC rollout capture replays forward_batched at any batch size, B=1 included, whatever can_batch returns, so can_batch's B=1 guard covers eager steps only; its comment now says so. Two comments described the code by comparison with an earlier version ("as before the capture", "same as before") and now state what the code does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan_vae.decode_dtype: say that an override of 0 forces bf16 The override variable is two-way, as Wan2.2's and Cosmos3's were before it was shared (and as the Wan fast-decode test uses it): set, it forces fp32, or bf16 when it is 0/false/no/off. The docstring read as if 0 meant "not forced". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Higgs Audio: declare text_output so inline decode tokens skip the per-row postprocess Higgs's decode token rides inline, but without a declared text_output Model.postprocess_inline built a tensor and called postprocess for every row on every step. Declaring TextOutput(self._detokenizer), with no stop ids as before, gives it the memoized id-to-bytes path Qwen3-ASR and Whisper use; postprocess hands text to the base class. Bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: say the API_SWITCH_INTERVAL_S gain predates #849 docs/serving.rst says the number was measured before #849 and awaits a re-measure; the comment pointing at it now says the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: unambiguous float32 reference wording (tail R2 P3) The parenthetical read as if the server runs float32; the shipped config runs a bfloat16 codec. Say the reference matches a float32-codec server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: build its instance table from WorkerGraph.replica_ranks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: serve the whole numbers main served Intake took only Python ints, so a whole-number float (90.0) or a number in a JSON string ("90"), which main read through int() and served, became a 400. It now accepts any whole number int() reads in range, and still refuses bools, fractions such as 50.9, infinities and out-of-range values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: build only the window walks a deployment can run A guided window runs on the branch nodes on CFG-parallel deployments and on LLM alone elsewhere, never both, yet CFG-parallel builds still placed the unsplit image_gen_guided2/3 walks and listed them in every request's needed-label map. LLM_GRAPH_WALKS is now keyed by the layout, and the walk graphs and the label map both read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say why a split LLM still publishes both branch labels The comment dated from when a remote image_gen walk's branch count varied per step. Each walk now fixes its count; what keeps cfg_img published on a split deployment is a remote decode reading it and the single-node denoise walks naming their labels per request rather than per walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL and Cosmos3: drop loop-counter leftovers from comments and names denoise_loop's docstring still named the step's loop counter, which BAGEL no longer reads; declare_step's walk-branch narrowing kept a requires_cfg guard that active_labels already implies, and its guided-row set shadowed the module's guided(). Cosmos3's decode-dtype comment now gives both directions of its override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output: check only the chosen format's compression knob Intake refused a request whose other-format knob was out of range (a JPEG request with png_compress_level 10, a PNG request with output_compression 150), which main served because encode_image never reads it. Check only the knob the format applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say LLM-disaggregated, not split, in the branch-label comment "Split" means CFG-parallel in the neighbouring code (DenoiseWalk.split, LLM_GRAPH_WALKS). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: derive the loop-back hooks from device_loopback = None The stand-in stubbed the deleted _device_loopback and restated the three hooks DeviceLoopback now derives; clearing the declaration gives the same answers through the base class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: drop the stale "emit" from the stand-in's comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge-up fixes: DeviceLoopback in the withhold check, chunk-sink stand-in poll, postprocess_s on inline failure - _refuse_unroutable_withholds read TokenStop.loopback_signal, which main moved to ARNodeSubmodule.device_loopback; it now reads the signal from the submodule's DeviceLoopback, and its test declares one. - test_chunk_sink: its request profile stub lacked output_postprocess_s, so every pass raised before relaying a chunk; it uses a real RequestProfile, its transport's poll ends the pass (the loop no longer sleeps), and an error in the pass is raised instead of retried. - An inline value whose batched postprocess_inline call failed is charged its share of that call as well as its own retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check: the loop-back signal is exempt only on its own walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check test: the loop-back exemption is scoped to its walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-ASR/Omni AuT node: pin the replay's windows and packed-clip parity on the shared node Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: pin the unprojected depth path, drop the unused DepthAttention.forward The depth-loop parity test now runs at both Talker widths: 2048 through the input projection (Qwen3-TTS 1.7B) and 1024 without one (Qwen3-Omni, Qwen3-TTS 0.6B), where each step's input is the raw embedding and the first norm reads cp_embed_accumulate's per-chunk partials. The cp_embed_accumulate test checks those partials for both the raw and the projected input. DepthAttention.forward and its attend reference were the unfused layer, and nothing runs them: run_depth_layers reads the weights and attends through the predictor's DepthKV. The class stays as the layers' parameter container that takes none of the node's resources, which the binding test now checks directly. run_depth_layers' eager note keeps only the verified reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * FLUX.2 klein: one shared kept-module forward, so blocks don't recompile per instance exclude_from_compile set a functools.partial forward on each kept 1-D RMSNorm. Dynamo ID-guards a partial set on an instance, so the single-stream block frame klein's blocks share failed its guards once per block until the recompile limit, and the remaining blocks ran eagerly: -6 to -10% img/s at c=1-4 on klein 4B and 9B against main. A kept module now changes class instead, to a subclass made once per (class, path) whose forward is the class's: the exact RMSNorm op for a plain 1-D nn.RMSNorm, torch.compiler.disable of the base forward for every other kept module (the same graph break as before). compile_kept_exact is a class attribute of that subclass, so kept_exact (Z-Image's fused q/k norm + rotary, the VAE's norm_act) still answers. Outputs are unchanged: the same exact_rms_norm call, the same disabled base forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compile_utils: state the guard that recompiled per block Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: say why the multi-token path stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: keep the compiled depth step free of per-position guards depth_step took the LM head index as an int and passed it to the disabled run_depth_layers, so under a static-int compile Dynamo compiled depth_step once per head (guards "head is None", "head == 0", ...): one frame per depth position, up to the recompile limit. depth_step itself is now the eager boundary, with run_depth_layers folded into it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compile_utils: every block passes the same guards, not one check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: name the row cap a multi-token opening would need raised Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: name every eager reader of the depth ints; count the test's groups Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: compile the incremental decode through compile_regions The codec compiled its own decode with torch.compile(dynamic=False), which recompiled once per captured shape (4 chunk widths x 14 batch buckets = 55 compiles) and skipped the engine's inductor overrides and recompile accounting. On an empty cache that was ~16 of the ~19 minutes of a 1.7B cold start. The decode is now a compile region: the engine compiles it once, with dynamic shapes, before capture. H100, slice-6 head (empty Inductor/Triton cache, 26 CPUs): cold start 1119 -> 444 s, warm 540 -> 333 s. Closed loop audio-s/s c=1/16/32/64, 2 rounds each: push4a 18.4-18.5 / 137-145 / 209-212 / 281-284 -> 18.4 / 139-146 / 208-212 / 289-301; push4c (server on 2 CPUs) 18.6/159/237/370 -> 18.4-18.5 / 160 / 237-251 / 371-379. Pinning the chunk width static (one compile per width, batch dynamic) measured no faster and started in 765 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: test and bench compile the decode as the engine does The codec's decode is compiled by the engine's compile_regions pass, so a CodecSubmodule built directly ran it eagerly. The per-node body of Engine._compile_regions becomes compile_node_regions, which the engine loop, the streamed-codec test's CUDA case and codec_bench's load_codec all call. The compile_regions docstring and models.rst now say what dynamic=True gives: a few graphs (batch 1 and width 1 specialise) for every chunk width and batch bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Codec and compile_node_regions docstrings: the measured cold start, no dynamic=None aside Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: start the codec stream with a 2-frame chunk The deployment config's ramp began with a 4-frame chunk, so a stream's first audio waited for four Talker steps. [2, 4, 8, 16] sends it after two, and the 160 ms first chunk still covers the time until the next one arrives under load. [1, 2, 5, 16] is documented as the lower-latency opt-in. H100 (push4a and push4c), closed loop, c=64: TTFA p50 84-95 -> 63-78 ms (c=1 33 -> 25 ms), no playback stalls either way, audio-s/s 379-424 vs 363-416 over 4-6 rounds each (same within noise). Open-loop SLO capacity (TTFA <= 200 ms, stall < 10 ms): Seed-TTS 80 @90/99 and passages 28 @90 either way. WER (Whisper large-v3, 128 sentences): 2.09% -> 1.73%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * qwen3tts_1p7b config: the header names the 2-frame start; date the TTFA numbers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: the bf16 codec is measured on H100 too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: klein's exact q/k norm, measured with the shared-forward fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B bf16 codec: state what the H100 runs measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: klein's shared kept-norm forward, measured against a forward per block Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS bf16 codec: tie each H100 WER to its chunk schedule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: kept norms share a forward per class Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attention: select_last_hidden is one eager boundary, so prefill buckets don't recompile its frame The disabled `qo_indptr_buf` call broke the graph inside `select_last_hidden`, leaving a resume frame that holds both the bucket's static token count (`hidden`) and its row count (`qo_indptr`). Captured forwards compile with dynamic=False, so that frame took one cache entry per (tokens, rows) bucket: Qwen3.5's 9 token buckets by 7 prefill/mixed row counts made 63, and the 13-step prefill ladder made 91, past the recompile limit of 84. Disabling the whole method moves the break to its call site, where the caller already breaks; no compiled frame holds the pair. Same for the MLA and XPU managers, which share the shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL takes its prefill's last rows through select_last_hidden; test docstrings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * GDN prefill run: state the zeroed-tail precondition the mixed wrapper also meets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: key the backbone's step outputs by the engine's row ids The packed backbone keyed its outputs by CanvasItem.request_id, which prepare_inputs now fills from fwd_info.request_id, the wire request id. The engine looks each row's outputs up by its worker-local handle, found none, and every OmniVoice request failed in postprocess with KeyError 'c_logits'. forward_batched now keys rows by engine_inputs.request_ids, as the other batched submodules do, and CanvasItem carries no request id at all, so the canvas can't be keyed by the wrong one again. The unread request_id kwarg goes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: keep the 8192 Thinker ladder off the 80 GB default qwen3omni_2gpu.yaml, the CLI default and the config sized for 80 GB GPUs, ran the Thinker's prefill ladder to 8192. On an H100 its 4096-8192 captures don't fit beside the 1,024-page KV pool (59 of 86 Thinker buckets captured), so those steps run eagerly and a deployment that requires accelerator graphs refuses to start. The default goes back to the model's 128-2048 ladder, as on the base. The 8192 ladder, measured on 2xH200, stays in qwen3omni_2gpu_large_kv.yaml, which is already the H200 variant; its header, the default's, the submodule comment and the Qwen3-Omni notes say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B configs: back to the base's codec, chunk ramp and pools (owned by #978) Kirill's #978 sets these himself (the [4, 8, 16] ramp, the bf16 codec, the codec_state slots, the GC knob and their comments), so this PR leaves both 1.7B configs as its base has them. This reverts the config hunks of fce78c0, f63eb79, aba669b, 9ef1d1e, 08c8b23, e4450ea and 5428cca, including the [2, 4, 8, 16] first-chunk schedule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the incremental codec, mixed walk and fused depth loop (owned by #978) Kirill's #978 makes the same Qwen3-TTS changes (from the same stacked-head work): the incremental codec with its codec_state pool and chunk knobs, the combined prefill+decode walk and its captures, the fixed prefill embeddings, and a fused depth loop (cp_linear with folded norms, cp_attend, fold_depth_weights) beside the codec's own compile. This PR leaves all of it to him: Qwen3-TTS, the shared code predictor kernels and Qwen3-Omni's two call sites go back to the base's code, and so do their tests, the real-weights codec test, the Qwen3-TTS and Qwen3-Omni code-predictor paragraphs in docs/models.rst, and the codec bench, which timed the removed decoder. That reverts fce78c0 and the fixes made to its code since: 59d1b38, f5b0422, 2f6333a, e59639f, 251d13e, add26fb, f7887b7, 4d9f545, e6b2a2f, a2331d0, ce5693c, 1158293, 8ebf639, 6131600, b13ab78, 9a588f9, 26ef71b, 15413c4, 5fea402, ab3350e, 4f935c1, a8c537e, c4ed93f, 88b5b68, 5bfeeac, a5687bd, the code-predictor half of 4791cb3 and the Qwen3-TTS halves of f63eb79, 7906d09 and 13b35b1. Kept: db8b241, a decode row reads its text token by its loop iteration rather than a counter a re-prepared step advances twice. The counter is the base's (and main's) code, which #978 leaves as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: drop compile_node_regions, which only the removed codec test and bench called 13b35b1 and 7a8be73 lifted Engine._compile_regions' body into a module-level compile_node_regions so the Qwen3-TTS codec test and bench could compile the decode as the engine does. Both are gone with the incremental codec (owned by #978), so _compile_regions takes its body back, as on the base. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: don't claim the remaining captures fit 80 GB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin video decode threads to the CPUs the plan leaves under pinning A case with a pinned CPU fails if video_decode_threads goes back to usable_cpus(); the MSTAR_STEP_THREAD_CORE row names FFmpeg's per-decode threads among what the pins shrink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ragged attention: a wrapper skips replanning the layout it last planned The dense manager kept the (cu_q, cu_k, causal) each of its wrappers was last planned with and skipped an unchanged plan; the ragged self- and cross-attention managers replanned every step, so a denoise loop paid FlashInfer's plan (a kernel and a stream sync) once per step for the same segments. The skip now lives in the wrapper both use (_RaggedPrefillBase._plan): each wrapper is one (label, slot) or one (bucket, slot, label), so the schedule it keeps is the one the step reading it planned. The dense manager's own table goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the attention and CFG-parallel timings as before #963 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: leave the idle wait to its own change The conductor's 1 ms nap becomes a wait on the poll in a change of its own, which waits until the next timed check instead of a fixed 1 ms. This branch's capped poll, its failed-pass back-off, the conductor stand-ins' poll_for_messages and their test go, so those files are main's again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server tests: the failed-wait test pins its back-off; stand-in waits as main has them The message loop always passes a computed timeout, so the 1 ms defaults left by the fixed poll go: the chunk-sink stand-in takes main's again and the failed-wait stand-in takes it as required. That test now asserts the 10 ms back-off, without which a persistent wait error would spin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Preprocess pool: at most the CPUs the plan leaves unpinned Every other API-server pool is sized from the CPU plan; the preprocess pool took its declared or configured size uncapped, so under step_thread_core a model declaring 8 workers could run them, each with its own FFmpeg threads, on fewer free CPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: drain routes before an input only when it prepares here The drain keeps a call's held items from waiting behind a preprocess that may take seconds. An input prepared on the pool only stores and sends when it is handed on, and each pass already drains routes at its top, so it no longer takes a drain of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: say what a RecurrentStateSpec max_slots override must leave Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: compile the block loop, so a step past the captured buckets runs compiled disable_torch_compile left the block loop uncompiled whenever it did not replay: a long video (over 16384 patches) ran the blocks' fp32 rope, norms and activations as eager kernels, slower than the whole-forward compile it replaced. The block loop is now a compile region like the embed and the merger, dynamic in the patch count; it compiles in its capture's warm-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop the api_switch_interval_s knob Its gain was measured before #849 and #939 changed the message-thread and data-worker path, and the re-measure it was waiting on never ran. The serving-config key, its validation, docs rows and test go; workers keep main's MSTAR_PY_SWITCH_INTERVAL_SEC unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the GIL switch-interval knob's model defaults (the knob is removed in #858) Reverts #859's additions on top of #858's api_switch_interval_s: the Model.API_SWITCH_INTERVAL_S class default, Qwen3.5's 0.001 override, the model argument to entrypoint._set_switch_interval, the doc wording and the test fakes that carried the attribute. Every file #858 owns is back to #858's text, so #858's removal of the knob merges in cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: the TP2 publish test's KV managers hold one rank's head KVManager shards its config in build, so the test's direct constructor shards it the same way and each rank moves its own shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop warmup_requests' concurrency option No config sets it, so its parsing, docs and tests covered a dead path. Each entry still goes out once per data-parallel replica, now as warmup-<spec>.<copy>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: a request too large for a replay gets an encoder step of its own Up to 8 requests share an encoder step, and a step that cannot replay as one runs them back to back (can_batch). For requests that are each past the captured buckets (long videos) that only held every request's embeddings, and so their prefills, until the last one was encoded: under load the prefills bunched into waves (TTFT +30-50% on 60 s video at c=16/32, decode almost never interleaved with a prefill). The vision node now splits its steps by capture key, a request too large for a replay being a group of its own, and the engine lets a node whose only capture is a piecewise region split that way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: Canary's decode cap leaves the prefill's token in the budget #794's cap-versus-stop-check agreement tests for Canary and Canary-Qwen assumed a stop check that counts max_tokens; TokenStop leaves the budget to the loop's cap, so they pin the cap instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: last_prefill_flag for a prompt prefilled over several passes; GLM-5.2 test comment names the loop cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision split: review cleanups - Engine.capture_group asks Engine.splits_by_capture_key; the module helper existed only for a test fake, which now builds a real Engine. - The split docstrings say a piecewise region is not tied to a walk, and that on a piecewise-only node the key only groups the scheduler's rows. - Stale 'eager' comments on the vision tower: past the top bucket the block loop runs compiled, uncaptured. - docs/models.rst: drop the one-boot measurement of a different overlay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chatterbox, Higgs Audio tests: a stop id at prefill ends the partition #858 moved TokenStop's prefill_walks onto END_PARTITION; the step closing a reply that ended at prefill no longer exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Orpheus test: the stop check's fake info names its walk TokenStop now reads each stopping row's walk to choose between its loop and END_PARTITION. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 9B DP2 config: the replica pick follows #920's router The conductor now weighs a predicted prefix hit beside the live requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: the vision split's capture-only scope, Qwen3.5 configs' start-up compile, Qwen3-Omni's encoder header - The Qwen3.5 vision split applies while the block loop is captured, and requests that fit alone but not together still run one after another in one step; cg_key_info's and capture_group's docstrings say a piecewise-only node's key is a scheduler group, not a bucket. - The Qwen3.5 configs' warmup comments: the vision regions compile at start-up. - Qwen3-Omni's submodule header takes main's line for its audio encoder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the timings by the DiT step they measured, not a PR number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: an earlier version of the DiT block, not an earlier DiT step "DiT step" reads as a denoising step in this section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: a capped deployment holds cuDNN's capture search to its share capture_cudnn_benchmark's search runs in capture sizing since the sized warm-ups took the node's scope, and it tries workspaces as large as the GPU has free: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized, and an over-cap try is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the preprocess pool's CPU cap does not count a prepare's FFmpeg threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chunk policy: leave the longer primed first chunk to #977 #977 makes the same change (ScheduledLeftContextChunkPolicy.prime past left_context, and its test), and its consumer, the incremental Qwen3-TTS codec, is no longer in this stack: every primer here passes at most left_context (Qwen3-TTS's lead is min(ref_frames, left_context_frames)). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ViT patch embed: the conv timing in its comment is Qwen3.5's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: _search_cap's docstring separates a skipped search try from a failed sizing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B DP2 config: say the vision regions compile at start-up Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: capture_cudnn_benchmark's search runs in capture sizing, held to a capped deployment's share Sizing runs a node's warm-up forwards; the capture after it warms up with no forward, so the scope around warmup_and_capture covered no conv on CUDA and Kokoro's graphs kept cuDNN's heuristic picks. The scope now wraps sizing. The search tries workspaces as large as the GPU has free, and the allocator keeps them until sizing's empty_cache: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized and restored to its previous value after; an over-cap search try is skipped, and the node's own sizing fails the boot when its eager step does not fit in the share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: capture the decode buckets uncompiled, as on main Compiling the seven decode buckets took a cold start from 180 s to 1372 s on H100, with no measured throughput to show for it. The 64-row bucket and the packed prefill captures stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: TF32 codec decode is the model's default The shipped config set codec_exact_fp32: false while the model defaulted to the exact decode, so other Fish configs decoded differently. The default now lives in the model; codec_exact_fp32: true still opts into the exact decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: open the depth loop with position 0 alone Both Talkers opened with one-token passes on main; the two-token opening changed greedy codes for both, and a one-token pass keeps every pass at bs rows. The opening is now main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the 24- and 48-row decode buckets They took the 1.7B's cold start from 204 s to 307 s on H100, and a step at 64 rows costs about what it does at 32, so padding into 32 and 64 is nearly free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * run_layers: the residual stream starts from its own copy The stream was the caller's hidden_states, overwritten in place from layer 0, so a caller had to know to pass a copy (the Qwen3-TTS Talker did for its packed replay's static input). One [tokens, hidden] copy per forward, made inside the forward so a capture records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice silence test: pydub is the oracle, installed with dev The comparison imported the reference package, which CI does not install, so it always skipped there. It now restates the reference's trim over pydub.silence (raw-sample segments, no ffmpeg), with pydub in the dev extras, and covers the rates and edge cases the self-referential trailing trim test stood in for; that test is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: each Talker keeps its own embedding-sum dtype The shared depth loop summed a frame's codec embeddings in fp32 for both Talkers. Qwen3-TTS accumulated in the model dtype, rounding after every add, so its greedy codes could differ at near-ties. ResidualCodePredictor takes embed_sum_dtype from its Talker: None (the embeddings' dtype) for Qwen3-TTS, fp32 for Qwen3-Omni, whose fp32 sum is rounded once to the model dtype as before. The accumulate kernel adds in fp32 and rounds to the sum's dtype, as torch's add_ does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: Qwen3-TTS's embedding sum, the cuDNN search in capture sizing; audioop-lts for pydub on 3.13 - Qwen3-TTS's per-add rounding is the sum this Talker had in M*; the reference sums in fp32 and rounds once. - adding_models.rst and Kokoro say the cuDNN search runs in capture sizing. - pydub 0.25.1 imports audioop, which Python 3.13 removed: the dev extras add audioop-lts there, so the silence oracle does not skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStop: withholding outputs and the runtime check refuse each other The graph runtime's stop check never calls check_stop, which is what withholds a stop step's outputs, so a TokenStop with both would drop the withhold silently. It is a frozen dataclass now, and that combination raises at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_knobs: take main's version back now that ingraph_decode_rope stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 CFG-parallel: the scaffold keeps ragged_for, docs say a rank runs its branch over the batch Since the dit moved onto the scaffold's denoise loop, DenoiseLoopSubmodule is RaggedAttentionMixin's only user, so the mixin folds back into it as on main. The declare_step test keeps only what the per-rank segment test does not cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: quote the guided text-to-image PSNR and the configuration of each number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan22 extra: install flashinfer-python for the default ragged attention Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2: leave CFG-parallel to the cfg_size axis Drop wan22_cfg_parallel_2gpu.yaml, my_rows and the dit's sp-group plumbing, which used the SP slot to mean a CFG split. The cfg_size axis carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Z-Image-Turbo: capture batch sizes 1, 2, 4, 8; every size up to 8 is a commented opt-in Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2-AC test: a rollout request's stamped clip shape keys the encoder capture Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B configs: keep main's 2176 bucket for a mixed step The 9B ladders replace the default, so they lost the slack bucket that lets a mixed step of four 512-token prompts and its decode rows replay at 2176 rather than pad to 4096. Adding it keeps the compile count at 82, strictly under Dynamo's limit of 84; the 9B ladders keep 3072-7168 in place of main's 4352 and 8704. The mixed-step bucket and the recompile limit are now checked on every ladder a Qwen3.5 config ships, not only the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2-AC capture-key test: decode the sampled frames as the request does process_prompt now decodes the frames load_video sampled, by index, so the test hands it sampled indices and stands in for the decoder rather than a decoded clip, which a request no longer carries. The Z-Image capture-size figures say they predate the fused q/k norm + rotary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sampler RNG offset test: both captured kernels, each against a draw it can match At V=64 the captured step takes fused_sample, whose Triton RNG never agrees with the eager step's FlashInfer draw, so the test passed whether or not the two steps shared an offset. The eager case now draws its captured step through FlashInfer too, and a second case runs two captured steps on fused_sample, which reads the same offset buffer the replay advances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: refuse a bad preprocess_workers before the thread starts The thread reads preprocess_workers as it starts, so a value it cannot take ("auto") ended it while the server came up, and every request then waited for its timeout; 0 or a negative quietly meant inline, and true meant 1. PreprocessWorker now refuses anything but a positive integer, on the caller's thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2 pool sizes: say what they cost and when to change them The configs' preprocess_workers is taken as given, so each says it was measured on an H200 host, how many intra-op threads the pool then runs, and to lower it (or drop the key for the data worker's CPU-aware default) on a smaller host. The model comments name CpuPlan.preprocess_threads, and the pool docs name V-JEPA2 beside Qwen3-Omni. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStepNode: copy only the outputs the per-request stop check reads The per-request fallback aliased text_inputs as new_token with a dict splat, which reads every output name, so the lazy host outputs copied all of them to the host. A ChainMap over the outputs reads only what the check reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check every shipped prefill ladder, not only Qwen3.5's The ladder tests read only configs/qwen3_5*.yaml, so the Qwen3-Omni large_kv config's ladder was never checked. Every config that sets model_kwargs.prefill_token_buckets is now built into its own model's prefill capture; the Qwen3.5-specific checks keep to Qwen3.5 configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: pin the config-to-model plumbing, drop the Qwen3.5 repeat Every shipped ladder's model must take prefill_token_buckets as its own argument (it would otherwise vanish into **kwargs); the Thinker's capture check stays for Qwen3-Omni, whose ladder no other test builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: say what the plumbing check covers, and why it fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: read the Thinker's prefill capture by walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: dispatch on the model class, and name what it checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address #971 review: num_step on the data worker, slot per label, postprocess on CPU CI - OmniVoice: process_prompt parses num_step too, so a non-integer is a 400 rather than a raise at the conductor that hangs the client. - RecurrentStateConfig: refuse a pool whose usable slots cannot hold one request's labels (two labels, one usable slot never admits). - test_omnivoice_silence: stand in for omnivoice.utils.audio's fade and pad where the package is missing, so the postprocess test (NaN refusal, clone reference level, PCM16) runs in CPU CI. - code2wav: masked_fill instead of a second waveform-sized buffer. - Qwen3-TTS: reword the loop-counter comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: stage the trim indices with to_device_nonblocking The shared helper pins and copies without a host stall, in place of a hand-rolled pin_memory + non_blocking copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Configs: drop prefill rungs no step reaches, and the restated speculation default Qwen3.5-9B's 2048-token step budget caps prefill and mixed steps, so its ladder stops at 2176; the Thinker's is 1024, so qwen3omni_2gpu_large_kv.yaml goes back to Omni's default ladder. speculation_mixes_walks {Thinker: true} restated the default (the Thinker's mixed walk has a token budget). The docs drop the claims measured on those rungs, and the DP2 note keeps to video, with the TP-async-on numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Thinker: start the residual stream from a copy of the input embeddings The fused add + norms update the stream in place, so started from input_embeds they wrote into a captured graph's static input. The copy moves from layer_0_embed to the stream (same count), and _stage's docstring goes back to main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: warn when MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS is still set It is no longer read; the warning names model_kwargs.prefill_token_buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address #972 review: say why V-JEPA2 opts out of batching, pin torchcodec's keyframe index - vjepa2 can_batch: lead with the throughput reason, and leave a TODO for warming the encoder's batch sizes if batching is measured to pay. - test_media_decode_limits: the installed torchcodec still has the private _get_key_frame_indices the GOP split reads (skips without torchcodec). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * exact_rms_norm: a first call under capture takes torch's kernel Review: the replica's one-time check does a torch.equal, a host read. Warm-up runs it before any capture today, but a first call that lands in a capture now records torch's kernel (the same result) and leaves the check for the next eager call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2: drop the 16-thread preprocess override Review asked whether 16 intra-op threads beat the CpuPlan default. On an H100 host, 64-frame 720p uploads: 3.34 vs 3.30 req/s at c=1 and 14.19 vs 14.34 at c=16 (16 vs default), within noise, so the override goes and both V-JEPA2 models take the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vjepa2_ac.yaml: re-wrap the pool comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…oice and others (#1140) * BAGEL: comments state the contract instead of the code they replaced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: give the measured output shift of guided text-to-image Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_kv_admission: name the CFG walk BAGEL defines Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3 and LingBot decode through wan_vae.FastDecoder and decode_dtype wan_vae.FastDecoder and decode_dtype had one user, Wan2.2, while Cosmos3's two decoder nodes and LingBot-World kept their own copies of the cuDNN >= 9.16 dtype gate, of the channels-last + DecoderStep setup and of the autocast-off decode_causal call. All of them now use the shared helpers, and the copies (_vae_decode_dtype, _fast_step, the streaming node's _step, LingBot's inline gate and _decode_step) are deleted. Numerics and switches are unchanged: Cosmos3 keeps COSMOS3_VAE_DECODE_FP32, its compile switch, its cast before the layout switch, the streaming step's non-deterministic compile (built only with the warm cache), its latents per call and cudnn_benchmark scope; LingBot keeps no override (decode_dtype's variable is now optional), and bf16 and channels-last only off reference numerics. Every node keeps the layout it set up, so FastDecoder's per-call layout check is a no-op for them. FastDecoder's autocast-off scope now follows the latents' device instead of naming cuda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan 2.2: drop the FFN GELU epilogue, which has no measurement of its own MLP(gelu_epilogue=) and Wan22DiT's fuse_ffn_gelu ran the FFN GELU in the up-projection's cuBLASLt epilogue through the private torch._addmm_activation. Nothing measured it apart from the ragged attention change it shipped with, so it goes, as unmeasured performance paths do: MLP is back to its plain form and the DiT builds without the flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3 text encoder: refuse a row without text_len instead of never replaying The shared encoder node picks its captured (masked) graph from each row's text_len step metadata. A model that reused the node without setting it got every batch run eager, with no error. cg_key_info now raises ValueError for such a row; FLUX.2-klein and Z-Image set it at intake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: an explicit null is the default, and only integers pass intake image_output_format let png_compress_level: null through, and encode_image then failed on int(None) after the denoise and decode. encode_image now reads null as the default level, as it already did for output_compression. Intake also refused nothing that int() accepted, so True passed as 1 and 50.9 was truncated to 50; it now takes only integers (not bools) in range, as BAGEL's width and height check does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2: say the B=1 guard routes eager steps only, and drop "as before" comments A step that leases the AC rollout capture replays forward_batched at any batch size, B=1 included, whatever can_batch returns, so can_batch's B=1 guard covers eager steps only; its comment now says so. Two comments described the code by comparison with an earlier version ("as before the capture", "same as before") and now state what the code does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan_vae.decode_dtype: say that an override of 0 forces bf16 The override variable is two-way, as Wan2.2's and Cosmos3's were before it was shared (and as the Wan fast-decode test uses it): set, it forces fp32, or bf16 when it is 0/false/no/off. The docstring read as if 0 meant "not forced". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Higgs Audio: declare text_output so inline decode tokens skip the per-row postprocess Higgs's decode token rides inline, but without a declared text_output Model.postprocess_inline built a tensor and called postprocess for every row on every step. Declaring TextOutput(self._detokenizer), with no stop ids as before, gives it the memoized id-to-bytes path Qwen3-ASR and Whisper use; postprocess hands text to the base class. Bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: say the API_SWITCH_INTERVAL_S gain predates #849 docs/serving.rst says the number was measured before #849 and awaits a re-measure; the comment pointing at it now says the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec bench: unambiguous float32 reference wording (tail R2 P3) The parenthetical read as if the server runs float32; the shipped config runs a bfloat16 codec. Say the reference matches a float32-codec server. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: build its instance table from WorkerGraph.replica_ranks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output knobs: serve the whole numbers main served Intake took only Python ints, so a whole-number float (90.0) or a number in a JSON string ("90"), which main read through int() and served, became a 400. It now accepts any whole number int() reads in range, and still refuses bools, fractions such as 50.9, infinities and out-of-range values. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: build only the window walks a deployment can run A guided window runs on the branch nodes on CFG-parallel deployments and on LLM alone elsewhere, never both, yet CFG-parallel builds still placed the unsplit image_gen_guided2/3 walks and listed them in every request's needed-label map. LLM_GRAPH_WALKS is now keyed by the layout, and the walk graphs and the label map both read it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say why a split LLM still publishes both branch labels The comment dated from when a remote image_gen walk's branch count varied per step. Each walk now fixes its count; what keeps cfg_img published on a split deployment is a remote decode reading it and the single-node denoise walks naming their labels per request rather than per walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL and Cosmos3: drop loop-counter leftovers from comments and names denoise_loop's docstring still named the step's loop counter, which BAGEL no longer reads; declare_step's walk-branch narrowing kept a requires_cfg guard that active_labels already implies, and its guided-row set shadowed the module's guided(). Cosmos3's decode-dtype comment now gives both directions of its override. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Image output: check only the chosen format's compression knob Intake refused a request whose other-format knob was out of range (a JPEG request with png_compress_level 10, a PNG request with output_compression 150), which main served because encode_image never reads it. Check only the knob the format applies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL: say LLM-disaggregated, not split, in the branch-label comment "Split" means CFG-parallel in the neighbouring code (DenoiseWalk.split, LLM_GRAPH_WALKS). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: derive the loop-back hooks from device_loopback = None The stand-in stubbed the deleted _device_loopback and restated the three hooks DeviceLoopback now derives; clearing the declaration gives the same answers through the base class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TP read-failure test: drop the stale "emit" from the stand-in's comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Merge-up fixes: DeviceLoopback in the withhold check, chunk-sink stand-in poll, postprocess_s on inline failure - _refuse_unroutable_withholds read TokenStop.loopback_signal, which main moved to ARNodeSubmodule.device_loopback; it now reads the signal from the submodule's DeviceLoopback, and its test declares one. - test_chunk_sink: its request profile stub lacked output_postprocess_s, so every pass raised before relaying a chunk; it uses a real RequestProfile, its transport's poll ends the pass (the loop no longer sleeps), and an error in the pass is raised instead of retried. - An inline value whose batched postprocess_inline call failed is charged its share of that call as well as its own retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check: the loop-back signal is exempt only on its own walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Withhold check test: the loop-back exemption is scoped to its walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-ASR/Omni AuT node: pin the replay's windows and packed-clip parity on the shared node Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: pin the unprojected depth path, drop the unused DepthAttention.forward The depth-loop parity test now runs at both Talker widths: 2048 through the input projection (Qwen3-TTS 1.7B) and 1024 without one (Qwen3-Omni, Qwen3-TTS 0.6B), where each step's input is the raw embedding and the first norm reads cp_embed_accumulate's per-chunk partials. The cp_embed_accumulate test checks those partials for both the raw and the projected input. DepthAttention.forward and its attend reference were the unfused layer, and nothing runs them: run_depth_layers reads the weights and attends through the predictor's DepthKV. The class stays as the layers' parameter container that takes none of the node's resources, which the binding test now checks directly. run_depth_layers' eager note keeps only the verified reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * FLUX.2 klein: one shared kept-module forward, so blocks don't recompile per instance exclude_from_compile set a functools.partial forward on each kept 1-D RMSNorm. Dynamo ID-guards a partial set on an instance, so the single-stream block frame klein's blocks share failed its guards once per block until the recompile limit, and the remaining blocks ran eagerly: -6 to -10% img/s at c=1-4 on klein 4B and 9B against main. A kept module now changes class instead, to a subclass made once per (class, path) whose forward is the class's: the exact RMSNorm op for a plain 1-D nn.RMSNorm, torch.compiler.disable of the base forward for every other kept module (the same graph break as before). compile_kept_exact is a class attribute of that subclass, so kept_exact (Z-Image's fused q/k norm + rotary, the VAE's norm_act) still answers. Outputs are unchanged: the same exact_rms_norm call, the same disabled base forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compile_utils: state the guard that recompiled per block Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: say why the multi-token path stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: keep the compiled depth step free of per-position guards depth_step took the LM head index as an int and passed it to the disabled run_depth_layers, so under a static-int compile Dynamo compiled depth_step once per head (guards "head is None", "head == 0", ...): one frame per depth position, up to the recompile limit. depth_step itself is now the eager boundary, with run_depth_layers folded into it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * compile_utils: every block passes the same guards, not one check Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * cp_attend: name the row cap a multi-token opening would need raised Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: name every eager reader of the depth ints; count the test's groups Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: compile the incremental decode through compile_regions The codec compiled its own decode with torch.compile(dynamic=False), which recompiled once per captured shape (4 chunk widths x 14 batch buckets = 55 compiles) and skipped the engine's inductor overrides and recompile accounting. On an empty cache that was ~16 of the ~19 minutes of a 1.7B cold start. The decode is now a compile region: the engine compiles it once, with dynamic shapes, before capture. H100, slice-6 head (empty Inductor/Triton cache, 26 CPUs): cold start 1119 -> 444 s, warm 540 -> 333 s. Closed loop audio-s/s c=1/16/32/64, 2 rounds each: push4a 18.4-18.5 / 137-145 / 209-212 / 281-284 -> 18.4 / 139-146 / 208-212 / 289-301; push4c (server on 2 CPUs) 18.6/159/237/370 -> 18.4-18.5 / 160 / 237-251 / 371-379. Pinning the chunk width static (one compile per width, batch dynamic) measured no faster and started in 765 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS codec: test and bench compile the decode as the engine does The codec's decode is compiled by the engine's compile_regions pass, so a CodecSubmodule built directly ran it eagerly. The per-node body of Engine._compile_regions becomes compile_node_regions, which the engine loop, the streamed-codec test's CUDA case and codec_bench's load_codec all call. The compile_regions docstring and models.rst now say what dynamic=True gives: a few graphs (batch 1 and width 1 specialise) for every chunk width and batch bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Codec and compile_node_regions docstrings: the measured cold start, no dynamic=None aside Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: start the codec stream with a 2-frame chunk The deployment config's ramp began with a 4-frame chunk, so a stream's first audio waited for four Talker steps. [2, 4, 8, 16] sends it after two, and the 160 ms first chunk still covers the time until the next one arrives under load. [1, 2, 5, 16] is documented as the lower-latency opt-in. H100 (push4a and push4c), closed loop, c=64: TTFA p50 84-95 -> 63-78 ms (c=1 33 -> 25 ms), no playback stalls either way, audio-s/s 379-424 vs 363-416 over 4-6 rounds each (same within noise). Open-loop SLO capacity (TTFA <= 200 ms, stall < 10 ms): Seed-TTS 80 @90/99 and passages 28 @90 either way. WER (Whisper large-v3, 128 sentences): 2.09% -> 1.73%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * qwen3tts_1p7b config: the header names the 2-frame start; date the TTFA numbers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B: the bf16 codec is measured on H100 too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: klein's exact q/k norm, measured with the shared-forward fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B bf16 codec: state what the H100 runs measured Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: klein's shared kept-norm forward, measured against a forward per block Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS bf16 codec: tie each H100 WER to its chunk schedule Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: kept norms share a forward per class Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attention: select_last_hidden is one eager boundary, so prefill buckets don't recompile its frame The disabled `qo_indptr_buf` call broke the graph inside `select_last_hidden`, leaving a resume frame that holds both the bucket's static token count (`hidden`) and its row count (`qo_indptr`). Captured forwards compile with dynamic=False, so that frame took one cache entry per (tokens, rows) bucket: Qwen3.5's 9 token buckets by 7 prefill/mixed row counts made 63, and the 13-step prefill ladder made 91, past the recompile limit of 84. Disabling the whole method moves the break to its call site, where the caller already breaks; no compiled frame holds the pair. Same for the MLA and XPU managers, which share the shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL takes its prefill's last rows through select_last_hidden; test docstrings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * GDN prefill run: state the zeroed-tail precondition the mixed wrapper also meets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: key the backbone's step outputs by the engine's row ids The packed backbone keyed its outputs by CanvasItem.request_id, which prepare_inputs now fills from fwd_info.request_id, the wire request id. The engine looks each row's outputs up by its worker-local handle, found none, and every OmniVoice request failed in postprocess with KeyError 'c_logits'. forward_batched now keys rows by engine_inputs.request_ids, as the other batched submodules do, and CanvasItem carries no request id at all, so the canvas can't be keyed by the wrong one again. The unread request_id kwarg goes with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: keep the 8192 Thinker ladder off the 80 GB default qwen3omni_2gpu.yaml, the CLI default and the config sized for 80 GB GPUs, ran the Thinker's prefill ladder to 8192. On an H100 its 4096-8192 captures don't fit beside the 1,024-page KV pool (59 of 86 Thinker buckets captured), so those steps run eagerly and a deployment that requires accelerator graphs refuses to start. The default goes back to the model's 128-2048 ladder, as on the base. The 8192 ladder, measured on 2xH200, stays in qwen3omni_2gpu_large_kv.yaml, which is already the H200 variant; its header, the default's, the submodule comment and the Qwen3-Omni notes say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS 1.7B configs: back to the base's codec, chunk ramp and pools (owned by #978) Kirill's #978 sets these himself (the [4, 8, 16] ramp, the bf16 codec, the codec_state slots, the GC knob and their comments), so this PR leaves both 1.7B configs as its base has them. This reverts the config hunks of fce78c050, f63eb7994, aba669bac, 9ef1d1e66, 08c8b2350, e4450ea53 and 5428cca16, including the [2, 4, 8, 16] first-chunk schedule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the incremental codec, mixed walk and fused depth loop (owned by #978) Kirill's #978 makes the same Qwen3-TTS changes (from the same stacked-head work): the incremental codec with its codec_state pool and chunk knobs, the combined prefill+decode walk and its captures, the fixed prefill embeddings, and a fused depth loop (cp_linear with folded norms, cp_attend, fold_depth_weights) beside the codec's own compile. This PR leaves all of it to him: Qwen3-TTS, the shared code predictor kernels and Qwen3-Omni's two call sites go back to the base's code, and so do their tests, the real-weights codec test, the Qwen3-TTS and Qwen3-Omni code-predictor paragraphs in docs/models.rst, and the codec bench, which timed the removed decoder. That reverts fce78c050 and the fixes made to its code since: 59d1b3854, f5b0422ff, 2f6333a90, e59639f21, 251d13e34, add26fbc7, f7887b71a, 4d9f54584, e6b2a2fef, a2331d014, ce5693c3d, 1158293d0, 8ebf63986, 613160070, b13ab78e4, 9a588f978, 26ef71b6e, 15413c499, 5fea4023e, ab3350e18, 4f935c1f6, a8c537e7d, c4ed93f0c, 88b5b68de, 5bfeeac60, a5687bdd7, the code-predictor half of 4791cb3ca and the Qwen3-TTS halves of f63eb7994, 7906d0902 and 13b35b17a. Kept: db8b241c7, a decode row reads its text token by its loop iteration rather than a counter a re-prepared step advances twice. The counter is the base's (and main's) code, which #978 leaves as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: drop compile_node_regions, which only the removed codec test and bench called 13b35b17a and 7a8be7397 lifted Engine._compile_regions' body into a module-level compile_node_regions so the Qwen3-TTS codec test and bench could compile the decode as the engine does. Both are gone with the incremental codec (owned by #978), so _compile_regions takes its body back, as on the base. No behaviour change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni 2-GPU config: don't claim the remaining captures fit 80 GB Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin video decode threads to the CPUs the plan leaves under pinning A case with a pinned CPU fails if video_decode_threads goes back to usable_cpus(); the MSTAR_STEP_THREAD_CORE row names FFmpeg's per-decode threads among what the pins shrink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ragged attention: a wrapper skips replanning the layout it last planned The dense manager kept the (cu_q, cu_k, causal) each of its wrappers was last planned with and skipped an unchanged plan; the ragged self- and cross-attention managers replanned every step, so a denoise loop paid FlashInfer's plan (a kernel and a stream sync) once per step for the same segments. The skip now lives in the wrapper both use (_RaggedPrefillBase._plan): each wrapper is one (label, slot) or one (bucket, slot, label), so the schedule it keeps is the one the step reading it planned. The dense manager's own table goes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the attention and CFG-parallel timings as before #963 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Conductor: leave the idle wait to its own change The conductor's 1 ms nap becomes a wait on the poll in a change of its own, which waits until the next timed check instead of a fixed 1 ms. This branch's capped poll, its failed-pass back-off, the conductor stand-ins' poll_for_messages and their test go, so those files are main's again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server tests: the failed-wait test pins its back-off; stand-in waits as main has them The message loop always passes a computed timeout, so the 1 ms defaults left by the fixed poll go: the chunk-sink stand-in takes main's again and the failed-wait stand-in takes it as required. That test now asserts the 10 ms back-off, without which a persistent wait error would spin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Preprocess pool: at most the CPUs the plan leaves unpinned Every other API-server pool is sized from the CPU plan; the preprocess pool took its declared or configured size uncapped, so under step_thread_core a model declaring 8 workers could run them, each with its own FFmpeg threads, on fewer free CPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: drain routes before an input only when it prepares here The drain keeps a call's held items from waiting behind a preprocess that may take seconds. An input prepared on the pool only stores and sends when it is handed on, and each pass already drains routes at its top, so it no longer takes a drain of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: say what a RecurrentStateSpec max_slots override must leave Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: compile the block loop, so a step past the captured buckets runs compiled disable_torch_compile left the block loop uncompiled whenever it did not replay: a long video (over 16384 patches) ran the blocks' fp32 rope, norms and activations as eager kernels, slower than the whole-forward compile it replaced. The block loop is now a compile region like the embed and the merger, dynamic in the patch count; it compiles in its capture's warm-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop the api_switch_interval_s knob Its gain was measured before #849 and #939 changed the message-thread and data-worker path, and the re-measure it was waiting on never ran. The serving-config key, its validation, docs rows and test go; workers keep main's MSTAR_PY_SWITCH_INTERVAL_SEC unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the GIL switch-interval knob's model defaults (the knob is removed in #858) Reverts #859's additions on top of #858's api_switch_interval_s: the Model.API_SWITCH_INTERVAL_S class default, Qwen3.5's 0.001 override, the model argument to entrypoint._set_switch_interval, the doc wording and the test fakes that carried the attribute. Every file #858 owns is back to #858's text, so #858's removal of the knob merges in cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: the TP2 publish test's KV managers hold one rank's head KVManager shards its config in build, so the test's direct constructor shards it the same way and each rank moves its own shard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * API server: drop warmup_requests' concurrency option No config sets it, so its parsing, docs and tests covered a dead path. Each entry still goes out once per data-parallel replica, now as warmup-<spec>.<copy>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision: a request too large for a replay gets an encoder step of its own Up to 8 requests share an encoder step, and a step that cannot replay as one runs them back to back (can_batch). For requests that are each past the captured buckets (long videos) that only held every request's embeddings, and so their prefills, until the last one was encoded: under load the prefills bunched into waves (TTFT +30-50% on 60 s video at c=16/32, decode almost never interleaved with a prefill). The vision node now splits its steps by capture key, a request too large for a replay being a group of its own, and the engine lets a node whose only capture is a piecewise region split that way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tests: Canary's decode cap leaves the prefill's token in the budget #794's cap-versus-stop-check agreement tests for Canary and Canary-Qwen assumed a stop check that counts max_tokens; TokenStop leaves the budget to the loop's cap, so they pin the cap instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: last_prefill_flag for a prompt prefilled over several passes; GLM-5.2 test comment names the loop cap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 vision split: review cleanups - Engine.capture_group asks Engine.splits_by_capture_key; the module helper existed only for a test fake, which now builds a real Engine. - The split docstrings say a piecewise region is not tied to a walk, and that on a piecewise-only node the key only groups the scheduler's rows. - Stale 'eager' comments on the vision tower: past the top bucket the block loop runs compiled, uncaptured. - docs/models.rst: drop the one-boot measurement of a different overlay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chatterbox, Higgs Audio tests: a stop id at prefill ends the partition #858 moved TokenStop's prefill_walks onto END_PARTITION; the step closing a reply that ended at prefill no longer exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Orpheus test: the stop check's fake info names its walk TokenStop now reads each stopping row's walk to choose between its loop and END_PARTITION. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5 9B DP2 config: the replica pick follows #920's router The conductor now weighs a predicted prefix hit beside the live requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: the vision split's capture-only scope, Qwen3.5 configs' start-up compile, Qwen3-Omni's encoder header - The Qwen3.5 vision split applies while the block loop is captured, and requests that fit alone but not together still run one after another in one step; cg_key_info's and capture_group's docstrings say a piecewise-only node's key is a scheduler group, not a bucket. - The Qwen3.5 configs' warmup comments: the vision regions compile at start-up. - Qwen3-Omni's submodule header takes main's line for its audio encoder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: date the timings by the DiT step they measured, not a PR number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 docs: an earlier version of the DiT block, not an earlier DiT step "DiT step" reads as a denoising step in this section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: a capped deployment holds cuDNN's capture search to its share capture_cudnn_benchmark's search runs in capture sizing since the sized warm-ups took the node's scope, and it tries workspaces as large as the GPU has free: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized, and an over-cap try is skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the preprocess pool's CPU cap does not count a prepare's FFmpeg threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Chunk policy: leave the longer primed first chunk to #977 #977 makes the same change (ScheduledLeftContextChunkPolicy.prime past left_context, and its test), and its consumer, the incremental Qwen3-TTS codec, is no longer in this stack: every primer here passes at most left_context (Qwen3-TTS's lead is min(ref_frames, left_context_frames)). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ViT patch embed: the conv timing in its comment is Qwen3.5's Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: _search_cap's docstring separates a skipped search try from a failed sizing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B DP2 config: say the vision regions compile at start-up Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Engine: capture_cudnn_benchmark's search runs in capture sizing, held to a capped deployment's share Sizing runs a node's warm-up forwards; the capture after it warms up with no forward, so the scope around warmup_and_capture covered no conv on CUDA and Kokoro's graphs kept cuDNN's heuristic picks. The scope now wraps sizing. The search tries workspaces as large as the GPU has free, and the allocator keeps them until sizing's empty_cache: Kokoro reserved ~140 GB of an H200 for ~25 s at startup, past the gpu_memory_fraction a co-located worker (or an 80 GB-capped comparison) plans around. With a fraction set, the allocator is held to it while such a node is sized and restored to its previous value after; an over-cap search try is skipped, and the node's own sizing fails the boot when its eager step does not fit in the share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: capture the decode buckets uncompiled, as on main Compiling the seven decode buckets took a cold start from 180 s to 1372 s on H100, with no measured throughput to show for it. The 64-row bucket and the packed prefill captures stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2 Pro: TF32 codec decode is the model's default The shipped config set codec_exact_fp32: false while the model defaulted to the exact decode, so other Fish configs decoded differently. The default now lives in the model; codec_exact_fp32: true still opts into the exact decode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: open the depth loop with position 0 alone Both Talkers opened with one-token passes on main; the two-token opening changed greedy codes for both, and a one-token pass keeps every pass at bs rows. The opening is now main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-TTS: drop the 24- and 48-row decode buckets They took the 1.7B's cold start from 204 s to 307 s on H100, and a step at 64 rows costs about what it does at 32, so padding into 32 and 64 is nearly free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * run_layers: the residual stream starts from its own copy The stream was the caller's hidden_states, overwritten in place from layer 0, so a caller had to know to pass a copy (the Qwen3-TTS Talker did for its packed replay's static input). One [tokens, hidden] copy per forward, made inside the forward so a capture records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice silence test: pydub is the oracle, installed with dev The comparison imported the reference package, which CI does not install, so it always skipped there. It now restates the reference's trim over pydub.silence (raw-sample segments, no ffmpeg), with pydub in the dev extras, and covers the rates and edge cases the self-referential trailing trim test stood in for; that test is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Code predictor: each Talker keeps its own embedding-sum dtype The shared depth loop summed a frame's codec embeddings in fp32 for both Talkers. Qwen3-TTS accumulated in the model dtype, rounding after every add, so its greedy codes could differ at near-ties. ResidualCodePredictor takes embed_sum_dtype from its Talker: None (the embeddings' dtype) for Qwen3-TTS, fp32 for Qwen3-Omni, whose fp32 sum is rounded once to the model dtype as before. The accumulate kernel adds in fp32 and rounds to the sum's dtype, as torch's add_ does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review wording: Qwen3-TTS's embedding sum, the cuDNN search in capture sizing; audioop-lts for pydub on 3.13 - Qwen3-TTS's per-add rounding is the sum this Talker had in M*; the reference sums in fp32 and rounds once. - adding_models.rst and Kokoro say the cuDNN search runs in capture sizing. - pydub 0.25.1 imports audioop, which Python 3.13 removed: the dev extras add audioop-lts there, so the silence oracle does not skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStop: withholding outputs and the runtime check refuse each other The graph runtime's stop check never calls check_stop, which is what withholds a stop step's outputs, so a TokenStop with both would drop the withhold silently. It is a frozen dataclass now, and that combination raises at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_knobs: take main's version back now that ingraph_decode_rope stays Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2 CFG-parallel: the scaffold keeps ragged_for, docs say a rank runs its branch over the batch Since the dit moved onto the scaffold's denoise loop, DenoiseLoopSubmodule is RaggedAttentionMixin's only user, so the mixin folds back into it as on main. The declare_step test keeps only what the per-rank segment test does not cover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * BAGEL docs: quote the guided text-to-image PSNR and the configuration of each number Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * wan22 extra: install flashinfer-python for the default ragged attention Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Wan2.2: leave CFG-parallel to the cfg_size axis Drop wan22_cfg_parallel_2gpu.yaml, my_rows and the dit's sp-group plumbing, which used the SP slot to mean a CFG split. The cfg_size axis carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Z-Image-Turbo: capture batch sizes 1, 2, 4, 8; every size up to 8 is a commented opt-in Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2-AC test: a rollout request's stamped clip shape keys the encoder capture Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5-9B configs: keep main's 2176 bucket for a mixed step The 9B ladders replace the default, so they lost the slack bucket that lets a mixed step of four 512-token prompts and its decode rows replay at 2176 rather than pad to 4096. Adding it keeps the compile count at 82, strictly under Dynamo's limit of 84; the 9B ladders keep 3072-7168 in place of main's 4352 and 8704. The mixed-step bucket and the recompile limit are now checked on every ladder a Qwen3.5 config ships, not only the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2-AC capture-key test: decode the sampled frames as the request does process_prompt now decodes the frames load_video sampled, by index, so the test hands it sampled indices and stands in for the decoder rather than a decoded clip, which a request no longer carries. The Z-Image capture-size figures say they predate the fused q/k norm + rotary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sampler RNG offset test: both captured kernels, each against a draw it can match At V=64 the captured step takes fused_sample, whose Triton RNG never agrees with the eager step's FlashInfer draw, so the test passed whether or not the two steps shared an offset. The eager case now draws its captured step through FlashInfer too, and a second case runs two captured steps on fused_sample, which reads the same offset buffer the replay advances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Data worker: refuse a bad preprocess_workers before the thread starts The thread reads preprocess_workers as it starts, so a value it cannot take ("auto") ended it while the server came up, and every request then waited for its timeout; 0 or a negative quietly meant inline, and true meant 1. PreprocessWorker now refuses anything but a positive integer, on the caller's thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2 pool sizes: say what they cost and when to change them The configs' preprocess_workers is taken as given, so each says it was measured on an H200 host, how many intra-op threads the pool then runs, and to lower it (or drop the key for the data worker's CPU-aware default) on a smaller host. The model comments name CpuPlan.preprocess_threads, and the pool docs name V-JEPA2 beside Qwen3-Omni. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attention: MLA and XPU share one select_last_hidden on the base class The FlashInfer MLA and XPU managers carried the same select_last_hidden body with its own torch.compiler.disable. One method on AttentionManager reads the backend's qo_indptr_buf and is eager as a whole, so a captured prefill forward keeps no compiled frame holding both the bucket's token count and its row count, and a new paged backend gets it without remembering the decorator. The base class declares qo_indptr_buf too, raising an error that names a backend which plans no query offsets. FlashInfer keeps an override of its own: a label planned as a decode has no query offsets there (every token is a row's last), and a label with no plan is an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish S2-Pro: decode each codec frame once from per-stream state; withhold <|im_end|> with TokenStop The codec re-decoded every window's left context. It now keeps each stream's causal conv and transformer state in a codec_state recurrent pool and decodes only a chunk's new frames (IncrementalDACDecoder, with the streaming conv steps and a time-major Snake in components/streaming_codec.py), with the transposed-conv weights laid out channels-last for their GEMM. The AR declares its stop as a TokenStop that withholds the <|im_end|> frame, so the codec stream carries only frames to decode and the codec needs no device read to drop it; the frame budget is the loop's cap, as for every TokenStop adopter, and batched AR outputs share one stop-check copy per step. The TF32 figures in the docs and the model's TF32-default comment are marked as measured on the earlier windowed codec. Tests: incremental decode stitches into the whole-utterance decode (eager and captured, any chunking), the conv weight layout, a CUDA-only compiled-vs-eager check, the TokenStop stop check, and the frameless final chunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice: score and reveal the whole step's canvases in one batched pass The unmask step scored and revealed each request's canvas in postprocess, one request at a time. forward_batched now combines CFG, draws the Gumbel noise and reveals every canvas of the step in one pass (unmask_step), keyed by the engine's row ids, with the CFG combine's dead buffers freed before the sampler. The backbone captures up to 8192 packed tokens, and the codec's acoustic decoder compiles as a region and is warmed at start-up for every canvas-length range. Tests: the parity tests drive the served unmask step, including a mixed-temperature batch; codec warm-up coverage; a CUDA-only compiled-vs-eager check of the codec decoder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: a cheaper ViT step and a GC threshold Vision encoder: the shared ViT's host grid helpers (components/qwen3_vit.py) build position-table taps once per frame shape and tile them over a video's frames through numpy, on the host only; Qwen3-Omni's vision encoder, their other caller, builds them there too and copies them without a sync, as Qwen3.5's does. The block loop gets a 12288-patch capture bucket (a 30 s video at 2 fps, which 16384 padded by 39%), and each ViT block compiles as its own region, so Dynamo traces one block and the rest reuse the graph instead of tracing the unrolled loop at every start; the start-up examples warm the embed and merger regions as before. Configs: 2B, 4B, 9B TP2 and 27B TP4 raise the gen-0 GC threshold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Attention: fused q/k norm, RoPE and KV write helpers on AttentionCallable The Qwen3 LM (Qwen3-ASR, Canary-Qwen, Fish's slow AR) called its fused q/k RMSNorm + rotary + KV-write prologue through its own glue. AttentionCallable now owns it, with one helper per rotation: norm_rope_attend rotates by positions and a theta, and norm_rope_table_attend by a precomputed cos/sin table (the multimodal-RoPE case). The table variant runs on a new Triton kernel, qk_norm_rope_table_kv_write, in fp32 and rounded once. Both refuse a Gemma-style norm, whose (1 + w) scale the kernels don't apply. Tests: the table kernel against an fp64 reference and the unfused norm and rotation, and both helpers' Gemma-norm refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3-Omni: an H100 config, and the Thinker's q/k norm, mRoPE and KV write in one kernel configs/qwen3omni_2gpu_h100.yaml serves 80 GB GPUs: the encoders run on the Talker's GPU so an encoder step doesn't wait behind a Thinker step, and the Thinker KV pool is 512 pages. That size was measured on an H100 while the encoders shared the Thinker's GPU and its text ladder ran to 8192: beside 1,024 pages 27 Thinker captures, all among those of 4096+ tokens (text and vision), ran eagerly, and 640 pages still left two vision buckets out. Chunked prefill now caps a Thinker step at 1,024 tokens, so those large buckets no longer replay, but they are still captured and still take that memory; whether 1,024 pages fit with today's ladders and encoder placement is unmeasured. The other 2-GPU configs and the Qwen3-Omni notes say which GPU each is for, and when a full Thinker pool needs max_concurrent_requests. The Thinker's q/k RMSNorm, interleaved multimodal RoPE and KV write run in one kernel through AttentionCallable.norm_rope_table_attend, in fp32 and rounded once where the unfused ops round to bf16 at each step. docs/models.rst, the fused-kernel section of adding_models.rst and the kernel's test module name the Thinker as its caller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fused_experts: per-pair GEMV experts for small bf16 steps; H100 tile tables A bf16/fp16 step of up to 7 tokens with no shared expert runs one block per (token, expert) pair (routed_experts) instead of the sorted grouped GEMM, on NVIDIA GPUs, for shapes whose tiles are whole and unclamped (routed_fits). H100 tile tables for Qwen3-Omni's Thinker and Talker experts; the per-pair path was swept on H100 only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * FLUX.2-klein, Z-Image: config and compile docs keep the reasons, not the narratives The klein and Z-Image configs and compile_transformer_forward point to compile_exact_ops in docs/models.rst for measured PSNR and cost instead of repeating the numbers; klein ships [norms], Z-Image the plain compile, both on sdpa. The docs give SGLang's PSNR from klein's output as the yardstick. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3, Orpheus, Pi0.5: per-model serving fixes - Cosmos3: a generation step past max_batched_vision_tokens (a model kwarg, 16384 by default) runs its requests one at a time. - Orpheus: SNAC windows past the decoder's left reach pad to the widest window, whose default left context comes from that reach. - Pi0.5: camera images travel to the worker as uint8, and the model refuses to start without a tokenizer (the docs say it needs the gated PaliGemma one). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * TokenStepNode: copy only the outputs the per-request stop check reads The per-request fallback aliased text_inputs as new_token with a dict splat, which reads every output name, so the lazy host outputs copied all of them to the host. A ChainMap over the outputs reads only what the check reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Check every shipped prefill ladder, not only Qwen3.5's The ladder tests read only configs/qwen3_5*.yaml, so the Qwen3-Omni large_kv config's ladder was never checked. Every config that sets model_kwargs.prefill_token_buckets is now built into its own model's prefill capture; the Qwen3.5-specific checks keep to Qwen3.5 configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: pin the config-to-model plumbing, drop the Qwen3.5 repeat Every shipped ladder's model must take prefill_token_buckets as its own argument (it would otherwise vanish into **kwargs); the Thinker's capture check stays for Qwen3-Omni, whose ladder no other test builds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: say what the plumbing check covers, and why it fails Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: read the Thinker's prefill capture by walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Ladder test: dispatch on the model class, and name what it checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address #971 review: num_step on the data worker, slot per label, postprocess on CPU CI - OmniVoice: process_prompt parses num_step too, so a non-integer is a 400 rather than a raise at the conductor that hangs the client. - RecurrentStateConfig: refuse a pool whose usable slots cannot hold one request's labels (two labels, one usable slot never admits). - test_omnivoice_silence: stand in for omnivoice.utils.audio's fade and pad where the package is missing, so the postprocess test (NaN refusal, clone reference level, PCM16) runs in CPU CI. - code2wav: masked_fill instead of a second waveform-sized buffer. - Qwen3-TTS: reword the loop-counter comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * AuT encoder: stage the trim indices with to_device_nonblocking The shared helper pins and copies without a host stall, in place of a hand-rolled pin_memory + non_blocking copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Configs: drop prefill rungs no step reaches, and the restated speculation default Qwen3.5-9B's 2048-token step budget caps prefill and mixed steps, so its ladder stops at 2176; the Thinker's is 1024, so qwen3omni_2gpu_large_kv.yaml goes back to Omni's default ladder. speculation_mixes_walks {Thinker: true} restated the default (the Thinker's mixed walk has a token budget). The docs drop the claims measured on those rungs, and the DP2 note keeps to video, with the TP-async-on numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Thinker: start the residual stream from a copy of the input embeddings The fused add + norms update the stream in place, so started from input_embeds they wrote into a captured graph's static input. The copy moves from layer_0_embed to the stream (same count), and _stage's docstring goes back to main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Qwen3.5: warn when MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS is still set It is no longer read; the warning names model_kwargs.prefill_token_buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address #1140 review: size Fish's codec pool to the AR batch, warm OmniVoice's short canvases - Fish S2-Pro: the codec state pool holds 64 streams (+ sink), as many as the AR decodes a step, not 256: about 0.55 GiB instead of 2.1 GiB. - OmniVoice: warm the 1- and 2-frame codec canvases too, so a very short reply does not compile on the GPU thread. - test_model_kv_layouts: Fish joins the decode-cap case that ties get_loop_caps to where TokenStop's decode ends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish: cap concurrent requests at the codec streams; tighten the cap test Review of the previous commit: a request keeps its codec stream until it ends, so past 64 requests a request waited a whole request for its first audio while its AR rows slowed the streams. The shipped yaml now sets max_concurrent_requests: 64 so later requests queue before the AR, and validate_config_yaml refuses a cap the codec pool cannot hold. The decode-cap case keys on TokenStop's own loop; the OmniVoice warm-up note says why one and two frames are warmed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address #972 review: say why V-JEPA2 opts out of batching, pin torchcodec's keyframe index - vjepa2 can_batch: lead with the throughput reason, and leave a TODO for warming the encoder's batch sizes if batching is measured to pay. - test_media_decode_limits: the installed torchcodec still has the private _get_key_frame_indices the GOP split reads (skips without torchcodec). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cosmos3: leave the batched vision-token cap off by default Review: 16384 was measured on H100 only, and at 832x480 two 121-frame Edge clips (about 12k vision tokens each) never batched under it. It is a per-GPU number, so it is now off unless a deployment sets max_batched_vision_tokens in its model_kwargs (16384 on an H100). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * exact_rms_norm: a first call under capture takes torch's kernel Review: the replica's one-time check does a torch.equal, a host read. Warm-up runs it before any capture today, but a first call that lands in a capture now records torch's kernel (the same result) and leaves the check for the next eager call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish: a null max_slots override means the default; drop a stale large_kv note Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * V-JEPA2: drop the 16-thread preprocess override Review asked whether 16 intra-op threads beat the CpuPlan default. On an H100 host, 64-frame 720p uploads: 3.34 vs 3.30 req/s at c=1 and 14.19 vs 14.34 at c=16 (16 vs default), within noise, so the override goes and both V-JEPA2 models take the default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * vjepa2_ac.yaml: re-wrap the pool comment Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fish: pin that a null max_slots override validates; re-wrap a docs line Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs: re-wrap the Qwen3-Omni H100 line Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OmniVoice graph test: the fake node carries lora_pools --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: merceod <atj10@stanford.edu> Co-authored-by: merceod <atindra@mstarlabs.ai>

…d them Measured on one H200 per pair, the arms alternated on the same GPU, streaming SLO = first audio within 200 ms and under 10 ms of pauses for 90% of Poisson arrivals, 60 s a rate: - codec dtype: Seed-TTS capacity 101 req/s bf16 vs 90 float32. Against the float32 whole decode of 206 utterances' frames, bf16 gives 35 dB SNR median (20 dB worst), 1.0 dB MCD, 0.9999 speaker cosine and the same WER (1.15%). A float32 final conv leaves the SNR at 35 dB, so the waveform rounding is not what limits it. - first chunks: (4, 8, 16) 101 req/s vs (1, 3, 8, 16) 90 on Seed-TTS (passages 29 vs 29); (1, 3, 8, 16) halves idle first audio, 15 vs 31 ms p50 at 1 req/s. So the defaults stay; the config comments and docs/models.rst carry the numbers and the trade, and the docs say the codec compiles twice per chunk width (batch 1 apart).

# Conflicts: # docs/models.rst

merceod · 2026-11-03T10:58:08Z

Taking this one over to get it in: merged main (e7f3c3c) into the branch, one conflict in docs/models.rst (kept main's Thinker q/k norm + RoPE note and this PR's code-predictor note, since cp_attend and cp_linear replace cp_attention). The rework covers my review: one ResidualCodePredictor for Omni and TTS on the new kernels, the stream starting from a copy on main's side, the codec's capture-key split, bf16 and (4, 8, 16) with the SNR and first-audio numbers behind them, 511 streams, compile per chunk width. Running the Omni, TTS, LoRA-target and kernel tests plus Qwen3-TTS and Qwen3-Omni speech cells against main on an H200 box now; CI runs on the merge. I'll merge when both are green.

Brings in #1071 and #1127. The Talker takes #1071's Qwen3Attention layers and declared rope_theta. The depth loop stays on cp_attend and cp_linear, so #1071's two-token first pass and its Blackwell gate are dropped, and the codec stays incremental, so conv_stack.py and its channels-last stack go.

The bare model now needs its codec dtype for the codec_state blocks, and the Talker only gets its own resources, as the engine binds them. The predictor runs at the checkpoint's widths with a Talker as wide as it: cp_linear tiles K by 1024 at one or two rows, so the old 256/512 widths failed the capture. Still fails with run_layers' clone taken out.

Brings in #1073. The codec can't take its side stream here: its windows read and write the codec_state pool, and the side lane refuses a node with resources. So the Qwen3-TTS configs drop side_stream: true (they would fail at startup), the docs say why, and the test expects the refusal.

merceod · 2026-11-03T14:12:34Z

Draft paragraph for the PR text

Merged main twice: #1071 and #1127, then #1073. The Talker now runs on #1071's Qwen3Attention (q/k norm, RoPE and KV write in one kernel)
with the checkpoint's rope_theta declared on its position resource. The depth loop stays on cp_attend and cp_linear. Dropped, not
coming back here: #1071's two-token first pass, its Blackwell gate and their tests. It doesn't fit this loop at 64+ rows, and #1071 put it at 5%
of the loop against 17-22% for the cp kernels at bs 8/32. Also dropped: #1071's ChannelsLastConvStack and conv_stack.py, which decode
whole windows, where the codec here decodes each frame once. #1073's side stream doesn't apply to this codec: it holds the codec_state
pool and the side lane refuses nodes with resources, so the Qwen3-TTS configs don't set side_stream. Follow-ups: try #1071's codec pieces
(the 4D snake, its conv layout) on the incremental decode, which already runs its convs channels-last but hasn't been measured with them; and
let the side lane admit and plan a recurrent-state step so the codec can take it. One test fix rode along: #1071's packed-replay test now
builds at widths cp_linear takes. On an H200 against main b06ae39, Qwen3-TTS 1.7B does 21.5 / 107.6 / 270 audio-s/s at c=1/8/32
against 17.6 / 83.5 / 200 (1.22x / 1.29x / 1.35x), same WER guard, 0 errors. Qwen3-Omni speech in/out, which shares the
predictor, is level to +7% with every WER guard passing.

Logs: /workspace/dcai/runs/pr978_merge/ (scripts/, g7/, gpu_tests_final/, cpu_pr978m/, qwen3tts/, qwen3omni/); WAVs under /root/dcai_pr978m/wav/. Nothing of mine left running; GPUs 0, 1, 7 at 0 MiB (13:41Z).

merceod

Reviewed the rework and the merges over #1071, #1127 and #1073 (resolution rules from #1071's author; details in the comment above). On H200: TTS, kernel, Omni and real-weights GPU tests clean, 0 new CPU Core failures, Qwen3-TTS 1.7B 1.22x / 1.29x / 1.35x main's audio-s/s at c=1/8/32 with the WER guard passing, Qwen3-Omni level to +7%. Merging when CI is through (the watermark test is red on main itself until #1158).

vasilevklart · 2026-11-04T08:42:48Z

Post-merge on main 482dfc7, 1×H200, Seed-TTS SLO: 101 req/s with 100% in SLO (5,985/5,985; first audio p50/p99 96/128 ms), vs 37 on main before and 96 on the old head (H100, description table).
Closed loop c=1/8/32 21.4/109.3/298.5 audio-s/s; Qwen3-TTS GPU tests 147 pass.

mstar-review Bot added area/model-shared model/qwen3_tts size/XL labels Oct 30, 2026

dghosef self-assigned this Oct 30, 2026

Gaurav-Shah05 added this pull request to stack #989 October 31, 2026 05:45

dghosef mentioned this pull request Oct 31, 2026

Perf push 3: beat vLLM, SGLang and the other baselines on every model dghosef/mstar#17

Draft

vasilevklart marked this pull request as ready for review October 31, 2026 07:46

vasilevklart mentioned this pull request Nov 1, 2026

Qwen3-TTS: Talker on Qwen3Attention, two-token first pass, channels-last codec #1071

Merged

vasilevklart force-pushed the users/kirill/qwen3-tts branch from ce5c995 to f771794 Compare November 1, 2026 06:47

vasilevklart mentioned this pull request Nov 1, 2026

streaming, engine: a longer primed first chunk, a step's piecewise leases in its forward, one sharding key per row batch #977

Merged

dghosef mentioned this pull request Nov 1, 2026

Performance 6/7: OmniVoice, and a Qwen3-TTS decode fix #971

Merged

vasilevklart force-pushed the users/kirill/qwen3-tts branch from f771794 to 7a6bbd7 Compare November 1, 2026 20:59

Base automatically changed from users/kirill/qwen3-tts-engine to main November 1, 2026 21:04

vasilevklart force-pushed the users/kirill/qwen3-tts branch from 7a6bbd7 to e2eba1d Compare November 1, 2026 21:04

This was referenced Nov 2, 2026

Per-model perf and serving fixes for Qwen3.5, Qwen3-Omni, Fish, OmniVoice and others #1140

Merged

Qwen3-TTS: run the codec beside the Talker on a side stream #1073

Merged

merceod requested changes Nov 3, 2026

View reviewed changes

dghosef and others added 8 commits November 3, 2026 00:00

vasilevklart force-pushed the users/kirill/qwen3-tts branch from e2eba1d to f531bd7 Compare November 3, 2026 08:29

mstar-review Bot added size/L and removed size/XL labels Nov 3, 2026

vasilevklart added 2 commits November 3, 2026 01:33

Merge main into the Qwen3-TTS branch

2684684

One conflict, docs/models.rst: the Qwen3-Omni numerics bullet is this branch's, and main's new Thinker prefill-ladder bullet follows it.

vasilevklart and others added 2 commits November 3, 2026 02:45

Merge remote-tracking branch 'origin/main' into HEAD

e7e20c5

# Conflicts: # docs/models.rst

merceod added 3 commits November 3, 2026 11:10

merceod mentioned this pull request Nov 3, 2026

Speed up MoE decode with three fused kernels #984

Open

merceod approved these changes Nov 3, 2026

View reviewed changes

merceod merged commit 7ed765e into main Nov 3, 2026
5 of 6 checks passed

merceod deleted the users/kirill/qwen3-tts branch November 3, 2026 14:16

This was referenced Nov 3, 2026

Qwen3-Omni: a faster Thinker decode step #1111

Merged

Side lane admits a recurrent pool: Qwen3-TTS codec beside the chain again #1175

Open

This was referenced Nov 4, 2026

Qwen3-TTS: Decode Codec Chunks of Every Width in One Step #1182

Merged

Qwen3-TTS: Run the 1.7B Depth Loop as One Persistent Kernel on Hopper #1183

Open

Navigation Menu

Repository navigation

Qwen3-TTS: stacked-head Talker, incremental codec, up to 128 rows a step - #978

Qwen3-TTS: stacked-head Talker, incremental codec, up to 128 rows a step#978
merceod merged 15 commits into
mainfrom
users/kirill/qwen3-tts

vasilevklart commented Oct 30, 2026 •

edited

Loading

Uh oh!

merceod left a comment

Uh oh!

merceod commented Nov 3, 2026

Uh oh!

merceod commented Nov 3, 2026

Uh oh!

merceod left a comment

Uh oh!

Uh oh!

vasilevklart commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

vasilevklart commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merceod left a comment

Choose a reason for hiding this comment

Uh oh!

merceod commented Nov 3, 2026

Uh oh!

merceod commented Nov 3, 2026

Draft paragraph for the PR text

Uh oh!

merceod left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vasilevklart commented Nov 4, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vasilevklart commented Oct 30, 2026 •

edited

Loading