[dghosef]

Stacked on #969. Two encoders were the gaps here. Qwen3.5-9B trailed vLLM by up to 14% on video: its vision tower's block loop ran eagerly, and cuDNN ran the patch embed as a slow implicit-GEMM conv. Qwen3-Omni trailed SGLang-Omni on audio→text, partly because its audio encoder read its chunk layout off the device, so every step paid host syncs. Compiling the block loop is now #859's. This PR makes the patch embed one GEMM and takes the host syncs out of the AuT encoder, which Qwen3-ASR and Qwen3-Omni share since main's #1072. On the stack before the split, with both tower changes, Qwen3.5-9B was within 1.5% of vLLM on video and MMMU on one H200. On Qwen3-Omni, this PR against its parent cuts first-token latency on single images and 60 s videos by 6–9%.

The patch embed's conv has its kernel as its stride, so it is one GEMM over flattened patches (about 17 ms → 1 ms for a 60 s video's patches on Qwen3.5). It lives in qwen3_vit.VisionPatchEmbed, which main's #1072 shares between Qwen3.5 and Qwen3-Omni, so both towers take it. The AuT encoder takes a list of clips instead of a padded stack, pads only the last conv launch, and builds the trim indices and cu_seqlens on the host, so it has no host sync. Its fp16 residual is clamped. The Talker's masks use device ops only, Thinker and Talker fuse the residual add into RMSNorm, and a Thinker step embeds its decode rows in one call. A deployment now sets the captured prefill ladder in its config (model_kwargs.prefill_token_buckets), which replaces MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS. The 9B configs keep main's 2176 bucket, for a mixed step's decode rows beside four prompts, and capture 3072–7168 in place of main's 4352 and 8704 (with those two as well, the ladder would use Dynamo's whole recompile limit); On 9B TP2, 2176 captures for prefill (batch 1–16) and mixed steps (batch 4–128). The mixed-step buckets (MIXED_TOKEN_BUCKETS) now follow each deployment's ladder: its buckets up to 2176, then its powers of two. A test checks every shipped config that sets prefill_token_buckets, qwen3omni_2gpu_large_kv.yaml included. qwen3omni_2gpu.yaml, meant for 80 GB GPUs, keeps Omni's default 128–2048 text ladder, and qwen3omni_2gpu_large_kv.yaml runs it to 8192 on H200-class GPUs. We also add qwen3_5_9b_dp2.yaml (one replica per GPU), H200 MoE tiles for Omni's two expert shapes, and a Thinker prefill row that rides the speculated decode step (speculation_mixes_walks: {Thinker: true}). Two outputs change by design. Vision outputs of Qwen3.5 and Qwen3-Omni aren't bit-identical to main, because the GEMM rounds bf16 differently, so greedy output on long videos and a few MMMU images can diverge. MMMU-200 accuracy is unchanged on both; Video-MME accuracy was not run. And a sub-second Omni or ASR clip encoded alone differs from HF, because every clip's last chunk is padded to keep the encoder batch-invariant.

Unless a row says otherwise: H200 on Eras, torch 2.12.1+cu132, ABBA with 3 boots per arm and a fresh Inductor cache per timed boot. Rows marked "pre-split" ran on the stack before it was split (base 29bf80124, #860's head then), where this PR also carried the compiled block loop. Rows marked "earlier stack top" ran #972's head 0d242a90a against main bb0e0a8d7, before many of main's changes since (among them #963, #919, #939, #1072, #1075, #858 and #920). The whole-stack Qwen3-Omni and Qwen3.5-9B TP2 numbers are in #972.

Check	Hardware	Arms	Result
Qwen3-Omni image and video in, qwen3omni_2gpu.yaml	2× H200	this PR at 3e49609ee (this PR's Omni and vision changes as at this head; main's Omni code has moved since) vs its parent #969 e2a8352b9; the whole PR, not the GEMM alone	TTFT p50: MMMU c=1 −8.6%, 60 s video c=1 −6.2% (both outside spread); 60 s video c=4 −18.3%, MMMU c=16 / 32 −6.9% / −5.9% (within spread). Tok/s: 60 s video c=4 +6.1%, MMMU c=32 +10.8%. MMMU-200 accuracy 0.630 vs 0.625. 60 s video greedy, 8 prompts: 0.89 identical across arms vs 2.33 and 3.33 within each arm. 0 boot hangs, 0 failed
Qwen3.5 patch-embed GEMM, greedy output	H200	earlier stack top vs the same tree with only the patch embed reverted	The deterministic video and MMMU shifts trace to it: reverting it returns 3 of 5 shifted videos and the shifted MMMU image to the parent's exact output
Qwen3.5-9B video 60 / 30 / 8 s and MMMU against vLLM, c=16; c=32	1× H200	pre-split	Gap to vLLM at c=16 −11 / −9 / −4 / −1% → +1.3 / +0.3 / +0.4 / +0.5% (vLLM 765 / 1193 / 1960 / 2260 tok/s); at c=32 −14 / −13 / −8 / −4% → −0.4 / −1.2 / −1.5 / −0.5% (vLLM 893 / 1527 / 2985 / 3658)
Qwen3.5-9B prefill ladder	H200	pre-split	c=64 +1.9% and +4.5% (two A/Bs), c=128 +1.9%, c=16 flat. 0.8B measured −5% and keeps the default
Qwen3.5-9B quality	H200	pre-split	MMMU-200 44.5% → 44.0–45.0%, 195–197 answers identical; GSM8K 93.4 → 94.0%
Qwen3.5-9B DP2 against TP2 (TP-async on), c=16 / 32	2× H200	earlier stack top, before #920's KV-aware routing	Video: DP2 ahead by 12.5–44% (30 s and 60 s clips). MMMU: DP2 16.5% behind at c=16, within spread at c=32. Repeated-prompt TTFT ~60 ms once per replica, then ~16–19 ms
Qwen3.5-9B DP2 against vLLM, c=32	2× H200	pre-split; vLLM DP2 and TP2	60 s video 1520 / 1524 tok/s (mstar TP2 877–1016; vLLM DP2 1504 / 1513, TP2 1324 / 1321). MMMU 4457 / 4500 (mstar TP2 3082–3107; vLLM DP2 4313 / 4540, TP2 5236 / 5121). The TP2 arm's TP-async setting was not recorded
Qwen3.5-9B DP2 under main's #920 routing, c=32	2× H200	qwen3_5_9b_dp2.yaml at #972's earlier head a2b96f6e6 (this PR's config, before it took the 2176 bucket) vs main 3e98682a8 with its 9B config and node_groups ranks [0, 1] (main has no DP2 config), ABBA with 2 boots per arm	Tok/s: 60 s video 1349 / 1336 → 1479 / 1489 (+10.6%), MMMU 4038 / 4052 → 4442 / 4465 (+10.1%). TTFT p50: 60 s video 2326 / 2287 → 2077 / 2063 ms, MMMU 230 / 240 → 155 / 125 ms. All outside spread; 4/4 boots clean, 0 failed. The docs/models.rst numbers (1520 / 4457 tok/s, from load-only routing) are within 2.4% / 0.1% of these. We did not run an arm with the old routing
Qwen3.5-9B TP2 with the 2176 bucket, ShareGPT c=16	2× H200	stack top b27e0609e (the same code as #972's head) vs main eceea1bee, ABBA with 4 boots per arm, with a profile pass	2176 captures for prefill (batch 1–16) and mixed steps (batch 4–128): 276 captures, at most 41 compiles of one function (limit 84), no recompile-limit hits. Tok/s median 3219 vs 3190 (+0.9%, within spread); TPOT p50 3.889 vs 3.945 ms; TTFT p50 18.7 vs 19.5 ms. Iteration p50 3.71 vs 3.70 ms, GPU-bound on both. 0 failed
Qwen3-Omni text→text, c=1 / 4 / 16 / 32	2× H200	pre-split; SGLang-Omni from the Oct-21 benchmark	247 / 556 / 1144 / 1763 → 252 / 580 / 1299 / 2010 tok/s (SGLang-Omni 278 / 633 / 1350 / 2087)
Qwen3-Omni MoE tiles; Thinker prefill riding decode steps	2× H200	pre-split	Tiles: audio→text c=16 753 → 876 tok/s (TTFT 100 → 76 ms), text→text c=32 1955 → 2010. Mixed steps: audio→text c=16 +12%, c=32 +15%; text→text +4–5%
Qwen3-Omni 8192 text-prefill ladder, first text→text c=32 burst	2× H200	pre-split, when qwen3omni_2gpu.yaml carried the ladder that qwen3omni_2gpu_large_kv.yaml has now	1786–1806 → 2029–2030 tok/s; TTFT p99 3.5–3.8 → 0.43–0.45 s
Qwen3-Omni audio→speech, c=8; quality	2× H200	pre-split	45 → 51 audio-s/s, TTFA 446 → 235 ms (text→speech unchanged). GSM8K (250, 5-shot) 0.964 → 0.960 (SGLang-Omni 0.948); LibriSpeech WER 1.0–1.15% on every arm
qwen3omni_2gpu.yaml captures	2× H200	#972's earlier head 7cb636390 vs main ad0a21eca	Thinker 68/68, Talker 24/24, Code2Wav 6/6 buckets captured on every boot, with no 8192 ladder; peak 90–91 GB on the Thinker's GPU on both arms
Qwen3-Omni recompile limit, qwen3omni_2gpu_large_kv.yaml	2× H200	stack top b27e0609e vs main eceea1bee, and the stack with Omni's default ladder; one boot each	Every arm hits Dynamo's recompile limit once in the Thinker's batched forward, from a guard on request-id values, so main does too, with any ladder. With the 8192 ladder, the Thinker's 80 prefill captures all come after that point, against 62 on main's config. ShareGPT c=8: 0 failed on every arm
Qwen3-Omni read-aloud speech (qwen3omni_2gpu.yaml): Whisper-large-v3 WER over 24 sentences	2× H200	main ad0a21eca, #859 576d05f4e, #969 e2a8352b9 and this PR at 3e49609ee, 3 cold boots per arm, one seed shared by all 24 sentences	WER mean: main 1.14% (0.95–1.33%), #859 1.65%, #969 1.84%, this PR 2.91%, against a bar of 2.14% (main's mean plus 1 point). A fourth cold boot each: #969 45% (a 48 s loop on one sentence), this PR 0.95%. The first spoken word is wrong on about 0.3 of 24 sentences per boot on main, 1.3 on #859, 2 on #969 and 4 here; the Thinker's text is correct on every boot. Runaway samples (20–48 s of audio) appear only on stack trees. The earlier Talker runaways on one prompt sit in the same spread (under the same long request history, main has 0 of 24 seeds of that prompt over 20 s). Reverting 98691fc or ad54171 (padding-row positions) or 15f7a20 (the Talker on run_layers) alone, the MoE tile files or speculation mixing changes none of it. These boots sent one seed (7) for all 24 sentences, and the paragraph below the table shows that the shared seed is what separates the arms: with a distinct seed per sentence, this PR is at main's level
Qwen3-Omni: a LibriSpeech question clip answered instead of transcribed	2× H200	main ad0a21eca and the stack's heads then (this PR at 3e49609ee), each after the same long timed history of audio, text and speech requests unless noted	Main, #859, #860 and #969 transcribe it; this PR, #971 and #972 answer it (#972 on 3/3 boots). It needs both the Thinker prefix cache and the long history: with prefix_cache: false, this PR and #972 transcribe it (2/2). Cold boots transcribe it whichever order the clips run in, so the reading that its KV stays frozen from a batched first serving was not confirmed. It is not from the MoE tile files, speculation mixing, or 98691fc, ad54171 or 15f7a20 (each reverted alone); the paragraph below the table has the verdict. Main answers the other 4 of the 5 question clips too; without the five, audio→text WER is equal (#972 1.18–1.29%, main 1.22–1.26%). We could not read the margin: Omni chat drops logprobs and top_logprobs without an error
ruff check .; CPU Core; Rust tests	Eras Linux CPU box, with --rust	this head 7c9b2f30b vs main eceea1bee; Rust at 0fac73afb vs main fdfa1239a	ruff clean. Core: no failures beyond main's (main's 35 come from the box's environment). Rust (test/rust/): 664 passed

Both Qwen3-Omni items above were followed up, and neither is a regression. The read-aloud drift came from the probe sending seed 7 on all 24 sentences. Since #969, fused_sample draws per-row Philox from the request's seed, so all 24 sentences replayed one random stream on every boot. With a distinct seed per sentence (4 cold boots per arm on this PR's earlier head ecba1a4dd), WER is 1.19% with fused_sample and 1.43% without it, and wrong first words are 0.0 vs 1.0 per boot. Main measured 1.38% in the same round. With no seed, WER is 2.19% vs 2.14%. Only the shared seed separates the arms (4.28% vs 1.47%). fused_sample's draws on 69 real Talker and 1019 real code-predictor rows match FlashInfer's at the FlashInfer-vs-FlashInfer noise floor, and the repetition penalty matches to 3.6e-7. Unseeded requests get a per-request seed, so serving traffic doesn't share a stream. The question clip is a near-tie between transcribing and answering: the answer responds to that clip's own question, and main also answered it on 1 of 3 boots of an earlier head.

Not yet run on GPU at this head (7c9b2f30b): Qwen3-ASR on the AuT encoder changes (transcripts against base, c=16 throughput, long-form batches, test_qwen3_asr_parity.py); Qwen3-Omni audio→text with this PR's encoder changes alone (audio_embeds against base for 5 s, 60 s and 10 min clips, encoder phase time); 8 concurrent 60 s clips through audio_attn, and fp16 on a 10 min clip; peak memory for 1 × 600 s + 7 × 2 s clips under the last-launch padding rule; qwen3omni_2gpu.yaml on an 80 GB GPU (whether every capture fits there is not measured); Talker codes and WER against base on DecoderLayer; Thinker padded decode and prefill captures at c=3 / 5 / 7 / 24; Video-MME accuracy.

🤖 Generated with Claude Code

[merceod]

This one can't go in as it stands. Once #969 lands, main conflicts with it in the Omni Thinker's audio and vision prefill (#1110), and the deepstack slice here is wrong on #1110's spliced rows, so the merge needs care and the Omni checks need a rerun on the merged tree. It also hasn't had CI, because the base isn't main. Two of the config changes don't seem to do anything on this base, the 9B rungs above 2176 and the Thinker mixing key, since #916's chunk budgets came after those numbers. The AuT changes reach Qwen3-ASR as well, so I'd like its parity test and a WER pass at this head. The model work itself (patch-embed GEMM, sync-free talker mask and AuT trim, one embed call for decode rows) looks good to me.

• mstar/model/qwen3_omni/submodules.py:615 — Heads up for the merge with main. #1110 builds a media row as the text before, the attachment, then the text after (_spliced_row), so [1:-1] only lines up when there's no text around the image. On a spliced row the media starts at len(before) + 1, so something like start = len(before) + 1 and full_deepstack[start:start + vision_len] = deepstack_inp keeps it host-known and works for both. Main's audio and vision blocks still do mm_mask[[0, -1]] = 0 too, so the scalar writes need to move over.
• mstar/model/components/aut_encoder.py:260 — Qwen3-ASR runs this encoder too since #1072, and the PR lists its parity test and transcripts as not run at this head. Could you run test_qwen3_asr_parity.py and a LibriSpeech pass on Qwen3-ASR before this goes in?
• mstar/model/components/aut_encoder.py:305 — mstar.utils.h2d.to_device_nonblocking already does the pinned copy, and it asks DeviceCapability instead of x.is_cuda. On XPU this branch ends up doing a pageable non_blocking copy, which syncs.
• configs/qwen3_5_9b.yaml:8 — With chunked prefill in the base, a prefill_text step is cut at MAX_BATCH_TOKENS (2048) and a mixed step tops out at 2176, and these configs don't raise max_batch_tokens. So as far as I can tell 3072 to 8192 never replay. The +1.9/+4.5% at c=64 is from the pre-split tree, before #916, and the stack-top TP2 A/B at +0.9% within spread fits that. Can you show a step landing on one of them? If not I'd keep 2176 and drop the rest, since they cost captures and startup on every 9B boot.
• configs/qwen3omni_2gpu_large_kv.yaml:8 — Same question. The Thinker's budget is 1024, so a burst of prompts gets cut into steps of about 1024 plus decode rows, and the 3.5 s to 0.45 s TTFT p99 is from before #916. Does anything reach 4096 to 8192 now?
• configs/qwen3omni_2gpu.yaml:26 — I think this is already the default. speculation_mixes turns on for any walk whose combined walk has a budget, and THINKER_MIXED gets 1024 from get_chunking_policies. So this pins the default rather than turning mixing on, and the +12/+15% mixed-step row predates #916 making it the default. I'd drop it, or say in the comment that it pins the default.
• mstar/model/qwen3_omni/components/thinker.py:251 — The stream starts from input_embeds, which under capture is the static buffer, so the fused norms write into it and the _stage docstring had to change to allow that. forward_residual's own docstring and run_layers start from a copy. You already clone once for layer_0_embed, so could the stream start from the clone and layer_0_embed take input_embeds instead? Same copy count, and static inputs stay read-only, assuming the runner is fine with an output that aliases a static input. With mask_padding off, which these configs use, padding rows also go through the router, so whatever is left in them now carries over between replays.
• docs/models.rst:435 — The MMMU c32 4457 vs ~3100 is from the TP2 arm whose TP-async setting wasn't recorded. Your TP-async-on A/B has DP2 16.5% behind on MMMU at c=16 and within spread at c=32, so I'd name the setting or drop the MMMU part.
• mstar/model/qwen3_5/submodules.py:115 — Someone who still sets MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS now silently gets the default ladder. A one-line warning at load if it's set would help.

[dghosef]

@merceod thanks - all done, and main is merged in (1c6ac24), so CI runs against main now.

• submodules.py:615 deepstack on spliced rows: now start = len(before) + 1, and the scalar mm_mask writes moved over to main's audio and vision blocks (in the merge, 1c6ac24). Added a text-before-and-after image case to test_qwen3_omni_prefix_cache.py that fails with [1:-1].
• Qwen3-ASR: test_qwen3_asr_parity.py passes at this head, and LibriSpeech test-clean (200 clips) is 1.64% / 1.62% WER at c=1 / c=16 vs main's 1.62-1.64% (H100). Transcripts agree on 192-195/200, about as much as main at c=1 vs c=16.
• aut_encoder.py:305: uses to_device_nonblocking now (39d3f72).
• 9B rungs above 2176 and the large_kv ladder: dropped, and the docs claims that came from before [Chunked Prefill] Chunked prefill as a graph Loop (replaces #365) #916 (77834bc).
• Thinker mixing key: dropped from both configs, it pinned the default (77834bc).
• thinker.py:251: the stream starts from a copy and layer_0_embed takes input_embeds, so static inputs stay read-only; the _stage docstring is back to main's (4b9400d). Decode returns layer_0_embed as thinker_states, which is cloned right after the replay, so the alias is fine.
• models.rst:435: the MMMU part is gone, the DP2 note is video only with the TP-async-on numbers (77834bc).
• MSTAR_QWEN35_PREFILL_TOKEN_BUCKETS: warns at load and names model_kwargs.prefill_token_buckets (849de9e).

Omni rerun on the merged tree (H100, one boot per arm, vs main): greedy image/video/audio outputs that differ all split on a near-tie in the HF reference (logit gap 0-0.375), and this branch matches HF on more tokens than main (488 vs 431). Audio WER 2.1% vs 1.9% (one word). a2t tok/s and TTFT within 1.5% at c=1 and c=8. Note qwen3omni_2gpu.yaml OOMs at boot on H100 on both arms; #1140 adds the H100 config.

[merceod]

All addressed, thanks. The spliced-row deepstack start with its test, the AuT index through to_device_nonblocking, the ladders and the mixing key trimmed along with the docs, the stream starting from a copy with the test that input_embeds stays as staged, the warning for the old env var, and the ASR parity and WER pass. CI is green on main. This one goes first, then #971, #972 and #1140 as their bases retarget.

[Gaurav-Shah05]

On CPU the AuT encoder matches the parent exactly for packed batches and clips of a second or more, and the fused add and RMSNorm chain matches the unfused one to 1.4e-6 in fp32, deepstack included.

Overall lgtm, and I have only a comment on the prefill ladder.

[Gaurav-Shah05]

Only Qwen3.5, the Qwen3-Omni Thinker and GLM-5.2 pass this through, so model_kwargs.prefill_token_buckets on Orpheus, Whisper, Qwen3-ASR or the five other AR models that read self.PREFILL_TOKEN_BUCKETS is dropped by the model's **kwargs with no error, a case this class's comment doesn't cover.

The engine should apply a per-node, per-walk ladder from the serving config to the packed capture config, as ChunkingManagement.set_budgets applies max_batch_tokens, refuse a walk with no packed capture, and drop the rungs past the first one that holds the walk's budget, so no model plumbs the key and no config lists rungs that never replay.

[Gaurav-Shah05]

forward now reads proj.weight directly, so module_lora_spec still binds patch_embed.proj as a patch conv, and an adapter's patch_embed.proj factors on the Qwen3.5 or Omni vision tower would validate and add nothing.

VisionPatchEmbed should declare proj in lora_unservable as SiglipEmbeddings in command_a_plus/components/vision.py does, with an ANNOTATED entry in test_lora_targets.py like bagel_converted_patch_embed.