[merceod]

Qwen3-Omni couldn't start on Blackwell. Both encoders were HF modules built with attn_implementation="flash_attention_2", FA3 has no sm_103 kernels, and flash_attn 2.x isn't installed on these boxes, so any config with the encoders died at worker init. This moves them onto the engine's attention resources, the way LingBot and Cosmos3 went, and configs/qwen3omni_2gpu.yaml now boots on 2 x B300 with flash_attn mapped in no worker (checked in /proc/<pid>/maps).

What needed flash_attn

Only the two encoders. The Thinker and Talker were already on the paged FlashInfer resource, and Code2Wav uses its own Triton sliding-window kernel, which runs on sm_103 as is.

node	before	now
audio_encoder	HF Qwen3OmniMoeAudioEncoder, FA2 varlen	AuTEncoder through RaggedAttentionSpec audio_attn, one segment per 8 s window
vision_encoder	HF Qwen3OmniMoeVisionEncoder, FA2 varlen	Qwen3OmniVisionTower through RaggedAttentionSpec vision_attn, one segment per image or frame group

Changes

• components/aut_submodule.py: Qwen3-ASR's encoder node moved here as AuTEncoderSubmodule, with the resource key as an argument. Qwen3-ASR and Omni subclass it for prepare_inputs only, so Omni gets request packing and the captured stack.
• components/qwen3_vit.py: Qwen3.5's ViT blocks and grid helpers moved here, with the attention key as an argument. Mergers stay per model since their parameter names differ.
• qwen3_omni/components/vision.py: Omni's tower on those blocks, plus the deepstack taps and mergers. Names match thinker.visual.*, so the checkpoint loads by name.
• VisionEncoderSubmodule follows Qwen3.5's node: one request per step, block loop captured by patch bucket up to 32768. The deepstack taps are extra outputs of the captured region.
• Both encoder nodes set disable_torch_compile. Their stacks are captured, and a compiled wrapper around them would recompile per frame count while the prefix cache holds what earlier graphs computed.
• Omni's tower keeps the rotary inv_freq table fp32 through the worker's dtype cast, as the reference does (median embed error 5.7e-2 to 5.3e-2 against a 4.7e-2 yardstick).
• max_windows_per_replay sizes audio_attn from what the largest bucket can hold (40 windows a request).
• patch_hf_fa2_int_maxlen and the encoders' AutoConfig calls are gone. README, docs/installation.rst, docs/models.rst, pyproject.toml and the e2e requires: lists no longer say Omni needs flash-attn.

No model code branches on device or kernel. FlashInfer picks the backend and a deployment can override it under resources:. Ragged kernels on sm_103 at these head dims, error against fp32 SDPA on random q/k/v, time for one layer:

backend	relative error	vision, 8 frames of 1024	audio, 8 windows of 104
auto	2.1-2.3e-3	220 us	173 us
cutlass	2.2-2.3e-3	216 us	148 us
fa2	2.1-2.3e-3	328 us	146 us
cudnn	2.2-2.4e-3	446 us	292 us
fa3, trtllm-gen	no sm_103 image / no varlen	-	-

Parity

Reference: HF's modules on SDPA, since flash_attn isn't available on this hardware. HF's SDPA path splits per window and frame, so it attends the same spans FA2 does.

Encoder outputs, relative L2 against HF in fp32 (SDPA math backend), median (max) over 20 clips, and 20 images plus a 16-frame clip. The yardstick is HF's own module in bf16:

output	HF bf16	mstar eager	mstar replayed
audio embeds	1.25e-2 (7.1e-2)	1.20e-2 (7.5e-2)	1.21e-2 (6.7e-2)
vision embeds	4.66e-2 (6.7e-2)	5.27e-2 (7.8e-2)	5.16e-2 (6.7e-2)
deepstack 0	1.49e-2 (1.9e-2)	1.49e-2 (1.9e-2)	1.51e-2 (1.9e-2)
deepstack 1	4.12e-2 (6.9e-2)	4.10e-2 (6.7e-2)	4.07e-2 (6.7e-2)
deepstack 2	5.15e-2 (7.4e-2)	5.29e-2 (7.4e-2)	5.15e-2 (7.6e-2)

Everything is at the bf16 yardstick except the final vision embeds, which sit about 12% above it. Four packed clips against each alone differ by 1.3e-2.

End to end, greedy, 128 tokens, 20 inputs per modality at c=1, HF's thinker.generate on M*'s own preprocessing, so a difference is model compute. Long greedy answers flip on rounding alone, so exact counts only mean something next to the yardsticks. Exact matches:

pair	audio transcript	spoken question	image	video letter	text (no encoder)	all
mstar vs HF fp32	19/20	4/20	0/20	20/20	3/20	46/100
HF bf16 vs HF fp32	19/20	6/20	5/20	20/20	5/20	55/100
mstar cold compile vs warm cache	19/20	8/20	3/20	20/20	5/20	55/100
mstar c=1 vs c=4, one process	19/20	5/20	3/20	20/20	5/20	52/100
mstar, two warm-cache processes	20/20	20/20	20/20	20/20	20/20	100/100

What the answers get right doesn't move: transcript WER on these 20 clips 1.81% for mstar and 2.26% for HF, 15/20 video answers on both, Food101 class named in 12-15 of 20 across all runs (HF bf16 13, HF fp32 15), reference answer present in 8 of 20 spoken questions on both.

To separate the encoders from the Thinker, one HF bf16 thinker was fed by different towers, with HF's fp32 towers as gold: mstar's replayed towers match it on 47/80, HF's own bf16 towers on 46/80. The remaining gap to HF is on the Thinker side, which this PR doesn't touch: with no encoder running at all, text-only input matches 3/20 against 5/20 for HF's own bf16. Attributing that to FlashInfer vs SDPA rounding is a hypothesis.

Qwen3-ASR and Qwen3.5 had code moved, so both ran main vs branch at c=1 on GPU, twice each: 20/20 identical transcripts for Qwen3-ASR 0.6B, 20/20 identical answers for Qwen3.5-0.8B image input.

Numbers

2 x B300 SXM6 (sm_103), torch 2.12.1+cu132, FlashInfer 0.7.2 at -O2. Each engine on its default 2-GPU split with prefix caching off, thinker greedy with a 256-token cap, talker at its default sampling, same OpenAI chat payload through benchmark/request.py. Two server sessions per engine, interleaved, server on 8 cores, client on 4, no CPU throttling. Mean (session range). All 2,200 requests of every session succeeded on all three engines.

A2T (200 LibriSpeech clips)

c	RTFx M* / vllm / sglang	TTFT p50/p95 ms M* · vllm · sglang	WER %
1	29.0 (29.0-29.1) / 34.7 (34.5-34.9) / 46.6 (46.3-46.9)	100/135 · 112/165 · 47/65	7.8 · 8.8 · 3.2
4	66.8 (66.8-66.9) / 82.7 (78.1-87.3) / 67.9 (66.5-69.2)	148/184 · 117/185 · 152/225	7.6 · 7.0 · 3.3
16	161.9 (160.5-163.4) / 178.0 (176.4-179.6) / 133.1 (130.8-135.4)	200/256 · 188/271 · 210/284	7.5 · 9.6 · 4.5

A2S (100 spoken questions)

c	audio-s/s M* / vllm / sglang	TTFA p50/p95 ms M* · vllm · sglang	WER %
1	12.1 (12.1-12.2) / 10.2 (10.1-10.2) / 12.0 (12.0-12.1)	211/230 · 146/240 · 225/255	4.6 · 6.8 · 4.7
4	33.9 (33.6-34.2) / 28.3 (27.8-28.8) / 38.1 (38.1-38.1)	281/344 · 181/336 · 274/472	5.5 · 9.4 · 5.1
16	67.1 (67.0-67.1) / 51.7 (50.9-52.5) / 72.0 (71.9-72.1)	458/759 · 314/1616 · 432/833	6.3 · 6.7 · 5.0

I2T (200 images)

c	tok/s M* / vllm / sglang	TTFT p50/p95 ms M* · vllm · sglang
1	178 (176-180) / 231 (230-231) / 206 (205-206)	107/125 · 119/132 · 166/228
4	483 (483-484) / 609 (608-610) / 553 (553-553)	155/193 · 89/145 · 163/240
16	1169 (1166-1173) / 1430 (1416-1443) / 1111 (1106-1117)	184/441 · 121/235 · 192/379

V2T (50 clips, 1-token answers, so TTFT is the number)

c	tok/s M* / vllm / sglang	TTFT p50/p95 ms M* · vllm · sglang
1	1.2 (1.2-1.2) / 1.5 (1.5-1.5) / 0.9 (0.8-0.9)	740/1341 · 618/1025 · 1018/1909
4	1.8 (1.8-1.9) / 6.1 (6.1-6.1) / 3.4 (3.3-3.4)	2181/2995 · 592/779 · 1118/1685

T2S (200 Seed-TTS sentences, read aloud)

c	audio-s/s M* / vllm / sglang	TTFA p50/p95 ms M* · vllm · sglang	RTF p50	WER %
1	10.2 (10.2-10.3) / 9.4 (9.4-9.5) / 9.4 (9.4-9.4)	142/160 · 97/109 · 178/209	0.099 · 0.107 · 0.106	2.9 · 3.2 · 1.7
8	43.5 (43.3-43.8) / 37.7 (37.5-37.9) / 48.6 (48.6-48.6)	252/294 · 201/375 · 204/303	0.182 · 0.210 · 0.163	3.3 · 2.7 · 1.6
32	80.5 (78.8-82.1) / 62.0 (62.0-62.1) / 81.5 (81.5-81.6)	502/755 · 702/1364 · 414/683	0.392 · 0.504 · 0.397	2.8 · 2.5 · 1.6

• M* leads vllm-omni in audio-s/s on every speech cell, even though vllm-omni produced about 17% more audio for the same words (791 s vs 675 s on T2S). Against sglang-omni it is level or ahead at c=1 and T2S c=32, while sglang-omni is 7-12% ahead at A2S c=4/16 and T2S c=8.
• vllm-omni's I2T throughput is 22-30% above M*'s and its A2T 10-24% above. sglang-omni's A2T c=1 is 61% above M*'s, and M* passes it at c=16.
• V2T at c=4 is M*'s worst cell, below.
• A2T WER includes clips where the checkpoint answers the speech instead of transcribing it: 3 on M*, 3-4 on vllm-omni, 1 on sglang-omni at c=1. Without them all three are 1.45-1.53%.
• Spoken-answer text matches 60-61 of 100 between M* and either competitor, and 64 between the two competitors.
• T2S WER is 1.2-1.7 points above sglang-omni's and level with vllm-omni's. The Talker isn't changed here and the cause isn't established.

Peak VRAM: M* 23.0 + 91.5 GB, vllm-omni 250 + 187 GB, sglang-omni 255 + 253 GB. The competitors reserve by fraction, so theirs is reservation rather than need.

What the others ran: vllm-omni 0.30.0 with FLASH_ATTN on the ViT and FlashInfer plus Triton MoE on the thinker and talker (from its log), --media-io-kwargs set to 2 fps to match. sglang-omni 0.2.0 with trtllm_mha on the thinker and talker (from its log), Triton packed attention on the encoders (from its code).

Where the time goes, and what's next

• V2T c=4: recv -> preprocess done is 527 ms at c=1 and 1563 ms of the 2190 ms at c=4, while the encoder is 43 ms and the Thinker prefill 60 ms. PreprocessWorkerThread runs load_video and video_processor inline, 604 ms a clip when timed alone against 556 ms a clip of throughput at c=4. A pool per request would likely bring c=4 TTFT near c=1 (hypothesis, not tried).
• A2T c=1: 269 ms a clip vs 169 ms on sglang-omni. About half is TTFT: the prompt runs as three Thinker prefills (text, audio, text), each a replay of about 14 ms of kernels, 43 ms in all. One step would save about 28 ms (hypothesis). The rest is decode at 5.0 vs 3.8 ms a token, host-bound at c=1 (event_sync p50 0.08 ms, plan about 1.4 ms, launch about 2.1 ms).

Not in this PR

• The two levers above.
• Qwen3.5's tower has the same bf16 inv_freq on main. Left alone so its numerics don't change here; its effect there isn't measured.
• Qwen3-ASR-0.6B doesn't boot on main on this hardware: 150 windows a request at bs=8 overflows the default 512 MB workspace during capture sizing. max_windows_per_replay would fix it; the guard above ran both arms with MSTAR_WORKSPACE_BUFFER_MB=2048.
• Omni's and Qwen3.5's vision nodes could share more than the blocks.
• Code2Wav's sliding-window kernel is still model-owned Triton rather than a resource.
• Prefix caching stays off in these numbers and hasn't been run on GPU with the new encoders. An encoder's output differs by bf16 rounding between a replayed and an eager run (median 9.7e-3 audio, 2.9e-2 vision) and packed vs alone (1.3e-2), so a cached prefix can differ from a recompute by that much.

Tests

• ruff check .: pass.
• CPU Core on this branch against main at c829a71, OMP_NUM_THREADS=2: 8928 passed, 828 skipped, 53 failed vs 8925, 830, 49. All 49 fail on both. The other 4 are test_eval_compare* job-lifecycle tests. All 4 pass on rerun with the machine idle, and the two I retried under load failed the same way on main. test_flex_attention_resource, test_waypoint_components and test_waypoint_dit crash on both. test/ci/test_cpu_selection.py passes.
• New test/modular/test_qwen3_omni_encoders.py (9 tests): the declared specs, the backend built with flash_attn absent, audio windows and deepstack against HF on tiny weights through a stand-in resource, the capture ceiling, the fp32 rotary table, parameter names.
• test_qwen3_omni_prefix_cache.py: the HF-wrapper cases are replaced by a check that the engine leaves both encoders uncompiled. test_trust_remote_code.py: the two encoder cases are gone with the AutoConfig calls.
• GPU, 2 x B300: encoder parity, 100-input end-to-end set, the matrix. The matrix and profiles ran on this code one rebase earlier (base 7eac23c). The rebase changed none of this PR's files, and every cell at c=4 or c=8 on the rebased tree gave 100/100 of the same parity answers and matching wall times.

[vasilevklart]

lgtm. On de85d7b on 2x H200 without flash_attn, qwen3omni_2gpu gets 1.79% WER on 20 LibriSpeech clips, with sane image and spoken replies. Both towers on real weights sit at or under HF bf16's error against HF fp32 SDPA. There is a conflict with #987 in qwen3_5/components/vision.py since both move the same Vit blocks out, to components/qwen3_vit.py here and components/qwen_vision.py there, can rebase if you ar e ready to merge

[merceod]

Thanks for the H200 run. Merging now (branch brought up to date with main first, so CI reruns); #987 can rebase onto it after, and either file is fine with me as the shared home for the ViT blocks.