[dghosef]

Note

Still needs some review.

Performance work across 18 models: a few shared changes, then per-model changes.

What changed

Shared

• Conductor: a request goes to the least-loaded data-parallel replica, not a random one.
• Warmup: each warmup request runs once per data-parallel replica, and a warmup spec's concurrency (an int or a list) warms batched sizes.
• Engine: a "one row per step" slot lease for piecewise capture, so a vision tower can batch every request's images into one captured step.
• Ragged attention: the resource owns an mstar::ragged_attention custom op; a resource that declares compile_op keeps attention inside a compiled region's graph instead of breaking there (Wan2.2). Other callers are unchanged.
• DiT scaffold: schedule staging without host syncs.
• Request video decode uses more FFmpeg threads.
• API server: a per-config GIL switch interval (api_switch_interval_s, opt-in).
• Fused MoE router kernel with tuned tiles (opt-in; Qwen3-Omni uses it).
• New serving configs: split (a second worker for the vocoder or codec, with MPS), data-parallel 2-GPU, sequence-parallel 2-GPU (Cosmos3) and CFG-parallel 2-GPU (BAGEL, Wan2.2) presets.

Per model, by technique

Technique	Models
Remove host syncs: stop the CPU waiting on the GPU	Qwen3.5, Qwen3-Omni, OmniVoice, Chatterbox, BAGEL, DiT scaffold
Capture more as CUDA graphs	Qwen3.5 vision, Qwen3-TTS, Fish S2-Pro, Higgs Audio, OmniVoice, Omnilingual ASR, FLUX.2-klein, Z-Image-Turbo, V-JEPA2-AC
Batch across requests: a node that took one request per step takes several	Qwen3.5 vision, Qwen3-Omni audio encoder, Higgs Audio encoder, Omnilingual ASR, Cosmos3 DROID, Qwen3-TTS and Fish S2-Pro (64-row decode)
Batched stop check and device loop-back: less per-request Python per decode step	Qwen3.5, Qwen3-TTS, Orpheus, Higgs Audio
Kernel fusion: fewer, larger GPU kernels	Qwen3-TTS, Fish S2-Pro, Qwen3-Omni, Kokoro, Wan2.2, OmniVoice, Omnilingual ASR
Skip work that doesn't change the output	BAGEL, Orpheus, LingBot-World, Qwen3.5
Compile choices	Omnilingual ASR, Kokoro, Cosmos3, Qwen3.5 vision
Opt-in faster numerics	Fish S2-Pro (TF32 codec), Kokoro (bf16), LingBot-World (fast config)
Output encoding and I/O	BAGEL, Wan2.2, Cosmos3

Per model

• Qwen3.5 text: rotary tables built inside the captured decode graph; one host-to-device sync removed. (Turning on TP async scheduling in the TP configs is left for a separate PR after a soak on current main.)
• Qwen3.5 image/video → text: the vision tower's block loop captured and batched across requests (the shared lease); images stay uint8 and are normalized on the GPU.
• Qwen3-Omni: host syncs removed; first audio chunk is 8 frames instead of 25; fused MoE router; code-predictor fusions; audio encoder batched.
• Qwen3-TTS: 64-row decode; codec batch buckets; a batched stop check; fused add + RMSNorm; voice-clone prefill captured.
• Fish S2-Pro: prefill and decode captured; fused Snake kernel; TF32 codec (config); 64-row decode.
• Orpheus: an LM head over the audio vocabulary only; a batched stop check.
• Higgs Audio: audio encoder batched and captured without per-shape recompiles; a Qwen3.5-style decode loop.
• OmniVoice: backbone captured; host stalls removed; its own layer loop with fused norms; a split config.
• Chatterbox (+ Turbo, Multilingual): a per-row host-to-device sync removed; split configs.
• Kokoro: fused BiLSTM directions; scoped cudnn.benchmark; an opt-in bf16 preset.
• Omnilingual ASR: feature extractor compiled; a packed, captured variable-length encoder; the positional conv as per-group GEMMs.
• BAGEL: CFG branches that don't affect a step are skipped; no host syncs in prepare_inputs; SDPA for single-image ViT; faster PNG encode; a 2-GPU parallel-CFG config.
• FLUX.2-klein: the Qwen3 text encoder is captured.
• Z-Image-Turbo: every batch size 1–8 is captured, not only powers of two.
• Wan2.2: self-attention through the ragged attention resource (FA3), kept inside the compiled graph; GELU fused into its GEMM; faster mp4 muxing; 2-GPU CFG-parallel and DP2 configs.
• Cosmos3: whole-clip VAE decode; the DROID batched action step compiled; per-batch-size warmup; SP2 and DP2 configs.
• LingBot-World: timestep embedding broadcast; channels-last VAE decode; the camera path once per chunk; an opt-in fast config. The default stays deterministic, within ~73 dB of the old decode rather than bit-exact.
• V-JEPA2 / V-JEPA2-AC: the AC rollout encoder captured; rotary tables shared.

Performance

Throughput of this branch vs main, on H200, across the concurrencies run (the vLLM / SGLang comparison benchmark's mstar-pr and mstar-main arms, 2–3 rounds on fresh servers; main at abdfd5802, so before main's last few commits).

Model	Improvement vs main
Qwen3.5-0.8B text	+5–19%
Qwen3.5-9B text	+0–5%
Qwen3.5-0.8B image/video → text	+3–76%
Qwen3.5-9B image/video → text	+1–22%
Qwen3-Omni	speech out +27–31% (TTFA 2.3–2.4× lower); audio → text +26–79%; text +5–14%
Qwen3-TTS (0.6B / 1.7B)	+4–35% / +2–31%
Fish S2-Pro	+48% (c=1) to 2.5× (c=64)
Orpheus	+5–15%
Higgs Audio (speech recognition)	+47% (c=1) to 2.2× (c=8–32)
OmniVoice	4× at c=1, +25–90% at higher concurrency
Chatterbox	+3–10%
Kokoro	+10–17%
Omnilingual ASR (300M / 1B–7B)	+17–30% (−3% at c=64) / 1.7–2.1×
BAGEL	text → image +64–65% on 1 GPU, +91–97% on 2 GPUs; edit +0–3%; understanding +7–23% at c≥4 but −17.5% at c=1
FLUX.2-klein (4B / 9B)	+2–6% / +1–2%
Z-Image-Turbo	+2–13%
Wan2.2	+9–10%
Cosmos3	DROID policy +39–46% (c=4–8); text → video +1–5%
LingBot-World	+19%
V-JEPA2 (ViT-L)	2.0–2.5×

🤖 Generated with Claude Code

[dghosef]

Split into a stack: #858 (shared) → #859 (speech and language models) → #860 (image, video and world models).