[Sy0307]

Purpose

Build on #5228 to reduce MiniCPM-o 4.5 end-to-end latency and improve throughput when Thinker, Talker, and Code2Wav share one physical H20.

This PR:

• adds a PyTorch SDPA path for SigLIP and preserves explicit vision-attention overrides;
• regionally compiles each SigLIP encoder layer and enables it in the single-H20 deploy;
• adds opt-in, exact-shape B1/B2 CUDA Graph replay for the full 10-step CFM path, including graph-safe timestep embeddings;
• adds an opt-in request-isolated Token2Wav initial-state template cache;
• adds bounded Code2Wav chunk coalescing and keeps inactive prewarmed streams out of the active scheduler window;
• caches Talker continue/stop logit rows to avoid repeated per-step tensor construction;
• adds focused coverage for attention parity, encoder compile, platform config isolation, graph fallback, cache isolation, scheduling, Talker batching, and connector behavior.

The Code2Wav and Talker performance knobs remain opt-in. Their checked-in defaults stay code2wav_batch_wait_ms=0, token2wav_cfm_cuda_graph=false, and token2wav_initial_state_cache=false. The single-H20 deploy now enables compile_mm_encoder; the NPU platform override replaces that CUDA compilation config with cudagraph_mode=PIECEWISE and does not inherit encoder compile.

The cumulative benchmark additionally used existing deploy overrides for Stage-0/Stage-1 FlashInfer, active_stream_window=4, and SigLIP SDPA. Those overrides are not hardcoded by this source diff.

No quality-oriented setting was reduced: CFM stays at 10 steps, and CFG, precision, codec sampling, output length, frame count, and 25-token audio chunking are unchanged.

Related: #5069, #5228.

Test Plan

• ruff check
• ruff format --check
• git diff --check
• Remote focused tests on exact commit 6051e87f: 32 passed
• Single-H20 Daily-Omni c=1/c=4 matched benchmark with warmup and three measured rounds
• Daily-Omni accuracy unchanged in the matched compile A/B
• Run the full 1,197-request Daily-Omni suite before moving this Draft PR to ready for review

vLLM Version: 0.25.0 (torch==2.11.0+cu128, transformers==5.8.0, CUDA driver 580.82.07)

vLLM-Omni Commit: benchmark prototype based on exact #5228 head f2a216917940a85c44801ccf555a7807426a475d; clean PR commit 6051e87ff326820aebb52555abe0d17b845a6ee3

Test Result

Hardware/topology: one NVIDIA H20 96 GB; all three stages colocated on the same physical GPU.

Workload: the same 20-request Daily-Omni subset and seed for every round. One c=4, 8-request warmup preceded three measured rounds at c=1 and c=4. Values are medians. All rounds completed 20/20 requests without HTTP or evaluation parse failures.

Cumulative result against #5228

Metric	#5228 c=1	Cumulative c=1	#5228 c=4	Cumulative c=4
Request throughput	0.332613 req/s	0.752709 req/s (2.26x)	0.376805 req/s	1.080269 req/s (2.87x)
Audio throughput	0.798272 audio-s/s	1.801986 audio-s/s (2.26x)	0.899621 audio-s/s	2.547274 audio-s/s (2.83x)
Median TTFT	2259.67 ms	834.47 ms (-63.1%)	8990.77 ms	2972.61 ms (-66.9%)
Median audio TTFP	2665.08 ms	1063.90 ms (-60.1%)	9936.30 ms	3259.31 ms (-67.2%)
Median audio RTF	1.26697	0.55277 (-56.4%)	4.58565	1.59431 (-65.2%)
Daily-Omni accuracy	13/20	13/20	13/20	13/20
Median audio duration	2.28 s	2.28 s	2.28 s	2.28 s

These are stacked end-to-end results for the cumulative configuration, not attribution to any single optimization. The clean main port has been revalidated with the focused test suite; the cumulative benchmark was collected on the corresponding prototype based on the exact #5228 head.

SigLIP regional compile attribution

The only variable in this matched A/B was compile_mm_encoder. Both sides included the same experimental Vision host-metadata fast path, so the relative deltas isolate encoder compile; the metadata fast path itself is not included in commit 6051e87f.

Metric	c=1 change	c=4 change
Request throughput	+2.08%	+0.74%
Audio throughput	+3.28%	+1.33%
Total time	-2.03%	-0.74%
Median TTFT	-3.07%	-2.56%
Median audio TTFP	-2.37%	-3.09%
Median audio RTF	-3.40%	-3.16%
Daily-Omni accuracy	65% to 65%	65% to 65%

Post-warmup profiling attributes the gain to SigLIP encoder fusion:

• Stage-0 accelerator time: 2834.9 ms -> 2737.1 ms (-3.45%);
• layout/direct-copy time: 73.4 ms -> 22.4 ms;
• residual add and LayerNorm fuse into a Triton kernel;
• the initial AOT compile takes about 9 seconds once, after which all encoder layers reuse the cache;
• no graph-break or recompilation error was observed.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

[Gaohan123]

@Sy0307 Hello, any updates?