[natureofnature]

Purpose

MiniCPM-o 4.5's default CUDA Code2Wav path runs CAM++ speaker embedding through CPU ONNX Runtime. Reference preparation still runs separately for each voice, so this host work delays the output of a batch even when its CFM decode can batch different references.

Use the existing torch CAM++ implementation on CUDA with FP32 weights and inputs, loading the same campplus.onnx weights. CUDA reference preparation follows the stage's matmul policy (token2wav_allow_tf32, enabled by default); cuDNN's convolution TF32 policy is independent. FP32 tensors do not imply strict FP32 arithmetic. token2wav_device_spk: false retains the CPU fallback, and explicitly enabling token2wav_trt retains its TensorRT path. Other callers of the shared device-speaker helper keep their existing default behavior.

Scheduling capacity, chunk sizes, sampling, CFM/HiFT precision, and MPS settings are unchanged. The existing MiniCPM performance contract documents the CUDA path and fallback.

Contributor intent

1. What outcome do you want to achieve?

Improve MiniCPM-o 4.5 serving throughput and first-audio latency with different reference voices.

2. What is the highest-leverage variable for that outcome?

The serial CPU work in reference preparation before Code2Wav can return a batch.

3. How will this work move that variable?

Run speaker embedding on CUDA using the existing torch implementation.

4. Whose customer or user needs does this address, and what are those needs?

Users serving concurrent MiniCPM-o 4.5 speech requests on one GPU who need responsive first audio and higher throughput with their reference voices.

Test Plan

vLLM Version: 0.32.0; PyTorch 2.13.0+cu130; ONNX 1.23.2; ONNX Runtime 1.31.0. Recorded NVIDIA driver: 535.161.08.

vLLM-Omni Commit: measured PR revision ac6df62263a9ac8b27b593fd43a34c3dde1c1bc4 on main 3e40008bbeedfdad7d82e1fd25e76db5820025c0. The full C1/4/8/16/32 sweep was frozen at baseline 18c3bd5d2d4323bc7adca1ac3c465c4a53bbfb6e, with and without this PR's production patch. After #8911 changed the shared Talker path, the PR was rebased and a separate C16 A/B plus C1/C8 quality checks were run on the final revisions. All three changed production files match between the two campaigns; all 32 official benchmark files match between sources. Follow-up commit 3f39c7187a9faa535943a796c9276ff0fa14f683 only clarifies precision documentation; production and test files are unchanged.

• One H800 80GB, MPS off. The selected GPU was idle before each server start; the host is shared.
• Unmodified official vllm bench serve --omni, openai-chat-omni backend, C1/4/8/16/32.
• Model openbmb/MiniCPM-o-4_5 at 073dbbc8c5bc0af2d789e1ce12e7c17a6be746e1.
• Seed-TTS English at 8f5e1aa2a35d42f42e940074c1983358b9491f89: 64 fixed evaluation requests with 63 reference WAVs. Exact IDs and hashes: input manifest.
• Thinker/Talker BF16, CFM FP32 with ordinary TF32, HiFT FP32, 25-frame chunks. Thinker capacity 16 / KV 4 GiB; Talker 8 / KV 2 GiB; Code2Wav 8; CFM microbatch 4 and graph budget 32. TP=1, no quantization. Thinker/Talker use enforce_eager: false; Code2Wav uses enforce_eager: true with its internal CFM/HiFT CUDA graphs enabled.
• Two independent starts per source in the primary comparison, with opposite source/concurrency orders. Each start has 32 disjoint warmup requests at C16, one discarded 64-request round per C, then two measured 64-request rounds per C. Four measured rounds per source/C.

Test Result

C	Source	Audio throughput (audio-s/s)	TTFA p50 (ms)	TTFA p95 (ms)	Mean RTF
1	main	7.883	366.69	417.42	0.1302
1	PR	8.345	324.09	365.69	0.1222
4	main	16.190	716.70	807.70	0.2593
4	PR	18.006	570.54	713.54	0.2291
8	main	18.584	1124.34	1330.76	0.4353
8	PR	23.307	871.32	1075.66	0.3505
16	main	20.392	1339.48	2438.64	0.7851
16	PR	24.978	1098.03	1639.47	0.6376
32	main	21.527	1570.50	4347.12	1.4830
32	PR	25.178	1301.35	3375.25	1.2430

Throughput is synthesized audio-seconds per benchmark wall-second. RTF is the official CLI's mean audio_rtf. TTFA columns average the official per-run percentiles.

Full-resolution PNG · SVG. Lines show means and shading shows the min–max across four measured rounds. This figure summarizes the full sweep on baseline 18c3bd5d2; the latest-main C16 confirmation is reported separately below.

Per-run ranges, request throughput, and output duration

Ranges cover all four measured rounds per source/C. Each row uses 64 requests per round.

C	Source	Audio-s/s range	TTFA p50 range (ms)	TTFA p95 range (ms)	Mean RTF range	Mean req/s	Mean output audio (s)
1	main	7.77–8.02	359.9–372.1	404.7–432.1	0.1279–0.1319	2.190	230.40
1	PR	8.22–8.47	321.5–327.3	361.9–369.4	0.1205–0.1240	2.318	230.40
4	main	15.96–16.51	703.3–738.7	797.9–823.7	0.2550–0.2638	4.460	232.32
4	PR	17.69–18.44	564.2–576.8	686.9–774.0	0.2235–0.2340	4.993	230.80
8	main	17.70–19.06	1050.5–1226.0	1268.4–1438.2	0.4235–0.4500	5.109	232.77
8	PR	22.62–24.02	843.1–890.4	1040.6–1118.2	0.3408–0.3597	6.370	234.33
16	main	19.82–21.19	1233.9–1488.3	2284.7–2579.5	0.7604–0.8063	5.640	231.36
16	PR	23.85–25.44	1038.4–1170.8	1550.0–1834.5	0.6254–0.6628	6.857	233.15
32	main	20.06–22.84	1505.8–1650.8	4225.2–4467.8	1.4119–1.5765	5.969	230.80
32	PR	24.71–25.95	1215.2–1418.7	3150.7–3857.2	1.2226–1.2690	6.959	231.56

Observed peak GPU memory from 1-second NVML samples across the primary starts (including startup): 55035 MiB main → 55057 MiB PR. These are sampled device-memory peaks, not allocator high-water marks.

Generated audio duration varies slightly with stochastic speech sampling. Both audio throughput and request throughput are reported. These results apply to this H800 reference-audio workload; H200 and BF16 CFM were not tested.

Latest-main confirmation (after #8911): two measured 64-request C16 rounds per source after fresh server starts, with the same settings and official CLI. All 256 measured requests succeeded. These results are separate from the full sweep above.

C	Source	Audio-s/s, mean (range)	TTFA p50 (ms)	TTFA p95 (ms)	Mean RTF (range)
16	main	21.697 (21.519–21.874)	1434.75	2267.65	0.7299 (0.7212–0.7386)
16	PR	25.957 (25.083–26.832)	1068.33	1651.67	0.6172 (0.5995–0.6349)

Mean request throughput: 5.848 → 7.059 req/s. Mean output audio per 64 requests: 237.44 → 235.54 s.

The first attempted main start failed during initial HiFT warmup with a cuDNN engine error followed by a CUDA illegal memory access, before any measured rounds. The replacement pair and the primary comparison were selected before inspecting candidate performance. All 2,560 primary measured requests succeeded. The additional successful PR sweep is retained separately: 640/640 requests, mean audio-s/s C1=8.50, C4=18.54, C8=23.98, C16=24.70, C32=25.18.

Speaker embedding parity: 63 real reference WAVs, with CUDA matmul TF32 both off and on (126 cases). Minimum cosine similarity of normalized embeddings: 0.9999985; maximum normalized absolute difference: 4.23e-4. Isolated mean forward time, after warming each input shape: 42.61 ms CPU → 10.52 ms CUDA. This component measurement is separate from the serving benchmark.

Native Seed-TTS WER: separate quality runs, 64/64 requests per source and case. The values are mean per-request normalized WER from the pinned Whisper model below.

Baseline revision	C	main WER	PR WER
18c3bd5d2	8	3.766%	5.896%
3e40008bb	1	4.373%	4.689%
3e40008bb	8	8.813%	4.815%

The initial C8 difference is retained: two utterances with extra trailing ASR text account for about 2.00 percentage points of the 2.13-point difference. The follow-up C8 comparison goes in the opposite direction and also contains long-tail baseline outputs. The two C8 comparisons also use different baseline revisions, so they are not repeated trials of the same source pair. These small samples do not establish a quality improvement, statistical equivalence, or the absence of a regression. Attribution remains unresolved, including any contribution from this PR's CUDA speaker-embedding change; sampling or concurrency variability has not been established as the cause.

In the C1 comparison, all 64 waveform durations match and 62 per-utterance WER values match. The two differences are thirty versus 30 in ASR output, and two theories abound at versus two theories abounded (the reference says Two theories abound to). The official scores above are unchanged.

All MRv2 ASR evaluations completed. Saved WAVs were validated as nonempty 24 kHz mono PCM16. Speaker similarity/MOS and NPU/TensorRT hardware were not measured.

V1 limitation: the V1 content check failed in this vLLM 0.32.0 environment. The PR returned 16/16 HTTP successes, but all outputs were about 82 seconds and Thinker emitted repeated ! up to the 256-token limit. A separate unmodified 18c3bd5d2 baseline reproducer showed the same text/length failure on both tested requests. This is retained as a baseline V1 failure; V1 end-to-end quality is not counted as passed.

Focused regressions:

221 passed, 14 warnings in 15.27s

python -m pytest -q \
  tests/model_executor/models/minicpmo_4_5/test_code2wav_device_spk_flag.py \
  tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py \
  tests/model_executor/models/step_audio2/test_step_audio2_token2wav_async_chunk.py

Local pre-commit passed, including mypy, test marks, and forbidden-import/CUDA-call checks.

Reproduction: inputs, server configuration, and official benchmark commands

Run from each source checkout. The full sweep uses baseline 18c3bd5d2d4323bc7adca1ac3c465c4a53bbfb6e with and without this PR's patch; the latest-main C16 check uses the two final revisions reported above. Set MODEL_PATH to the pinned model snapshot above. Paths below can be placed in any working directory.

Create the two native Seed-TTS input directories from the fixed manifest:

import hashlib
import json
from pathlib import Path
from urllib.request import urlopen

manifest_url = "https://gist.githubusercontent.com/natureofnature/c7012977b58bbd737fd51a0b7f1c6cd3/raw/00ad2db82a7d5e46419fa94c779adb414ca88bbd/input-manifest.json"
manifest = json.loads(urlopen(manifest_url).read())
prefix = "https://huggingface.co/datasets/zhaochenyang20/seed-tts-eval/resolve/" + manifest["dataset_revision"] + "/en/"
rows = {line.split("|", 1)[0]: line for line in urlopen(prefix + "meta.lst").read().decode().splitlines() if "|" in line}
refs = {row["id"]: row for row in manifest["references"]}
for subset, key in (("evaluation", "evaluation_ids"), ("warmup", "warmup_ids")):
    target = Path("inputs") / subset / "en"
    target.mkdir(parents=True, exist_ok=True)
    (target / "meta.lst").write_text("\n".join(rows[item] for item in manifest[key]) + "\n")
    for item in manifest[key]:
        ref = refs[item]
        path = target / ref["reference"]
        path.parent.mkdir(parents=True, exist_ok=True)
        if not path.exists():
            path.write_bytes(urlopen(prefix + ref["reference"]).read())
        assert hashlib.sha256(path.read_bytes()).hexdigest() == ref["sha256"]

Both measured source copies use the same one-line compatibility adjustment for the vLLM 0.32.0 embedding API; it is excluded from this PR:

from pathlib import Path
p = Path("vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py")
p.write_text(p.read_text().replace(
    "self.llm.get_input_embeddings(input_ids)",
    "self.llm.embed_input_ids(input_ids)",
))

cat > turn-bench.yaml <<YAML
base_config: $PWD/vllm_omni/deploy/minicpmo_4_5_turn_mrv2.yaml
connectors:
  connector_of_shared_memory:
    extra:
      micro_batch_size: 4
      cfm_graph_precapture: true
      cfm_max_graphs: 32
      hift_graph_capture_batch_sizes: [1, 2, 4, 8]
stages:
  - stage_id: 0
    max_num_seqs: 16
    max_model_len: 4096
    max_num_batched_tokens: 4096
    limit_mm_per_prompt: {image: 0, audio: 0, video: 0}
  - stage_id: 1
    max_num_seqs: 8
  - stage_id: 2
    max_num_seqs: 8
platforms:
  cuda:
    cuda_mps: false
    stages:
      - stage_id: 0
        kv_cache_memory_bytes: 4294967296
      - stage_id: 1
        kv_cache_memory_bytes: 2147483648
YAML

CUDA_VISIBLE_DEVICES=0 PYTHONPATH="$PWD" HF_HUB_OFFLINE=1 \
VLLM_WORKER_MULTIPROC_METHOD=spawn \
vllm serve "$MODEL_PATH" --omni --deploy-config "$PWD/turn-bench.yaml" \
  --trust-remote-code --served-model-name crossref-test \
  --host 127.0.0.1 --port 18479

In another terminal with the same checkout, model, and input paths:

cat > extra-body.json <<'JSON'
{"modalities":["text","audio"],
 "chat_template_kwargs":{"enable_thinking":false,"use_tts_template":true},
 "sampling_params_list":[
   {"temperature":0.0,"top_p":1.0,"top_k":-1,"max_tokens":256,"seed":42,"detokenize":true,"repetition_penalty":1.0},
   {"temperature":0.8,"top_p":0.85,"top_k":25,"repetition_penalty":1.05,"min_tokens":50,"max_tokens":2048,"seed":42,"detokenize":false},
   {"temperature":0.0,"top_p":1.0,"top_k":-1,"max_tokens":65536,"detokenize":true}
 ]}
JSON

export PYTHONPATH="$PWD" HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false
export SEED_TTS_WER_EVAL=0 SEED_TTS_SIM_EVAL=0 SEED_TTS_UTMOS_EVAL=0
# Use a separate output directory for every server start.
export RESULT_DIR="$PWD/results"
bench() {
  vllm bench serve --omni --backend openai-chat-omni \
    --base-url http://127.0.0.1:18479 --endpoint /v1/chat/completions \
    --model crossref-test --tokenizer "$MODEL_PATH" --trust-remote-code \
    --dataset-name seed-tts --dataset-path "$PWD/inputs/$1" \
    --seed-tts-locale en \
    --seed-tts-system-prompt "Please read the following text out loud in English." \
    --disable-shuffle --num-prompts "$2" --output-len 256 --seed 42 \
    --request-rate inf --max-concurrency "$3" --num-warmups 0 \
    --ready-check-timeout-sec "$5" --extra-body "$(cat extra-body.json)" \
    --percentile-metrics audio_ttfp,audio_rtf,audio_duration,e2el \
    --metric-percentiles 50,95 --disable-tqdm \
    --save-result --save-detailed --result-dir "$RESULT_DIR/c$3" \
    --result-filename "$4.json"
}
bench warmup 32 16 warmup 60
for c in 1 4 8 16 32; do
  bench evaluation 64 "$c" discard 0
  bench evaluation 64 "$c" r1 0
  bench evaluation 64 "$c" r2 0
done

The reverse-order pair uses C32/16/8/4/1. Each source starts a fresh service for each sweep.

Quality evaluation is a separate native CLI run, excluded from the performance
table. Set ASR_PATH to the local snapshot of
openai/whisper-large-v3@06f233fe06e710322aca913c1bc4249a0d71fce1 and use the same bench function:

SEED_TTS_WER_EVAL=1 SEED_TTS_WER_SAVE_ITEMS=1 \
SEED_TTS_WER_SAVE_AUDIO_DIR="$RESULT_DIR/audio" \
SEED_TTS_EVAL_DEVICE=cuda:0 SEED_TTS_HF_WHISPER_MODEL="$ASR_PATH" \
CUDA_VISIBLE_DEVICES=0 bench evaluation 64 8 quality 0

The V1 smoke uses a fresh candidate server with the same overrides and
minicpmo_4_5_turn.yaml as base_config, and 16 evaluation requests at C8.
The baseline V1 reproducer uses baseline 18c3bd5d2, the same V1 overrides,
two warmup requests, two discarded evaluation requests, and two scored requests
at C8. It checks the termination failure; it is excluded from performance results.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-20T17:28:33.988894Z	ac6df62	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@natureofnature, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[amy-why-3459]

Change looks good to me. The use_trt=False keyword keeps the shared StepAudio2Token2WavCore.enable_device_spk_embedding() default (TRT on CUDA) for other callers, token2wav_trt still takes its own branch first, and get_campplus_torch is cached per (onnx_path, device index), so multiple Code2Wav replicas share one CAM++ copy. One doc precision nit inline.

[amy-why-3459]

Small precision nit: on CUDA the reference preparation (chunk 0 and the idle prefetch) runs inside _stage_matmul_policy, which turns on torch.backends.cuda.matmul.allow_tf32 under the default token2wav_allow_tf32, and cuDNN convs use TF32 by PyTorch default. So with default settings CAM++ isn't strictly FP32 on CUDA. Your parity table already covers TF32 on/off, so behavior is fine; it might just be worth saying "FP32 weights; matmuls follow token2wav_allow_tf32" here so the accuracy-triage guidance (token2wav_device_spk: false) is read correctly.

[natureofnature]

Agree with @amy-why-3459. In ac6df62, both chunk-0 preparation and idle prefetch still enter _stage_matmul_policy, while CampplusTorch keeps weights and inputs in FP32. Non-blocking: describing this as FP32 weights with the stage's CUDA matmul policy (and separate cuDNN policy) would make the precision guidance clearer.

[natureofnature]

@natureofnature

[natureofnature]

Reviewed all five changed files and traced backend selection, prompt preparation, CAM++ weight loading/device caching, and speaker conditioning. The CPU opt-out and explicit TRT branch remain intact, and other callers retain the shared helper's TRT default. The reported final-base H800 MRv2 checks, TF32-on/off embedding parity, and focused tests cover the changed CUDA path; current CI checks pass. This was static review, without rerunning those tests. The remaining documentation suggestion is non-blocking and noted in the existing thread.

【Conclusion: Found no blocking issues in commit ac6df62.】
【Action recommended: None. Optional: Clarify FP32 weights versus CUDA compute precision in the documentation.】

[natureofnature]

@natureofnature

[natureofnature]

Rechecked the update against ac6df62. The documentation now correctly distinguishes FP32 weights and inputs from CUDA matmul and cuDNN compute policies, resolving the earlier non-blocking suggestion. Runtime code and tests are unchanged, so the previously reviewed CUDA-path validation still applies. Current-head documentation, CodeQL, pre-commit, wheel builds and DCO checks have all passed; I did not rerun the hardware tests.

【Conclusion: Found no blocking issues in commit 3f39c71.】
【Action recommended: None.】