Repository navigation
[Bugfix] Separate MiniCPM-o seeded instructions from explicit TTS - #8219
Draft
natureofnature wants to merge 1 commit into
Draft
natureofnature wants to merge 1 commit into
natureofnature wants to merge 1 commit into
Conversation
Keep conditional native input autonomous and start explicit Base TTS once after forced-listen warmup. Preserve pending input across response controls, exclude cap-rejected tokens from repetition history, and share exact prompt accounting. Signed-off-by: natureofnature <wzliu@connect.hku.hk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Follow up on #8152 and #8160 (related to #8138) by distinguishing a native
duplex instruction from an explicitly requested Base TTS response.
#8160 already makes all 128 targets in the tested Seed-TTS subset complete,
including the five previously stalled targets. This PR does not claim those
startup fixes as new work. The remaining issues reproduced on main are:
trigger has not occurred. Both reference/no-reference controls produced
SPEAK and audio while instructed to remain silent until a red ball appears.
completed 128-target run still has mean WER 0.074027.
token rejected by the per-unit character cap can still enter repetition
history.
Keep ordinary seeded instructions as completed user messages before native
audio units. Only the existing explicit
task_type=Baseselects the fullordinary-TTS response prefix and a one-time start on the first eligible
sampling step. Delay that start through forced LISTEN; do not rearm it after
cancel or a worker rebuild. Preserve #7974's discarded-lookahead guard and
#8160's control handling, extending the latter to SPEAK. Use the same template
for the worker, first-append reserve, and window suffix accounting.
This does not change the scheduler, Talker, Code2Wav, decoder parameters,
deployment configuration, benchmark inputs, or scoring. The explicit Base
startup policy is not claimed to be the official native duplex default.
The upstream window-test input changes remain untouched.
Test Plan
vLLM Version: 0.30.0 (PyTorch 2.13.0+cu130, Transformers 5.14.1).
vLLM-Omni Commit: main
a8576ccb725c4e21cd13c3eb5f9a546b21149d2bplusfix
58d669f8735c5e91aa47771868b1781947bd547d. Real MiniCPM-o 4.5 checkpoint503e754207c94da6bb26850b4469f367c9ea3582.Validation ran in a remote container on H800, one serving GPU and a separate
GPU for Whisper-large-v3. The repository test selection uses the H100 marks.
Before/after runs use the same original 128 English targets, 32 groups of
four sharing each group's original first reference, concurrency 1, no
shuffle/oversampling, and two warmups. The current backend opens an independent
session per utterance; a four-target benchmark group is not a persistent
four-turn server session.
The benchmark uses Thinker temperature 0.7, top-p 0.8, top-k 20; the conditional
E2E controls use temperature 0. WER below is the macro average of per-target
WER, not a corpus-level ratio of all errors to all words.
Accuracy command (the local JSON selects those parameters and the unmodified
vllm_omni/deploy/minicpmo_4_5.yaml):SEED_TTS_WER_EVAL=1 SEED_TTS_WER_SAVE_ITEMS=1 \ SEED_TTS_WER_SAVE_AUDIO_DIR=/artifacts/seed-tts-audio \ SEED_TTS_EVAL_DEVICE=cuda:1 \ SEED_TTS_HF_WHISPER_MODEL=/cache/whisper-large-v3 \ python -m pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \ --test-config-file /artifacts/wer-c1-p32.json \ -m 'H100 and B200 and cards_1'Exact local performance-runner configuration
Save as
wer-c1-p32.json, changing only model/dataset cache paths as needed.The test identifier includes the issue number for artifact naming.
[ { "test_name": "test_minicpmo_4_5_duplex_seed_tts_wer_8138", "mark": [ {"hardware_marks": {"res": {"cuda": ["H100", "B200"], "npu": "A3"}, "num_cards": 1}}, "full_model", "omni" ], "server_params": { "model": "/cache/hub/models--openbmb--MiniCPM-o-4_5/snapshots/503e754207c94da6bb26850b4469f367c9ea3582", "extra_cli_args": ["--deploy-config", "vllm_omni/deploy/minicpmo_4_5.yaml", "--trust-remote-code"] }, "benchmark_params": [{ "dataset_name": "seed-tts", "dataset_path": "/data/seed-tts", "backend": "openai-realtime-tts", "endpoint": "/v1/realtime", "num_prompts": [32], "max_concurrency": [1], "no_oversample": true, "disable_shuffle": true, "trust_remote_code": true, "seed_tts_locale": "en", "seed_tts_turns_per_session": 4, "expected_duplex_audio_turns_per_session": 4, "extra_body": {"save_duplex_request_metrics": true}, "percentile-metrics": "ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,audio_duration" }] } ]Original config SHA256 (before the whitespace compaction above):
75faba993ac723447858e0e456486b7864cd32f0c18b141f101804bc43b9216f.E2E commands:
Test Result
Both benchmark runs exit 0. Their completion assertion is separate from the
WER <= 0.05 acceptance target; main does not meet that target, the candidate
does. There are no ASR failures, missing PCM, or audio-save failures in either
run. Every expected target ID and every saved WAV was checked.
The improvement is aggregate, not pointwise: 37 targets improve, 74 are
unchanged, and 17 worsen. No performance improvement is claimed from these
content-validation runs.
2 skipped.
continuous two-minute input, camera, buffered flush, and session reuse.
cancel/post-completion controls: 5 passed (198.05s), run together in a
shared-server validation batch. No forced replay after cancel or completion.
The original d31 candidate's completed 128-target run independently obtained
the same mean WER. An additional obsolete d31 control was deliberately
interrupted after reproducing three known timeout groups; it is not counted
as a completed baseline. The table above uses the fully completed latest-main
comparison instead.