Skip to content

[Bugfix] Separate MiniCPM-o seeded instructions from explicit TTS - #8219

Draft
natureofnature wants to merge 1 commit into
vllm-project:mainfrom
natureofnature:fix/minicpmo-explicit-base-a857-20260925
Draft

natureofnature wants to merge 1 commit into
vllm-project:mainfrom
natureofnature:fix/minicpmo-explicit-base-a857-20260925

Conversation

@natureofnature

Copy link
Copy Markdown
Collaborator

Purpose

Follow up on #8152 and #8160 (related to #8138) by distinguishing a native
duplex instruction from an explicitly requested Base TTS response.

#8160 already makes all 128 targets in the tested Seed-TTS subset complete,
including the five previously stalled targets. This PR does not claim those
startup fixes as new work. The remaining issues reproduced on main are:

  • A conditional instruction can generate an initial response even when its
    trigger has not occurred. Both reference/no-reference controls produced
    SPEAK and audio while instructed to remain silent until a red ball appears.
  • Explicit Base TTS uses only a bare assistant header. The
    completed 128-target run still has mean WER 0.074027.
  • SPEAK start controls can consume pending input before content, and a text
    token rejected by the per-unit character cap can still enter repetition
    history.

Keep ordinary seeded instructions as completed user messages before native
audio units. Only the existing explicit task_type=Base selects the full
ordinary-TTS response prefix and a one-time start on the first eligible
sampling step. Delay that start through forced LISTEN; do not rearm it after
cancel or a worker rebuild. Preserve #7974's discarded-lookahead guard and
#8160's control handling, extending the latter to SPEAK. Use the same template
for the worker, first-append reserve, and window suffix accounting.

This does not change the scheduler, Talker, Code2Wav, decoder parameters,
deployment configuration, benchmark inputs, or scoring. The explicit Base
startup policy is not claimed to be the official native duplex default.
The upstream window-test input changes remain untouched.

Test Plan

vLLM Version: 0.30.0 (PyTorch 2.13.0+cu130, Transformers 5.14.1).

vLLM-Omni Commit: main a8576ccb725c4e21cd13c3eb5f9a546b21149d2b plus
fix 58d669f8735c5e91aa47771868b1781947bd547d. Real MiniCPM-o 4.5 checkpoint
503e754207c94da6bb26850b4469f367c9ea3582.

Validation ran in a remote container on H800, one serving GPU and a separate
GPU for Whisper-large-v3. The repository test selection uses the H100 marks.
Before/after runs use the same original 128 English targets, 32 groups of
four sharing each group's original first reference, concurrency 1, no
shuffle/oversampling, and two warmups. The current backend opens an independent
session per utterance; a four-target benchmark group is not a persistent
four-turn server session.

The benchmark uses Thinker temperature 0.7, top-p 0.8, top-k 20; the conditional
E2E controls use temperature 0. WER below is the macro average of per-target
WER, not a corpus-level ratio of all errors to all words.

Accuracy command (the local JSON selects those parameters and the unmodified
vllm_omni/deploy/minicpmo_4_5.yaml):

SEED_TTS_WER_EVAL=1 SEED_TTS_WER_SAVE_ITEMS=1 \
SEED_TTS_WER_SAVE_AUDIO_DIR=/artifacts/seed-tts-audio \
SEED_TTS_EVAL_DEVICE=cuda:1 \
SEED_TTS_HF_WHISPER_MODEL=/cache/whisper-large-v3 \
python -m pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
  --test-config-file /artifacts/wer-c1-p32.json \
  -m 'H100 and B200 and cards_1'
Exact local performance-runner configuration

Save as wer-c1-p32.json, changing only model/dataset cache paths as needed.
The test identifier includes the issue number for artifact naming.

[
  {
    "test_name": "test_minicpmo_4_5_duplex_seed_tts_wer_8138",
    "mark": [
      {"hardware_marks": {"res": {"cuda": ["H100", "B200"], "npu": "A3"}, "num_cards": 1}},
      "full_model", "omni"
    ],
    "server_params": {
      "model": "/cache/hub/models--openbmb--MiniCPM-o-4_5/snapshots/503e754207c94da6bb26850b4469f367c9ea3582",
      "extra_cli_args": ["--deploy-config", "vllm_omni/deploy/minicpmo_4_5.yaml", "--trust-remote-code"]
    },
    "benchmark_params": [{
      "dataset_name": "seed-tts", "dataset_path": "/data/seed-tts",
      "backend": "openai-realtime-tts", "endpoint": "/v1/realtime",
      "num_prompts": [32], "max_concurrency": [1],
      "no_oversample": true, "disable_shuffle": true, "trust_remote_code": true,
      "seed_tts_locale": "en", "seed_tts_turns_per_session": 4,
      "expected_duplex_audio_turns_per_session": 4,
      "extra_body": {"save_duplex_request_metrics": true},
      "percentile-metrics": "ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,audio_duration"
    }]
  }
]

Original config SHA256 (before the whitespace compaction above):
75faba993ac723447858e0e456486b7864cd32f0c18b141f101804bc43b9216f.

E2E commands:

python -m pytest -s -v \
  tests/e2e/online_serving/test_minicpmo_4_5_duplex.py \
  tests/e2e/online_serving/test_minicpmo_4_5_window.py \
  -m 'H100 and cards_1' --run-level=advanced_model

python -m pytest -q tests/worker/test_native_duplex_hooks.py \
  -m 'core_model and cpu' --run-level=core_model

Test Result

Same-base run Completed targets Scored WAVs Mean WER
Unmodified main a8576cc 128/128 128 0.074027
This change on a8576cc 128/128 128 0.044177

Both benchmark runs exit 0. Their completion assertion is separate from the
WER <= 0.05 acceptance target; main does not meet that target, the candidate
does. There are no ASR failures, missing PCM, or audio-save failures in either
run. Every expected target ID and every saved WAV was checked.

The improvement is aggregate, not pointwise: 37 targets improve, 74 are
unchanged, and 17 worsen. No performance improvement is claimed from these
content-validation runs.

  • Broad model/worker/engine/client/connector regressions: 1660 passed,
    2 skipped
    .
  • All 21 pre-existing duplex/window E2Es: 21 passed (1098.46s), including
    continuous two-minute input, camera, buffered flush, and session reuse.
  • The two added conditional-silence cases plus three reference-question and
    cancel/post-completion controls: 5 passed (198.05s), run together in a
    shared-server validation batch. No forced replay after cancel or completion.
  • Applicable changed-file pre-commit hooks passed, including mypy.

The original d31 candidate's completed 128-target run independently obtained
the same mean WER. An additional obsolete d31 control was deliberately
interrupted after reproducing three known timeout groups; it is not counted
as a completed baseline. The table above uses the fully completed latest-main
comparison instead.

Keep conditional native input autonomous and start explicit Base TTS once after forced-listen warmup. Preserve pending input across response controls, exclude cap-rejected tokens from repetition history, and share exact prompt accounting.

Signed-off-by: natureofnature <wzliu@connect.hku.hk>
@hsliuustc0106 hsliuustc0106 added bug Something isn't working tts code related to tts models labels Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working tts code related to tts models

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants