[Bugfix] Skip listen-terminated MiniCPM-o units in Talker handoff

[natureofnature]

Purpose

Fixes #8216.

The benchmark exit code is downstream of a Talker handoff failure. In nightly build 16226, the server raises missing own-token hidden states for [151717] while handling [speak, turn_eos, listen, speak, text..., chunk_eos]; the client then loses its WebSocket and exits with code 1.

Listen-terminated units take the plugin's direct-response path and do not advance the Talker handoff cursor. The next cumulative delta can therefore include an entire earlier unit, rather than just leading listen tokens. The old code attempts to condition the Talker on that earlier unit using the current ledger.

Trim through the last listen token before looking up TTS boundaries. Keep the cumulative cursor recorded before trimming, and retain strict validation of the current unit's hidden states. The regression covers both speak and tts_bos decisions in the skipped/current units, checking the actual condition IDs, hidden rows, and absence of a stale turn-end flag.

Test Plan

vLLM Version: 0.30.0
vLLM-Omni Commit: base c8a1419e02efc74d4a2ac5dfec4e9b2848ecd5a8, fix 4f31c3a166dee2a97a839aafcf7c983764935af4
Environment: isolated container, NVIDIA H800 80 GB; PyTorch 2.13.0+cu130, Transformers 5.14.1, pytest 9.1.1.
Model/config: openbmb/MiniCPM-o-4_5, checkpoint revision 503e754207c94da6bb26850b4469f367c9ea3582; repository minicpmo_4_5.yaml, default three-stage/single-GPU execution.

The unit regression needs the repository test dependencies; the E2E command additionally needs the checkpoint and a supported GPU. Commands run from the source checkout with PYTHONPATH=$PWD:

# New regression applied to the unmodified base processor:
python3 -m pytest -q \
  tests/model_executor/stage_input_processors/test_minicpmo_4_5_omni.py \
  -k skips_prior_listen_terminated_units

# Patched processor, complete module:
python3 -m pytest -q \
  tests/model_executor/stage_input_processors/test_minicpmo_4_5_omni.py \
  -m 'core_model and cpu' --run-level core_model

# Real model/server, default deployment configuration:
python3 -m pytest -q -s \
  tests/e2e/online_serving/test_minicpmo_4_5_duplex.py \
  --run-level advanced_model \
  -k 'test_duplex_single_session_response_required or test_duplex_single_session_video_input or test_duplex_single_session_still_image_input'

Test Result

• Base processor + new regression: 4 failed, each with the hidden-state lookup error.
• Patched complete processor module: 32 passed on 8905bd5949cc9aaee0602afb95d144b787c22ccf.
• Added a same-context regression for two handoffs separated by skipped listen units, including replay checks and exact hidden-row assertions. Moving listen trimming before the cumulative cursor update makes the new test fail; restoring the implementation passes the whole module.
• GPU E2E (production fix 4f31c3a1; subsequent commit only adds a test): 3 passed, 9 deselected, 15 warnings in 616.30s (10m16s): audio response, video input, and still-image input, each with two turns. Exit code 0; no missing-own-token-hidden-states error in the server log.
• Changed-file pre-commit: all applicable checks passed except mypy-3.10. Its 5 diagnostics also reproduce on the unmodified base, with identical messages after normalizing shifted line numbers; no new diagnostic. A subsequent run skipping only this known baseline failure passed.
• Full Omni-DuplexEval generation/judge/scoring nightly is still unverified. Verifying in Progress.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/input_output_modality_contracts.md, docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @gcanlin @tzhouam @alex-jw-brooks

Routing: @gcanlin via module of the changed files, CODEOWNERS; @tzhouam via module of the changed files, CODEOWNERS; @alex-jw-brooks via module of the changed files

@natureofnature, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 8905bd5949cc produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[twu3202]

I tried to find where this started, in case it's useful. I used one RTX 6000 Ada, vLLM 0.30.0, the CI command for the duplex and window files, and a stage memory split adjusted to fit 48 GB. The three tests from #8217 pass at 6715c57 (just before #7695) and fail at 798d52c (#7695) with the own-token error.

Bisecting inside #7695, the change that matters is the MiniCPM-o plugin replacing its _multimodal_output with the shared multimodal_output helper:

• With only that function restored, the full CI selection passes: 20 passed, no own-token errors.
• With only _coerce_int / _coerce_int_list restored, or with the warmup / silence-unit / auto-response changes reverted, it still fails.

The two helpers differ on the Stage-0 output here.

• output.multimodal_output is None, and the completion carries a MultimodalPayload, which is a Mapping but not a dict.
• The old helper returned {}; the shared one copies the payload.
• decide_output then finds listen_token_id in the payload's meta. So a unit that ends a spoken turn and then listens ([speak, …, turn_eos, listen], stop reason listen) is classified as a pure listen and consumed, instead of going to the Talker as before. That unit later sits at the front of the cumulative delta.

In one failing run, a probe computed each decision with both helpers on the same inputs. 63 decisions differed: 62 listen decisions only gained five metadata keys (including latent), and one flipped from forwarding to listen.

One concern with trimming at the handoff is that these units can carry text. In one of my failures, the skipped unit was [151706, 102108, 1773, 151717, 151705, …], i.e. <|speak|>接触。<|turn_eos|><|listen|>…. With the trim, "接触。" would never reach the Talker and wouldn't be spoken. Would it make sense to also keep decide_output from consuming units that contain speech, for example by restoring the dict-only helper in the MiniCPM-o plugin (the one-function change above), and keep the trim as a guard?

[natureofnature]

closed as #8227 tries to solve same problem