[Bugfix] Fix Qwen3-Omni realtime playback interruption and conversation history

[psv666]

Purpose

Qwen3-Omni realtime audio could keep playing after the user started speaking, including when generation had already completed. Cancellation could also wait behind an outstanding audio decode. Stop interrupted playback immediately, discard late output from the old response, and preserve queued events for the next response.

Forward clears for the latest completed response to the engine so its playback cursor is released before the next input; late clears for older responses cannot interrupt a newer turn. Preserve the assistant turn position while unaligned audio awaits playback acknowledgement, remove history items by identity, and evict assistant replies together with discarded input audio.

Follow-up to #7585. The default voice prompt and protocol migration are outside this change.

Test Plan

• Backend regressions: tests/engine/duplex/ and tests/model_executor/models/qwen3_omni/test_duplex_plugin.py.
• Browser regressions: node --test tests/examples/test_realtime_web_profiles.cjs, including actual playback-worklet clearing and asynchronous decode ordering.
• Real Qwen checkpoint, production qwen3_omni_duplex.yaml, Silero VAD, checked-in speech and synthetic camera image: reuse one deployment for 20 interruptions during generation and 20 after generation completes with audio still queued.
• Six-turn image/audio replay with complete playback acknowledgements: require nonempty text/audio, one terminal per response, confirmed history commits, and exact prompts from the real processor. The final two turns must retain four user audio inputs and only their three intervening assistant replies. Interruption cases separately verify empty assistant boundaries.

The original six-turn fixture withheld every playback ACK and required nonempty generation from repeated audio plus consecutive empty assistant turns. A raw Thinker trace showed only EOS (token 151645) on turn three, before event projection. The corrected fixture acknowledges completed playback and checks paired eviction of real answers; output assertions remain enabled.

vLLM Version: 0.29.0; PyTorch 2.13.0+cu129; Transformers 5.14.1.

vLLM-Omni Commit: 2b987db3b7cb216c3eb5ac81cf29e21478d439eb (base 6df62d51573ca5be7826321ba2de1cd071d997f8).

Test Result

Relevant validation	Result
Backend duplex/Qwen regressions	414 passed
Browser/worklet Node regressions	31 passed
Real Qwen interruption during generation	20/20 passed
Real Qwen interruption after generation, before playback completes	20/20 passed
Corrected six-turn image/audio history replay, separate final run	1 passed; all 6 rounds and playback ACKs verified
Pre-commit, including Ruff and Python 3.10 mypy	Passed for the changes; rerun on the final E2E file

GPU results use two NVIDIA L20X GPUs, a local substitute for CI hardware. They do not establish H100/L4 equivalence. No performance claim is made.

Targeted GPU invocation (report paths omitted):

gpu run --gpus 2 --wait 10m --timeout 20m \
  --note "Qwen six-turn history regression" -- \
  uv run --no-sync --project /data/pengsiv/vllm-omni \
  pytest -sv \
  tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::TestQwen3OmniDuplexPlayback::test_mixed_image_audio_history_pruning \
  -m 'advanced_model and cuda' --run-level advanced_model

The interruption run selected the whole TestQwen3OmniDuplexPlayback class with the default 20 repetitions, on one deployment. Its original history case failed as described above; the corrected history case was then rerun independently. Test weights and VAD were supplied from local caches through the existing test environment variables.

Additional local ready/merge CI attempts and remaining gaps

The selected matrix contains 24 deduplicated tasks. It was not completed; nightly/weekly were not run. Results below are the latest attempts, not an all-CI-pass claim.

CI task	Observed result / limitation
CPU Model Executor	2425 passed, 1 explicit legacy Talker skip
CPU Engine / Entrypoints	3024 passed, 1 HF-config-dependent skip, 1 existing images/edits xfail
Custom Pipeline	9 passed
GPU Engine / Model Executor	802 passed, 4 missing-FlashAttention-2 skips; earlier working-tree snapshot, not rerun
Ready Qwen3	9 passed; earlier working-tree snapshot
CPU Diffusion	Interrupted after 3858 passed, 17 failed, 35 skipped; failures were physical-device mapping IndexErrors during multi-rank setup with one visible GPU; remaining cases unverified
Other CPU	Interrupted; log records 2892 passed and 22 runtime skips, plus 4 collection skips; no final JUnit
Merge Qwen3	Interrupted during a subsequent deployment after 3 of 13 selected tests passed; no recorded assertion failure
VeRL	Interrupted in setup (SystemExit: 15); no completed test
Remaining 15 tasks	Not run

Missing-dependency skips (including FlashAttention, torchao, boogu and cosmos), incomplete tasks and hardware-specific paths are not counted as verified. Local prerequisite overrides used isolated temporary directories; no dependency or CI configuration changes are included in this PR. Full upstream CI remains necessary.

AI assistance: Used Codex to review changes, add regression coverage, diagnose the history fixture, run validation, and prepare this description.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/engine_orchestration.md, docs/design/module/observability.md.

Module owners: @tzhouam @fake0fan @lishunyang12

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @lishunyang12 via module of the changed files

@psv666, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit bfe12bb25caa produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[psv666]

@natureofnature @amy-why-3459 @linyueqian PTAL

[amy-why-3459]

I did not find a confirmed blocking correctness issue in this revision. The interruption path, late-response isolation, preservation of queued newer-response events, and identity-based history removal look reasonable. Two non-blocking suggestions are attached inline.

Validation performed: browser/playback-worklet regressions (31 passed), and targeted event projection, EngineSession, and Qwen duplex plugin regressions (140 passed). I did not rerun the real-checkpoint GPU tests; the hardware results in the PR description are author-provided evidence.

[amy-why-3459]

Non-blocking: could we document the lifetime of interruptedResponses, or consider a safe bounded-retention policy?

The set retains every interrupted response ID throughout a running session and is cleared when a new session starts. For a long-running call with repeated interruptions, this grows with the number of interruptions. This is a low-priority lifecycle concern, not a blocker for the playback fix.

Any cleanup must preserve the protection against queued late events and outstanding asynchronous decodes; removing an ID immediately on response.done would not necessarily be safe. A short lifecycle comment or a follow-up with a long-session regression would be useful.

[psv666]

Documented in 11a1fd7. Interrupted response IDs are retained until startSession() clears the set, since queued events and pending audio decodes can outlive response.done. The comment also calls out growth during a call and how sessionGeneration rejects old-session decodes after the set is cleared.

[amy-why-3459]

Non-blocking test suggestion: please cover, or document why ordering rules exclude, a camera image inserted between an evicted audio input and its assistant reply:

evicted user audio -> user camera image -> assistant reply -> retained user audio

Every user message resets dropped_user, including camera image messages. A test would clarify which input owns the assistant reply in this ordering and whether it should survive audio eviction.

I have not established that this ordering causes a reachable correctness bug, so this is a request to make the ordering/ownership invariant explicit rather than a request to change the algorithm without evidence.

[psv666]

Added a session regression that commits audio with create_response=False, inserts a camera image, then creates the response. It checks that the image and audio share a turn and that, after the audio-history limit is reached, that turn's image and assistant reply are evicted together. A separate byte-budget test covers the same ordering with an image-only item.

The adapter now groups and prunes complete turns before format_history() merges user items. Empty assistant boundaries and individually addressable source items are preserved. The ownership rule is documented in realtime_duplex_api.md (11a1fd7).

[Gaohan123]

Please resolve conflicts

[psv666]

@Gaohan123 Resolved in a9c96d9 by merging upstream/main at 6a03e45. Preserved both the playback-interruption regressions and #7800's single-truncation fix and regression test.

Validation of the merged tree: 420 backend duplex/Qwen tests and 31 browser/playback-worklet tests passed. Pre-commit passed, including Python 3.10 mypy.

[Gaohan123]

LGTM. Thanks