[Core] Generalized turn-based /v1/realtime implementation by NickCao · Pull Request #8339 · vllm-project/vllm-omni

NickCao · 2026-09-30T18:09:57Z

Purpose

Fully generalized turn-based /v1/realtime implementation based on /v1/chat/completion semantic. It works the same way as /v1/chat/completion, except for that the messages are in /v1/realtime shape, and the session history is maintained on the server.

Test Plan

vLLM Version: 0.30.0

vLLM-Omni Commit: 49f9fbf

Unit/E2E Tests

# from test-ready.yml
pytest -sv tests/entrypoints tests/engine -m 'core_model and cpu'
# from test-merge.yml
pytest -sv tests/entrypoints -m "advanced_model and cuda and H100" --run-level "advanced_model" --deselect tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::TestQwen3OmniRealtimeWebSocket::test_server_vad_multi_turn_without_client_commit

Manual Tests

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --omni --port 8091 --enable-auto-tool-choice --tool-call-parser hermes

MIMO_AUDIO_TOKENIZER_PATH=<path to XiaomiMiMo/MiMo-Audio-Tokenizer> \
vllm serve XiaomiMiMo/MiMo-Audio-7B-Instruct \
  --omni --port 8091 \
  --chat-template ./examples/online_serving/mimo_audio/chat_template.jinja

vllm serve openbmb/MiniCPM-o-4_5 \
  --omni --port 8091 \
  --deploy-config vllm_omni/deploy/minicpmo_4_5_turn.yaml --trust-remote-code \
  --default-chat-template-kwargs '{"use_tts_template": true, "enable_thinking": false}' \
  --enable-auto-tool-choice --tool-call-parser hermes

Then chat with them using any openai /v1/realtime client of your choice (Use https://github.com/NickCao/livekit-openai-test/blob/master/src/livekit_openai_test/agent.py as an example.)

Should work with any omni model supporting /v1/chat/completion.

Test Result

Unit/E2E Tests

3641 passed, 4 skipped, 134 deselected, 1 xfailed, 34 warnings in 168.37s (0:02:48)
15 passed, 2363 deselected, 22 warnings in 867.17s (0:14:27)

Manual Tests

Qwen/Qwen3-Omni-30B-A3B-Instruct
XiaomiMiMo/MiMo-Audio-7B-Instruct
openbmb/MiniCPM-o-4_5

All behaves correctly in a conversation, and qwen3 omni and minicpm are able to make tool calls.

Known issues

Automatic truncation would try dropping turns iteratively until the rendered prompt fits, this can incur high prompt rendering cost if there's a large number of short turns. And auto truncation pre-flight requires rendering the prompt twice, first time bypassing mm cache.

vllm-omni-review-bot · 2026-09-30T18:37:40Z

This PR appears to belong to: docs/design/module/profiling.md, docs/design/module/entrypoints.md, docs/design/module/observability.md.

Module owners: @alex-jw-brooks @linyueqian @gcanlin

Routing: @alex-jw-brooks via module of the changed files, semantic router, CODEOWNERS; @linyueqian via module of the changed files, semantic router, CODEOWNERS; @gcanlin via module of the changed files

@NickCao, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

NickCao · 2026-09-30T19:32:22Z

@vllm-omni-review-bot

vllm-omni-review-bot · 2026-09-30T19:34:22Z

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

vllm-omni-review-bot

Omni ReviewBot review

PR description

This PR replaces the model-gated / STT turn-based /v1/realtime fallback with a single chat-backed OpenAI Realtime session path. Prompt building now goes through OmniOpenAIServingChat._preprocess_chat (standard input_audio WAV content, shared template kwargs, rendered prompt_token_ids for auto-truncation), and audio output is resampled to the Realtime 24 kHz PCM contract. Native duplex remains opt-in via /v1/realtime?duplex=1 / /v1/duplex; non-duplex sessions require a live engine_client and openai_serving_chat.

Change flow

flowchart TD
  A["[EXISTING] Client /v1/realtime"]:::existing --> B["[CHANGED] api_server duplex vs turn route"]:::changed
  B -->|"duplex=1"| C["[EXISTING] OmniDuplexSessionHandler"]:::existing
  B -->|"turn / duplex=0"| D["[CHANGED] dispatch_realtime_websocket"]:::changed
  D --> E["[CHANGED] OpenAIFullDuplexConnection + chat preprocess"]:::changed
  F["[REMOVED] OpenAIServingRealtime / RealtimeConnection STT"]:::removed --> D
  E --> G["[NEW] WAV input + resampled PCM deltas + server history"]:::new
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

CI at 49f9fbf3b614 (2026-09-30T19:34:24.801256+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

Findings

[P1] Turn-based realtime forces every stage to DELTA, breaking MiniCPM-o audio — vllm_omni/entrypoints/openai/realtime/connection.py:898
_run_response_inner clones default_sampling_params_list and sets output_kind=RequestOutputKind.DELTA on every stage before engine.generate. Chat streaming already documents that MiniCPM-o 4.5 audio requires Stage0 (llm) FINAL_ONLY and only Talker/TTS as DELTA (OmniOpenAIServingChat._fix_minicpmo45_audio_stream_output_kinds); otherwise llm2tts sees per-token spans and fragments audio. This PR newly routes MiniCPM turn deploy (minicpmo_4_5_turn.yaml) through OpenAIFullDuplexConnection, so the chat fix is bypassed. Reuse chat_handler._fix_minicpmo45_audio_stream_output_kinds(sampling_params_list, modalities) after the DELTA default, and add a regression that asserts MiniCPM Stage0 stays FINAL_ONLY for audio modalities.
[P2] STT realtime removal leaves shipped example docs on a deleted contract — examples/online_serving/realtime_web/README.md:83
App init no longer mounts OpenAIServingRealtime, and realtime_connection.py is deleted, so duplex=0 / non-duplex now always uses chat-semantic OpenAIFullDuplexConnection (or unsupported). examples/online_serving/realtime_web/README.md still documents the legacy STT adapter against vllm_omni/entrypoints/openai/realtime_connection.py and claims duplex=0 selects that STT handler. Update or remove that STT section in the same change so operators are not guided to a removed wire protocol.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-09-30T19:41:49Z

Omni ReviewBot: finding feedback

[p1] Turn-based realtime forces every stage to DELTA, breaking MiniCPM-o audio — vllm_omni/entrypoints/openai/realtime/connection.py:898

_run_response_inner clones default_sampling_params_list and sets output_kind=RequestOutputKind.DELTA on every stage before engine.generate. Chat streaming already documents that MiniCPM-o 4.5 audio requires Stage0 (llm) FINAL_ONLY and only Talker/TTS as DELTA (OmniOpenAIServingChat._fix_minicpmo45_audio_stream_output_kinds); otherwise llm2tts sees per-token spans and fragments audio. This PR newly routes MiniCPM turn deploy (minicpmo_4_5_turn.yaml) through OpenAIFullDuplexConnection, so the chat fix is bypassed. Reuse chat_handler._fix_minicpmo45_audio_stream_output_kinds(sampling_params_list, modalities) after the DELTA default, and add a regression that asserts MiniCPM Stage0 stays FINAL_ONLY for audio modalities.

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

NickCao · 2026-10-01T12:46:58Z

Retested with the in-tree webui:

NickCao · 2026-10-01T12:55:44Z

And livekit with minicpm testing tool calling and barge-in:

vllm-omni-review-bot

Omni ReviewBot review

PR description

This PR removes the legacy STT /v1/realtime adapter and routes turn-based realtime (duplex=0 / non-duplex) through chat-semantic OpenAIFullDuplexConnection, building prompts via OmniOpenAIServingChat._preprocess_chat so any chat-capable omni model can keep server-side session history. It also deletes realtime_connection.py / OpenAIServingRealtime wiring, updates the shared realtime web UI from --stt to --manual, and reuses MiniCPM Stage0 FINAL_ONLY streaming kinds for audio. Operators get one OpenAI Realtime turn protocol instead of a Qwen-only STT path, including on duplex servers that previously refused duplex=0.

Change flow

flowchart TD
  A["[EXISTING] Realtime WS client<br/>/v1/realtime?duplex=0"]:::existing --> B["[CHANGED] api_server / duplex.openai<br/>dispatch_realtime_websocket"]:::changed
  C["[REMOVED] OpenAIServingRealtime<br/>realtime_connection.RealtimeConnection"]:::removed --> B
  B --> D["[CHANGED] OpenAIFullDuplexConnection<br/>chat preprocess + engine.generate"]:::changed
  E["[EXISTING] OmniOpenAIServingChat<br/>_preprocess_chat / MiniCPM output_kinds"]:::existing --> D
  F["[NEW] realtime prompt preflight tests<br/>WAV input + FINAL_ONLY regression"]:::new --> D
  D --> G["[CHANGED] Turn-based Realtime session<br/>server history, text/audio/tool events"]:::changed
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

CI at bff6754fed60 (2026-10-01T20:55:31.863208+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

Findings

[P2] Shipping docs still claim duplex=0 is refused on duplex servers — docs/serving/README.md:157
docs/serving/README.md still says duplex=0 selects a turn-based handler that a duplex server does not mount and is refused with Realtime API is not available. This PR intentionally changes that contract: dispatch_realtime_websocket now builds OpenAIFullDuplexConnection whenever engine_client and openai_serving_chat exist, and test_realtime_opted_out_of_duplex_falls_through_to_the_turn_based_route asserts session.created on duplex+chat app state for duplex=0/false/off. Duplex servers that also serve chat (the common MiniCPM/Qwen case) will now accept the opt-out as chat-semantic turn realtime instead of hard-refusing. Update the serving docs to the new mount/opt-out semantics so operators do not follow a removed refuse contract.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-10-02T13:20:45Z

Omni ReviewBot: superseded

The CI failure noted on a2babc7cf32d refers to an earlier head; the pull request now points at 899521ebd9c9.

vllm-omni-review-bot

Omni ReviewBot review

PR description

This PR replaces the Qwen-only / STT fallback /v1/realtime stack with one chat-semantic turn-based handler: buffered Realtime messages are converted into ordinary chat multimodal inputs (PCM wrapped as WAV), rendered through OmniOpenAIServingChat._preprocess_chat, and generated with server-side history. On duplex servers, duplex=0/false/off now opts into that turn-based path instead of being refused, while duplex=1 remains native duplex. The shared realtime web UI and serving docs are updated for the new opt-out and --manual turn mode.

Change flow

flowchart TD
  A["[EXISTING] Realtime WebSocket client"]:::existing
  B["[CHANGED] api_server /v1/realtime duplex routing"]:::changed
  C["[CHANGED] dispatch_realtime_websocket"]:::changed
  D["[REMOVED] OpenAIServingRealtime + RealtimeConnection STT path"]:::removed
  E["[CHANGED] OpenAIFullDuplexConnection"]:::changed
  F["[NEW] chat preprocess + PCM16-to-WAV prompt bridge"]:::new
  G["[EXISTING] AsyncOmni.generate + session history"]:::existing
  A --> B
  B -->|"duplex=1 / default duplex"| G
  B -->|"duplex=0 or non-duplex"| C
  C --> E
  D -.->|"removed fallback"| C
  E --> F
  F --> G
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

CI at a2babc7cf32d (2026-10-02T13:46:35.054718+00:00): required check(s) blocking: buildkite/vllm-omni (failed). Observed Buildkite: buildkite/vllm-omni (failed), buildkite/vllm-omni-amd-ci (failed), buildkite/vllm-omni-intel-ci (passed), and 1 more.

See inline comments below.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-10-02T13:50:51Z

+    await websocket.close()


 async def dispatch_realtime_websocket(websocket: WebSocket) -> None:


[P1] Test Plan omits covering MiniCPM duplex CI guards

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and H100 advanced entrypoints sweep (plus manual chats), and does not state that these duplex e2e guards were skipped. That leaves the new duplex↔turn-based dispatch contract without the CI-defined MiniCPM duplex evidence.

vllm-omni-review-bot · 2026-10-02T14:05:24Z

Omni ReviewBot: finding feedback

[p1] Test Plan omits covering MiniCPM duplex CI guards — vllm_omni/entrypoints/duplex/openai.py:44

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and H100 advanced entrypoints sweep (plus manual chats), and does not state that these duplex e2e guards were skipped. That leaves the new duplex↔turn-based dispatch contract without the CI-defined MiniCPM duplex evidence.

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

vllm-omni-review-bot

Omni ReviewBot review

PR description

This PR replaces the Qwen3-specific / STT turn-based /v1/realtime path with a chat-completions-backed handler that renders Realtime history through OmniOpenAIServingChat._preprocess_chat, keeps session history server-side, and resamples model audio to the Realtime 24 kHz PCM contract. On duplex servers, ?duplex=0 now reaches that turn-based handler instead of being refused, while default/duplex=1 still uses native duplex. Docs and the realtime_web UI are updated for the new turn/manual vs VAD split, and STT RealtimeConnection is removed.

Change flow

flowchart TD
  A["[EXISTING] Client /v1/realtime"]:::existing --> B["[CHANGED] api_server realtime_websocket<br/>duplex vs duplex=0"]:::changed
  B -->|"duplex default/1"| C["[EXISTING] OmniDuplexSessionHandler"]:::existing
  B -->|"duplex=0 / turn server"| D["[CHANGED] dispatch_realtime_websocket"]:::changed
  D --> E["[CHANGED] OpenAIFullDuplexConnection<br/>chat preprocess + resample"]:::changed
  E --> F["[REMOVED] STT RealtimeConnection / OpenAIServingRealtime"]:::removed
  E --> G["[NEW] test_realtime_prompt_preflight.py"]:::new
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

CI at 899521ebd9c9 (2026-10-02T14:28:51.271745+00:00): required check(s) blocking: buildkite/vllm-omni (pending), docs/readthedocs.org:vllm-omni (pending). Observed Buildkite: buildkite/vllm-omni (pending), buildkite/vllm-omni-npu-ci (pending), buildkite/vllm-omni-intel-ci (pending), and 1 more.

See inline comments below.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-10-02T14:32:17Z

+    await websocket.close()


 async def dispatch_realtime_websocket(websocket: WebSocket) -> None:


[P1] Test Plan omits covering duplex and Qwen3 Omni CI guards

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The serving_chat.py tok_params plumbing is also listed under *qwen3_omni → omni_qwen3_omni_function, whose Ready step is pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model' --run-level 'core_model'. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and the H100 advanced entrypoints sweep (plus manual chats), and does not state that these covering e2e guards were skipped. That leaves the new duplex↔turn-based dispatch and chat-preprocess realtime path without the CI-defined MiniCPM duplex / Qwen3 online evidence.

vllm-omni-review-bot · 2026-10-02T14:36:56Z

Omni ReviewBot: finding feedback

[p1] Test Plan omits covering duplex and Qwen3 Omni CI guards — vllm_omni/entrypoints/duplex/openai.py:44

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The serving_chat.py tok_params plumbing is also listed under *qwen3_omni → omni_qwen3_omni_function, whose Ready step is pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model' --run-level 'core_model'. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and the H100 advanced entrypoints sweep (plus manual chats), and does not state that these covering e2e guards were skipped. That leaves the new duplex↔turn-based dispatch and chat-preprocess realtime path without the CI-defined MiniCPM duplex / Qwen3 online evidence.

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

hsliuustc0106

Reviewed head 899521e. Two findings below.

Validation: static Python syntax and diff hygiene checks passed. Runtime/GPU tests were not run; this review is based on pinned source inspection, including the vLLM 0.30.0 renderer and multimodal cache contracts.

Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

Document duplex=0 selecting the turn-based Realtime handler on duplex servers and remove the outdated refusal guidance. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

Render prompt-budget preflights with the processor-only multimodal cache, then rerender accepted prompts through the normal sender cache for engine submission. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

Stage auto-truncation deletions during preflight and apply them only after the candidate prompt fits and its final render succeeds. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

Use a cache-isolated preflight before cancelling an active response, then prepare the submitted prompt again from finalized session history. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

vllm-omni-review-bot

Omni ReviewBot review

Changes since the previous review

1 new inline finding(s); 1 finding(s) below.
Still needs attention: Test Plan omits covering duplex and Qwen3 Omni CI guards — vllm_omni/entrypoints/duplex/openai.py
Still needs attention: Test Plan omits covering MiniCPM duplex CI guards — vllm_omni/entrypoints/duplex/openai.py

CI at 07245967a7e6 (2026-10-05T15:30:12.628442+00:00): required check(s) blocking: pre-commit (pending), buildkite/vllm-omni (pending), docs/readthedocs.org:vllm-omni (pending). Observed Buildkite: buildkite/vllm-omni-npu-ci (pending), buildkite/vllm-omni-amd-ci (pending), buildkite/vllm-omni (pending), and 1 more.

Full review analysis

PR description

This PR replaces the Qwen-only plus legacy STT /v1/realtime paths with one turn-based OpenAI Realtime handler that renders session history through ordinary /v1/chat/completions preprocessing and keeps conversation state on the server. Explicit duplex=0 on a duplex server now opts into that turn-based handler instead of being refused, and the shared realtime web UI/docs follow the same manual-turn contract. Speculative auto-truncation preflight bypasses the multimodal sender cache and, when replacing an active reply, re-renders after cancellation so truncated transcript prefixes are included.

Change flow

flowchart TD
  A["[EXISTING] Realtime WS client"]:::existing
  B["[CHANGED] api_server duplex query routing"]:::changed
  C["[CHANGED] dispatch_realtime_websocket uses chat_handler"]:::changed
  D["[REMOVED] RealtimeConnection STT handler"]:::removed
  E["[CHANGED] OpenAIFullDuplexConnection chat preprocess"]:::changed
  F["[NEW] skip_mm_cache preflight + post-cancel re-render"]:::new
  G["[EXISTING] AsyncOmni.generate + session history"]:::existing
  A --> B
  B -->|"duplex=0 / no duplex"| C
  C --> D
  C --> E
  E --> F
  F --> G
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Findings

[P2] Preflight skip_mm_cache fix lacks a regression pin — vllm_omni/entrypoints/openai/realtime/connection.py:487
Existing thread: #8339 (comment)

Evidence for Preflight skip_mm_cache fix lacks a regression pin

_prepare_engine_input_with_auto_truncation correctly keeps speculative renders on skip_mm_cache=True and only submits a final render through the normal sender cache. That addresses the MultiModalCacheMissError risk called out on this PR. However tests/entrypoints/openai_api/test_realtime_prompt_preflight.py never asserts skip_mm_cache (its fake _preprocess_chat even drops the kwarg) and does not exercise real sender/receiver cache behavior. A later change that forgets the bypass can pass the current unit suite and still fail at runtime when auto-truncation preflight populates MultiModalProcessorSenderCache. Add a regression that records skip_mm_cache on speculative renders and asserts the submitted path does not.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-10-05T15:33:43Z

+    await websocket.close()


 async def dispatch_realtime_websocket(websocket: WebSocket) -> None:


[P1] Test Plan omits covering duplex and Qwen3 Omni CI guards

Evidence and suggested fix

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The serving_chat.py tok_params/skip_mm_cache plumbing is also listed under *qwen3_omni → omni_qwen3_omni_function, whose Ready step is pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model' --run-level 'core_model'. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and the H100 advanced entrypoints sweep (plus manual chats), and does not state that these covering e2e guards were skipped. That leaves the new duplex↔turn-based dispatch and chat-preprocess realtime path without the CI-defined MiniCPM duplex / Qwen3 online evidence.

Account for the final cache-backed render after truncation preflight accepts the prompt. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

vllm-omni-review-bot

Omni ReviewBot review

Changes since the previous review

1 new inline finding(s); 1 finding(s) below.
Still needs attention: Preflight skip_mm_cache fix lacks a regression pin — vllm_omni/entrypoints/openai/realtime/connection.py
Still needs attention: Test Plan omits covering duplex and Qwen3 Omni CI guards — vllm_omni/entrypoints/duplex/openai.py
Still needs attention: Test Plan omits covering MiniCPM duplex CI guards — vllm_omni/entrypoints/duplex/openai.py

CI at 4f19ab0e4f6e (2026-10-06T03:37:30.391974+00:00): all required and operator-watched checks reported green. Observed Buildkite: buildkite/vllm-omni-amd-ci (failed), buildkite/vllm-omni (passed), buildkite/vllm-omni-npu-ci (failed), and 1 more.

Full review analysis

PR description

This PR generalizes turn-based /v1/realtime so any Omni model that already supports /v1/chat/completions can run Realtime sessions with server-side history. Prompt building now goes through serving_chat._preprocess_chat (WAV audio parts, tools, chat templates) instead of Qwen3-specific placeholders, and duplex servers can opt into that turn-based path with duplex=0. The old STT RealtimeConnection / OpenAIServingRealtime stack is removed, with auto-truncation preflight kept off the multimodal sender cache and re-rendered after cancel/truncate before submission.

Change flow

flowchart TD
  Client["[EXISTING] OpenAI Realtime WS client"]:::existing
  Dispatch["[CHANGED] duplex/openai.dispatch_realtime_websocket<br/>duplex=0 → turn-based"]:::changed
  Conn["[CHANGED] OpenAIFullDuplexConnection<br/>chat_handler + WAV prompts"]:::changed
  Pref["[NEW] _prepare_engine_input_with_auto_truncation<br/>preflight_only + skip_mm_cache"]:::new
  Chat["[CHANGED] serving_chat._preprocess_chat<br/>tok_params / skip_mm_cache"]:::changed
  Removed["[REMOVED] realtime_connection.RealtimeConnection<br/>+ OpenAIServingRealtime"]:::removed
  Engine["[EXISTING] AsyncOmni /v1/chat stages"]:::existing
  Client --> Dispatch --> Conn --> Pref --> Chat --> Engine
  Removed -.-> Conn
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Findings

[P2] Preflight skip_mm_cache fix lacks a regression pin — vllm_omni/entrypoints/openai/realtime/connection.py:487
Existing thread: #8339 (comment)

Evidence for Preflight skip_mm_cache fix lacks a regression pin

_prepare_engine_input_with_auto_truncation correctly keeps speculative renders on skip_mm_cache=True and only submits a final render through the normal sender cache. That addresses the MultiModalCacheMissError risk called out on this PR. However tests/entrypoints/openai_api/test_realtime_prompt_preflight.py never asserts skip_mm_cache (its fake _preprocess_chat even drops the kwarg) and does not exercise real sender/receiver cache behavior. A later change that forgets the bypass can pass the current unit suite and still fail at runtime when auto-truncation preflight populates MultiModalProcessorSenderCache. Add a regression that records skip_mm_cache on speculative renders and asserts the submitted path does not.

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

vllm-omni-review-bot · 2026-10-06T03:40:34Z

+    await websocket.close()


 async def dispatch_realtime_websocket(websocket: WebSocket) -> None:


[P1] Test Plan omits covering duplex and Qwen3 Omni CI guards

Evidence and suggested fix

Changed paths under vllm_omni/entrypoints/duplex/, vllm_omni/entrypoints/openai/api_server.py, and tests/entrypoints/openai_api/test_duplex_api_server.py are covered by source_file_dependencies: omni_minicpmo_4_5_duplex_function in .buildkite/common/ci_source_file_dependencies.yml. Ready runs pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'; merge runs the duplex/window suite with advanced_model and cuda. The serving_chat.py tok_params/skip_mm_cache plumbing is also listed under *qwen3_omni → omni_qwen3_omni_function, whose Ready step is pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model' --run-level 'core_model'. The PR Test Plan/Result only reports the CPU entrypoints/engine selector and the H100 advanced entrypoints sweep (plus manual chats), and does not state that these covering e2e guards were skipped. That leaves the new duplex↔turn-based dispatch and chat-preprocess realtime path without the CI-defined MiniCPM duplex / Qwen3 online evidence.

NickCao · 2026-10-06T14:18:11Z

Working on a followup addressing the known issues.

…m-project#8339) Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Codex <noreply@openai.com>

Upstream's turn-based /v1/realtime rework (vllm-project#8339) dropped OpenAIServingRealtime from vllm_omni's api_server. The monkeypatch.setattr(api_server, "OpenAIServingRealtime", ...) calls in the generative-scoring guards now raise AttributeError and take the whole generate path down with them: AttributeError: <module 'vllm_omni.entrypoints.openai.api_server'> has no attribute 'OpenAIServingRealtime' Remove the two dead patch entries. Signed-off-by: apex-mochen <2756823972@qq.com>

NickCao marked this pull request as ready for review September 30, 2026 18:36

NickCao requested review from alex-jw-brooks, linyueqian and yenuo26 as code owners September 30, 2026 18:36

NickCao mentioned this pull request Sep 30, 2026

[Refactor] Use rendered prompt for turn based /v1/realtime auto truncation #8292

Closed

NickCao added the omni code related to omni models label Sep 30, 2026

vllm-omni-review-bot reviewed Sep 30, 2026

View reviewed changes

NickCao requested review from Gaohan123, hsliuustc0106 and ywang96 as code owners September 30, 2026 22:12

hsliuustc0106 added the high priority high priority issue, needs to be done asap label Sep 30, 2026

vllm-omni-review-bot reviewed Oct 1, 2026

View reviewed changes

NickCao force-pushed the codex/realtime-generalization branch from bff6754 to a2babc7 Compare October 2, 2026 13:06

NickCao requested a review from david6666666 as a code owner October 2, 2026 13:06

NickCao added the ready label to trigger buildkite CI label Oct 2, 2026

vllm-omni-review-bot reviewed Oct 2, 2026

View reviewed changes

NickCao added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 2, 2026

vllm-omni-review-bot reviewed Oct 2, 2026

View reviewed changes

NickCao mentioned this pull request Oct 2, 2026

[RFC]: vLLM-Omni Q4 2026 Roadmap — Community Discussion #8264

Open

45 tasks

hsliuustc0106 reviewed Oct 3, 2026

View reviewed changes

Comment thread vllm_omni/entrypoints/openai/realtime/connection.py Resolved

Comment thread vllm_omni/entrypoints/openai/realtime/connection.py Resolved

NickCao and others added 7 commits October 5, 2026 10:43

[Core] Generalize Realtime transcript truncation

0633452

Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Doc] Use standard Realtime events for manual turns

4ee07a8

Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Bugfix] Preserve MiniCPM audio streaming in Realtime

a8efadf

Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Doc] Clarify turn-based Realtime routing

32d7078

Document duplex=0 selecting the turn-based Realtime handler on duplex servers and remove the outdated refusal guidance. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Bugfix] Defer Realtime history truncation

393f4fa

Stage auto-truncation deletions during preflight and apply them only after the candidate prompt fits and its final render succeeds. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Bugfix] Rebuild Realtime prompt after cancellation

0724596

Use a cache-isolated preflight before cancelling an active response, then prepare the submitted prompt again from finalized session history. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

NickCao force-pushed the codex/realtime-generalization branch from 899521e to 0724596 Compare October 5, 2026 15:25

NickCao added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 5, 2026

vllm-omni-review-bot reviewed Oct 5, 2026

View reviewed changes

[Bugfix] Update Realtime preflight render expectations

4f19ab0

Account for the final cache-backed render after truncation preflight accepts the prompt. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Nick Cao <ncao@redhat.com>

NickCao added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 5, 2026

vllm-omni-review-bot reviewed Oct 6, 2026

View reviewed changes

This was referenced Oct 6, 2026

[RFC]: Native-duplex tool-call timeout, cancellation, and recovery #8542

Open

[RFC]: Optional Semantic VAD in native-duplex sessions: scope and lifecycle safety #8543

Open

amy-why-3459 approved these changes Oct 6, 2026

View reviewed changes

amy-why-3459 merged commit 65699fe into vllm-project:main Oct 6, 2026
6 of 9 checks passed

MrlixiangWE mentioned this pull request Oct 6, 2026

[Perf][Qwen3-Omni] Capture the thinker image encoder with CUDA graphs (part of #7452) #8215

Merged

NickCao mentioned this pull request Oct 6, 2026

[BugFix] Optimize turn-based /v1/realtime auto truncation #8566

Merged

NickCao added a commit to neuralmagic/nm-vllm-omni-ent that referenced this pull request Oct 8, 2026

[carry][Core] Generalized turn-based /v1/realtime implementation (vll…

81da504

…m-project#8339) Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Codex <noreply@openai.com>

khairulkabir1661 mentioned this pull request Oct 8, 2026

Add /v1/realtime WebSocket conformance tests (RHAISTRAT-2749) #8258

Merged

NickCao added a commit to neuralmagic/nm-vllm-omni-ent that referenced this pull request Oct 9, 2026

[carry][Core] Generalized turn-based /v1/realtime implementation (vll…

68b72d0

…m-project#8339) Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Codex <noreply@openai.com>

dougbtv mentioned this pull request Oct 9, 2026

[carry] Add v0.30.0 realtime carry record neuralmagic/nm-vllm-omni-ent#64

Merged

apex-mochen mentioned this pull request Oct 10, 2026

[Bugfix] Register tool-call parser metrics in omni's init paths #8212

Closed

natureofnature mentioned this pull request Oct 12, 2026

[RFC]: Integrate JoyAI-VL-Interaction into the Unified Duplex Framework #8803

Closed

psv666 mentioned this pull request Oct 12, 2026

[RFC]: Duplex: Qwen3 Omni duplex 上下文优化 JiusiServe/vllm-omni#326

Closed

1 task done

		await websocket.close()


		async def dispatch_realtime_websocket(websocket: WebSocket) -> None:

Conversation

NickCao commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Unit/E2E Tests

Manual Tests

Test Result

Unit/E2E Tests

Manual Tests

Known issues

Uh oh!

vllm-omni-review-bot commented Sep 30, 2026

Uh oh!

NickCao commented Sep 30, 2026

Uh oh!

vllm-omni-review-bot commented Sep 30, 2026

Uh oh!

vllm-omni-review-bot left a comment

Choose a reason for hiding this comment

Omni ReviewBot review

PR description

Change flow

Findings

Uh oh!

vllm-omni-review-bot commented Sep 30, 2026

Omni ReviewBot: finding feedback

Uh oh!

NickCao commented Oct 1, 2026

Uh oh!

NickCao commented Oct 1, 2026

Uh oh!

vllm-omni-review-bot left a comment

Choose a reason for hiding this comment

Omni ReviewBot review

PR description

Change flow

Findings

Uh oh!

vllm-omni-review-bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

vllm-omni-review-bot left a comment

Choose a reason for hiding this comment

Omni ReviewBot review

PR description

Change flow

Uh oh!

vllm-omni-review-bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

vllm-omni-review-bot commented Oct 2, 2026

Omni ReviewBot: finding feedback

Uh oh!

vllm-omni-review-bot left a comment

Choose a reason for hiding this comment

Omni ReviewBot review

PR description

Change flow

Uh oh!

vllm-omni-review-bot Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

vllm-omni-review-bot commented Oct 2, 2026

Omni ReviewBot: finding feedback

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vllm-omni-review-bot left a comment

Choose a reason for hiding this comment

Omni ReviewBot review

Changes since the previous review

PR description

Change flow

Findings

Uh oh!

NickCao commented Sep 30, 2026 •

edited

Loading

vllm-omni-review-bot commented Oct 2, 2026 •

edited

Loading