[NickCao]

Purpose

So it works with all models not only qwen3 omni.

Test Plan

vLLM Version: 0.30.0

vLLM-Omni Commit: b5b7f27

pytest -sv tests/entrypoints tests/engine -m 'core_model and cpu'
pytest -sv tests/entrypoints -m "advanced_model and cuda and H100" --run-level "advanced_model" --deselect tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::TestQwen3OmniRealtimeWebSocket::test_server_vad_multi_turn_without_client_commit

Test Result

3610 passed, 5 skipped, 134 deselected, 1 xfailed, 29 warnings in 186.64s (0:03:06)
15 passed, 2364 deselected, 22 warnings in 898.27s (0:14:58)

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/profiling.md, docs/design/module/entrypoints.md, docs/design/module/observability.md.

Module owners: @alex-jw-brooks @linyueqian @gcanlin

Routing: @alex-jw-brooks via module of the changed files, semantic router, CODEOWNERS; @linyueqian via module of the changed files, semantic router, CODEOWNERS; @gcanlin via module of the changed files

@NickCao, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Direct under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Realtime /v1/realtime turn-based auto truncation no longer estimates token budget with Qwen3-Omni-specific audio math. It builds the chat prompt, renders it through the engine renderer with max_total_tokens=None, and truncates conversation history until the rendered prompt_token_ids length fits the model budget. The same rendered EngineInput is then passed to engine.generate, so admission and execution share one token count. A new CPU unit test pins truncation driven by renderer expansion rather than raw text tokens.

Change flow

flowchart TD
  A["[EXISTING] response.create handler"]:::existing
  B["[CHANGED] cancel then _prepare_engine_input_with_auto_truncation"]:::changed
  C["[NEW] _render_engine_input via render_cmpl_async"]:::new
  D["[CHANGED] generate uses rendered EngineInput"]:::changed
  E["[NEW] test_realtime_prompt_preflight"]:::new
  A --> B
  B --> C
  C --> D
  E -.-> C
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 2ce95e6014a5 (2026-09-29T20:10:57.237082+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

See inline comments below.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] Cancels in-flight response before admission can fail — vllm_omni/entrypoints/openai/realtime/connection.py:577

_handle_response_create now calls _cancel_active_response() before _prepare_engine_input_with_auto_truncation. On the old path, truncation/admission ran first and only cancelled after a successful admit, so a rejected over-budget response.create left the prior generation running. On this head, an over-limit or VLLMValidationError path still aborts the active response (lines 577–590 return after cancel), so clients lose a valid in-progress turn solely because the replacement request cannot start. Restore cancel-after-successful-preflight (or re-admit / avoid cancel until engine_input is non-None), and add a regression that proves a failing create does not cancel an existing active response.

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] PR Test Plan is a subset of the covering CPU entrypoints guard — tests/entrypoints/openai_api/test_realtime_prompt_preflight.py:14

Changed production path vllm_omni/entrypoints/openai/realtime/connection.py and new test tests/entrypoints/openai_api/test_realtime_prompt_preflight.py (markers core_model + cpu) are covered by Buildkite ready step "Simple · Engine&Entrypoints Test" which runs pytest -sv tests/entrypoints tests/engine -m 'core_model and cpu' (.buildkite/cuda/test-ready.yml). The PR Test Result only reports pytest tests/entrypoints/openai_api/test_realtime_prompt_preflight.py (1 passed) and does not state that the rest of that guard selector was skipped. Either run the guard command (or an equivalent tests/entrypoints tests/engine -m 'core_model and cpu' sweep) or explicitly document the uncovered remainder in the Test Plan. Frozen CI note: required buildkite/vllm-omni was missing at checked_at 2026-09-29T20:10:57Z (not treated as a code defect).

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

[alex-jw-brooks]

I don't think this will work generically, because_build_full_prompt still hardcodes AUDIO_PLACEHOLDER for qwen3omni. The render engine input eventually applies vLLM's multimodal processor (i.e., wraps around this), so because of that, it wont be able to do the multimodal token expansion correctly

[alex-jw-brooks]

Have you benchmarked this? Since rendering is a lot more complex than the arithmetic from before and will redo a lot of the multimodal processing, I think this + rerunning rendering later on will actually cause a lot of scaling issues

[alex-jw-brooks]

Not sure if it belongs in this PR or separately since things are already hardcoded, but this should probably use get_placeholder_str, because it's implemented on SupportsMultimodal here. So we should be able to look up the unexpanded placeholder for any modality through that, e.g., for qwen3omni it's this

[alex-jw-brooks]

Similar to above, if we're calling rendering, I think it would be more ideal to try to reuse previous rendering. E.g., from the first render of everything, we should have the spans of expanded audio placeholders, and from there, be able to determine what needs to drop without iterative re-rendering with the caveat that things need to be correctly updated after dropping stuff unless you do one final render (since mm placeholders etc will be stale)

[NickCao]

Replaced by #8339