Repository navigation
Conversation
… and PersonaPlex Add a native /v1/live/sessions WebSocket endpoint implementing the OpenAI Live protocol on top of one resumable request per session. - entrypoints/openai/live: protocol, session handler, resumable request driver, audio format conversion and the per-model processor interface. - live_session_config in the pipeline and deploy configs; optional max_sessions (over the limit: session_limit_reached, close 1013). - On context_length_exceeded the session trims to half the context after the prefix and resubmits; vllm_omni:live_* metrics. - MiniCPM-o 4.5: Live processor following the official duplex demo (per-unit transcripts, capped previous text, short-unit codec hold). - PersonaPlex: Live processor and prompting helpers; Mimi pools sized by max_sessions. - Deploy overlays, docs and a LiveKit example. - L3 E2E tests for both models (conversation, context trim, concurrent sessions and the session limit) and a merge-pipeline job for MiniCPM-o. - L1 CPU tests for the handler, trimming, config, audio, protocol and the MiniCPM-o logits processor and stage hand-off; the route manifest guard lists /v1/live/sessions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: vraiti <vraiti@redhat.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md, docs/design/module/entrypoints.md. Module owners: @alex-jw-brooks @linyueqian @NickCao Routing: @alex-jw-brooks via module named in the PR description, CODEOWNERS; @linyueqian via module named in the PR description; @NickCao via module named in the PR description @vraiti, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
Thanks for putting this together. Given the size, I focused on the core runtime paths: the Live handler/driver/session/processor under entrypoints/openai/live/, the config plumbing (live_session.py, stage_config.py, omni_config.py), the MiniCPM-o Live encoder, the llm2tts / tts2code2wav_async_chunk Live hand-off, and the Talker preprocess changes. I only skimmed the PersonaPlex processor/stage0, the protocol/audio helpers, docs and tests.
A few things inline. One more small one: segment_finished is plumbed through messages.py / orchestrator.py / omni_base.py / outputs, but the only reader I can find is the driver's debug log; the handler detects segment end from the stage-0 finish_reason. Either use it there, or drop the shared-code change from this PR.
| processor.push_audio(model_samples, start_ms) | ||
| while (unit := processor.next_unit()) is not None: | ||
| self._pending.append(unit) | ||
| if len(self._pending) > max(BACKLOG_WARNING_UNITS, self._backlog_warned): |
There was a problem hiding this comment.
_pending has no upper bound, only a warning. The server doesn't pace input: session_ms advances by the decoded audio duration, and expiry is wall-clock, so a client that sends faster than real time (or a slow/stalled stage 0) can grow this deque without limit. Each unit holds its raw audio, plus mel features for MiniCPM-o. Could we cap it (e.g. N units or N seconds) and send an error and close, or drop, once it's exceeded?
| # Each session keeps a streaming audio-encoder cache of up to ~150 MB on the | ||
| # Stage 0 GPU that vLLM's memory profiling does not count. Set a bound when | ||
| # the Stage 0 headroom is tight. | ||
| max_sessions: null |
There was a problem hiding this comment.
With max_sessions: null, SessionSlots admits unlimited sessions, while the comment above says each one keeps ~150 MB of encoder cache on Stage 0 that profiling doesn't count. That makes the shipped overlay easy to OOM under load. Also, when this is null, streaming_session_capacity() falls back to 1 and duplex_max_sessions isn't propagated, so anything sized from it (e.g. the audio-encoder graph grid when duplex_audio_encoder_cuda_graph_batch_sizes_from_sessions is set) is sized for one session while the API server admits many. A finite default here (as in personaplex_live.yaml) seems safer.
| available, | ||
| info.get("request_id"), | ||
| ) | ||
| rows = rows[-available:] if available > 0 else rows[:0] |
There was a problem hiding this comment.
After preemption, the recompute re-runs earlier units' rows, which then keep the placeholder token embedding, and only a warning is logged. The request carries on with KV that was computed from placeholders instead of audio, so the model silently loses those units. Since the handler keeps the full unit history and can already re-render it (render_resume), raising here so the session resubmits seems better than continuing with corrupted context. Note that the handler currently resubmits only on context_length_exceeded, so that path would need extending too.
| f"prompt_len={info_dict.get('_omni_prompt_len')}" | ||
| ) | ||
| if native_duplex: | ||
| if live_budget is not None: |
There was a problem hiding this comment.
The Live TTS request spans the whole session and is re-prefilled for each unit, but only the native_duplex branch below drops _request_codec_device_states / _request_codec_device_inputs for the request. The batched GPU path is fine, because the new state dict has no _gpu_slot and CodecState.prepare reloads step/limit/min. But on the scalar sampler path (_codec_sample_setup), unit 2+ would resume the previous unit's device step counter and keep the first unit's cached min_tokens (0 at turn start), so the per-unit codec_min wouldn't apply. Should that cleanup be if native_duplex or live_budget is not None:?
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
3 participants
Adds a native OpenAI Live (
/v1/live/sessions) WebSocket endpoint and serves MiniCPM-o 4.5 and PersonaPlex on it as full-duplex models.Implements RFC #8085 for the native-VAD models. The RFC's external-VAD models, Qwen3-Omni and Nemotron VoiceChat, already have full OpenAI support through
/v1/realtime, so/v1/live/sessionsserves only models that decide for themselves when to listen and speak.Contributor intent (required human answers)
1. What outcome do you want to achieve?
Support OpenAI-compliant
/v1/live/sessionsfor serving native fullduplex models.2. What is the highest-leverage variable for that outcome?
An OpenAI-compliant
/v1/live/sessions, which is completely absent in upstream right now.3. How will this work move that variable?
By adding the endpoint using an architecture that leverages existing vLLM-Omni features to constrain blast radius to the API server and model pipeline code.
4. Whose customer or user needs does this address, and what are those needs?
Anyone who wishes to interface with a vLLM-Omni native VAD model via standard OpenAI tooling.
Purpose
Before: native full-duplex speech models are reachable only through vLLM-Omni's own
/v1/realtime?duplex=1stack. OpenAI Live clients, such as LiveKit Agents'GPTLiveModelor the OpenAI SDK'sopenai.types.liveevents, cannot connect to them.After: a deployment whose deploy config has a
live_session_configblock serves/v1/live/sessions, which speaks the OpenAI Live protocol. Each session drives one resumable engine request: audio arrives assession.input_audio.append, is cut into the model's native units and sent as streaming updates, and the model's audio (and MiniCPM-o's transcript) streams back assession.output_audio.delta/session.output_transcript.delta. The model decides when to listen and when to speak; there is no server-side VAD or turn detection.vllm_omni/entrypoints/openai/live/):openai.types.livemodels;LiveSessionProcessorinterface that each model implements.live_session_confignames its processor and an optional per-stage logits processor;live_session_configenables the endpoint, withsession_lifetime_sand an optionalmax_sessionscap. Over the cap,session.startgetssession_limit_reachedand the socket closes with 1013.context_length_exceededthe session drops its oldest units so that what remains fills half of the context left after the prefix, then resubmits. MiniCPM-o carries the dropped units' text forward as "previous" context. PersonaPlex restarts from the newest frame.vllm_omni:live_active_sessions,live_rejected_sessions,live_audio_seconds{direction},live_context_trimsandlive_errors{reason}.personaplex/prompting.py, shared with the existing duplex path, and its Mimi codec pools are sized bymax_sessions.vllm_omni/deploy/minicpmo_4_5_live.yamlandpersonaplex_live.yaml./v1/realtimeand duplex deployments behave as before. The Live logits processor and session sizing apply only when the deploy config enables Live sessions, and the AR model runner is unchanged.segment_finishedflag toOmniRequestOutput(engine/messages.py,engine/orchestrator.py,entrypoints/omni_base.py,outputs/__init__.py). The flag defaults toFalse, and there's no scheduling or engine change;config/stage_config.pyandconfig/omni_config.py;config/live_session.pyandmetrics/live.py;openai>=3.18.0inrequirements/common.txt, foropenai.types.live.Docs:
docs/serving/live_sessions_api.md. Example:examples/online_serving/live_sessions/livekit/agent.py, a LiveKit Agents voice agent on the endpoint.Not supported by native-VAD models, and therefore narrower than the RFC's event list:
session.instructions.append,response.item.createandresponse.create, since these models have no mid-stream text slot and their output is clocked by input audio rather than requested;store;The events, tools and
storeare rejected withunsupported_*errors, and each session start lists them in aninfoevent. A built-in voice name is rejected withinvalid_value. Input transcripts are simply never sent.Test Plan
vLLM Version: 0.32.0
vLLM-Omni Commit: abbd3d9 (on 1c11eef)
L1 (CPU):
pytest tests/entrypoints/openai/live tests/model_executor/models/minicpmo_4_5/test_live_logits_processor.py tests/model_executor/stage_input_processors/test_minicpmo_4_5_live.py -m "core_model and cpu". Covers the session handler against a fake engine (session limit, protocol errors, context overflow and resubmission, close), context trimming, config validation, audio conversion, protocol events against the openai SDK'sServerEventschema, and the MiniCPM-o logits processor and stage hand-off.L3 (1× GPU):
pytest tests/e2e/online_serving/test_minicpmo_4_5_live.py tests/e2e/online_serving/test_personaplex_live.py. Each model gets three raw-WebSocket tests:session_limit_reached/ 1013.CI runs the MiniCPM-o file in the merge pipeline ("Omni · MiniCPM-o 4.5 Live Test"). PersonaPlex has no CI step, because its checkpoint (
nvidia/personaplex-7b-v1) is gated.Manual: LiveKit Agents
GPTLiveModel(livekit-agents 1.8.3) through a local LiveKit server and theagent-starter-reactfrontend, talking to both models live.Test Result
tests/entrypointsCPU suite passes (2,716 tests), includingtest_api_server_guards.pywith/v1/live/sessionsadded to its route manifest.vllm_omni/engine/messages.pythat were already there before this change.