diff --git a/docs/design/fullduplex-personaplex.md b/docs/design/fullduplex-personaplex.md index 6d32cd8e3a0..c4e6547f67b 100644 --- a/docs/design/fullduplex-personaplex.md +++ b/docs/design/fullduplex-personaplex.md @@ -1,142 +1,137 @@ -# PersonaPlex Unified Full-Duplex Design +# PersonaPlex on the Unified Full-Duplex Framework ## Status and target -This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on -top of: - -- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47` -- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4` - -The target is the engine-native path: +PersonaPlex (`nvidia/personaplex-7b-v1`, a Moshi finetune) is served by the +[Unified Full-Duplex Framework](fullduplex.md) through one model plugin, +`PersonaPlexDuplexPlugin` (`vllm_omni/model_executor/models/personaplex/duplex/plugin.py`), +selected by `PipelineConfig.duplex_plugin`. The serving path is: ```text /v1/realtime?duplex=1 - -> OpenAI Realtime session actor - -> DuplexRequestClient - -> AsyncOmni correlated RPC - -> DuplexControlPlane - -> resumable Stage 0 request - -> PersonaPlex Talker - -> streaming PersonaPlex Code2Wav - -> PersonaPlex data-plane projector + -> OmniDuplexSessionHandler (thin: websocket I/O, Realtime <-> DuplexCommand/DuplexEvent) + -> DuplexOmni / DuplexOmniEngine + -> DuplexSessionManager -> DuplexSessionRunner (one per session) + plugin.plan_append: one 80 ms frame -> one resumable Stage 0 append + -> PersonaPlex Talker (Stage 0, lockstep temporal transformer + depformer) + -> streaming PersonaPlex Code2Wav (Stage 1, Mimi) + -> plugin.data_plane: cumulative audio/text -> deltas -> response.output_audio.delta + response.output_audio_transcript.delta ``` -The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the -original PR was demo-only and has been removed from the tree; the unified -engine path above is the only serving surface. It was never evidence that the -unified engine path works. - -## Why configuration-only enablement was invalid - -The staged pipeline as shipped by the original PR was explicitly turn based. -Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and -`pplex_silence_codes`, but no production staged input path wrote those fields. -The voice prompt, persona prefill, and streaming Mimi state lived only in the -standalone `PersonaPlexEngine` (demo-only, since removed). - -Setting only the following fields would therefore advertise an endpoint whose -model never receives the live microphone stream: - -```python -duplex_control_enabled = True -duplex_runtime_extension = "..." -duplex_serving_adapter = "..." -``` - -The adapter must supply a real scheduler data plane, not just endpoint -capabilities. +The pre-framework pair (engine runtime extension plus serving adapter) and +the standalone `/api/chat` / `/v1/audio/duplex` server are gone; the plugin is +the only integration surface, and this document describes it. ## Supported scope -The unified implementation supports: - -- up to two engine-owned sessions on one replica; -- 24 kHz mono float PCM input; -- one 1920-sample, 80 ms model frame per physical append unit; -- continuous user input while assistant audio is generated or played; -- bundled `.pt` voice prompts and a session persona; -- greedy text and depformer sampling, matching the current PersonaPlex port; -- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the - [Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide); +The integration supports: + +- up to `duplex_session.max_sessions` engine-owned sessions on one replica + (the shipped deploy sets two); +- 24 kHz mono float PCM input, one 1920-sample (80 ms) model frame per + physical append unit; clients may send any chunking, the session buffers + whole frames; +- continuous user input while assistant audio is generated or played + (pure lockstep: the model listens while it speaks); +- bundled `.pt` voice prompts (`voice`) and a session persona (`instructions`); +- greedy text and depformer sampling (one temporal token per frame); - the public client preset `vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz - `pcm_f32le` input format, voice prompt, persona) as the canonical - session-config source for `DuplexClient` consumers; -- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit - response cancellation without cross-session state reuse. - -The implementation does not claim: - -- more than two simultaneous PersonaPlex sessions on one replica; -- arbitrary WAV voice cloning; -- turn-based `response.create` semantics for an otherwise continuous model; -- scheduler migration of live codec state between replicas; -- exact output equality with the standalone engine after different scheduling - boundaries. - -The capability payload derives multi-session support from the configured -session limit. The shipped two-session deployment reports -`supports_multi_session=true` and `supports_multi_session_same_replica=true`. + `pcm_f32le` input, voice, persona) as the canonical session-config source + for `DuplexClient` / `InlineDuplexClient` consumers; +- engine lease close, disconnect cleanup and explicit response cancellation + without cross-session state reuse. + +It does not claim: + +- arbitrary WAV voice cloning (voices are bundled basenames, resolved by the + worker under the checkpoint); +- turn-based `response.create` semantics, client commits or external turn + signals (`supports_client_commit=false`, `supports_external_turn_signal=false`); +- text seeding, hence no `/v1/chat/completions` route + (`supports_chat_completions=false`); +- session resume across a transport drop (`supports_session_resume=false`); +- destructive output interruption or model-state rewind at a playback cursor + (`supports_barge_in=false`, `supports_audio_truncate=false`): overlapping + speech is native model behaviour, not a barge-in contract. + `duplex_session.max_sessions` is the only capacity source: config resolution -propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read -that model-config value. Connector extras do not carry a second model-specific -capacity knob that could drift from engine admission. -It still reports `supports_barge_in=false`: the generic epoch fence can suppress -stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5 -adapter proves that model-owned streaming state can be destructively rewound or -restarted at a playback cursor. Continuous overlapping speech is model-native -duplex behavior, not by itself a barge-in contract. +propagates it to every stage as `duplex_max_sessions`, and both Mimi pools +(Stage 0 encoders, Stage 1 decoders) read that model-config value. The +capability payload derives `supports_multi_session` / +`supports_multi_session_same_replica` from it. ## Components and ownership -### Serving adapter - -`PersonaPlexServingRuntimeAdapter` owns only serving-side state: - -- a transactional PCM append buffer; -- validation of `voice_prompt` and `instructions`; -- public capabilities; -- PersonaPlex data-plane output projection. - -It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample -frames, zero-pads only the final residual, and rolls a reservation back when an -engine append fails. It never loads CUDA weights and never encodes user audio. - -Client input cannot provide local filesystem paths. A voice is a bundled -basename such as `NATF2.pt`; the worker resolves it under the local model -checkpoint. `instructions` is the persona string. - -### Runtime extension - -`PersonaPlexDuplexRuntimeExtension` is pure model policy. It: - -- configures greedy Stage 0 sampling and bounded segment lengths; -- maps each accepted PCM append to a scheduler prompt; -- places immutable session identity, append sequence, PCM payload, voice, and - persona under `model_intermediate_buffer["duplex"]`; -- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the - first-append voice/persona prefill length; -- never performs model inference or owns session state. - -The extension returns no turn/listen decision. PersonaPlex is an always-clocked -model, so visible audio/text comes from the final stage data plane. - -### Stage 0 streaming runtime - -The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to -MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics. - -For each admitted session it owns: - -- streaming Mimi encoder convolution and transformer state; -- the selected voice embedding bundle; -- persona tokenization and prefill embeddings; -- the prior user code frame needed by the one-frame acoustic delay; -- append identity used to make a retried scheduler update idempotent. - -The first append builds this ordered prefill: +### The plugin (engine side) + +`PersonaPlexDuplexPlugin` owns both halves of the contract. + +Engine policy: + +- `configure_sampling_params`: Stage 0 greedy (`temperature=0`, `top_k=1`, + `max_tokens=1`), other stages untouched. +- `plan_append`: validates exactly one 1920-sample 24 kHz `pcm_f32le` frame + (through `model_executor/common/duplex/payload.py`) and reserves one + scheduler slot per frame plus, on the first append of an epoch (`seq == 1`), + the `personaplex_prefill_slots` of the voice/persona prefill. The prompt + carries the fence, `seq`, the payload and the runtime config under + `model_intermediate_buffer["duplex"]`. +- `decide_output`: never decides. PersonaPlex is always-clocked; what the + client hears comes from the final-stage data plane. +- `silence_unit_payload`: one frame of zeros at 24 kHz + (`silence_continuation_samples=1920`, + `silence_continuation_sample_rate_hz=24000`). The runner uses it to keep a + model turn clocked when the client pauses; the startup warmup sends it. + +Session policy: + +- `capabilities`: `personaplex_capabilities(max_sessions)` (80 ms units, + append-only, no commits, no barge-in, no resume, no chat route). +- `prepare_runtime_config`: `voice` must be a bundled `.pt` basename, + `instructions` defaults to the shipped persona; the prefill slot count is + computed once per `(model, voice, persona)` off the orchestrator loop + (`personaplex_prefill_slots`, itself caching the tokenizer and the voice + bundle row count). The result is server-owned runtime config + (`personaplex_model_path`, `personaplex_voice_prompt`, + `personaplex_persona`, `personaplex_prefill_slots`); the same keys are + refused in a client's `extra_body`. +- `runtime_config_for_update`: persona and voice are immutable for the + session (`persona_update_unsupported`, `voice_update_unsupported`). +- session state, extra-body validation and data-plane context are the + framework defaults (`DefaultDuplexModelSessionState` with the 80 ms + `PersonaPlexPcmAppendBuffer`, `DuplexDataPlaneContext`). + +Because `supports_client_commit` is off, the session auto-responds without +`extra_body.auto_response`: a stock Realtime client streams audio and hears +the model without any vendor flag. + +### Input framing + +`PersonaPlexPcmAppendBuffer` is the `FixedFramePcmAppendBuffer` of +`model_executor/common/duplex/pcm_buffer.py` at 24 kHz / 1920 samples / +80 ms. It accepts 24 kHz `pcm_f32le` only, groups client packets into whole +frames, takes one frame out per append as a reservation (committed when the +stage accepted the append, rolled back to the front of the buffer when it did +not), zero-pads only a final residual on commit, and never encodes audio. + +### Stage 0 streaming runtime (worker side) + +The Talker owns a `PersonaPlexStage0DuplexRuntime` +(`model_executor/models/personaplex/duplex/stage0.py`), created lazily in +`_duplex_stage0_runtime()` and released through `on_requests_finished`. + +For each live `(session_id, epoch)` it owns: + +- the streaming Mimi encoder convolution and transformer state; +- the selected voice embedding bundle and the persona prefill embeddings; +- the prior user code frames needed by the one- and two-frame acoustic + delays; +- the append identity `(epoch, seq)` that makes a retried scheduler update + idempotent. + +The first append of an epoch builds this ordered prefill: ```text voice embeddings @@ -146,45 +141,47 @@ voice embeddings -> first live user frame ``` -Later appends encode only new 1920-sample frames. The helper returns the +Later appends encode only new 1920-sample frames. The runtime returns the per-frame user codes and prompt embeddings through the request's -`model_intermediate_buffer`. The normal vLLM runner remains authoritative for -attention metadata, block tables, KV allocation, scheduling, and sampling. - -The Talker must distinguish resumable prompt prefill from decode positions. -Prompt rows consume the exact prepared embeddings; sampled decode rows continue -to use the existing delayed agent/user frame construction. No code path may -fall back to an all-initial user stream for a duplex request. - -Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live -session has an independent Mimi encoder instance; encoder convolution/KV state -is never shared between asynchronously scheduled sessions. A finished or +`model_intermediate_buffer`; the normal vLLM runner remains authoritative for +attention metadata, block tables, KV allocation, scheduling and sampling. + +**Epochs.** The framework identifies a Stage 0 request by +`(session_id, epoch)`; `response.cancel` and `output_audio_buffer.clear` +advance the epoch, abort the current request and start the next append at +`seq == 1` on a fresh request with fresh KV. The Stage 0 runtime therefore +keys its state by `(session_id, epoch)`: when an append of a newer epoch +arrives, any older-epoch state of the same session is closed first (its Mimi +encoder returns to the pool, so the codec budget never counts a superseded +epoch), and the voice/persona prefill is replayed because the plan reserved +the slots again. The user-visible consequence is that a cancel restarts the +model's conversation context. A late `on_requests_finished` for the aborted +request finds nothing to close. + +Every live session has an independent Mimi encoder instance; encoder state is +never shared between asynchronously scheduled sessions. A finished or aborted scheduler request resets and returns only that session's encoder. ### Stage 1 streaming decoder -The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not -CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent -streaming Mimi decoder for every active request. Decoder ownership is keyed by -the stable Stage 1 request id and released by `on_requests_finished`; a mixed -batch must never advance another request's convolution or transformer state. - -Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the -new code frames, emits only the new PCM suffix, and resets state when the -request is closed. Connector chunk boundaries retain the final raw code frame -needed to de-delay the next chunk. - -The deploy default sets Stage 1 `enforce_eager: true`; a default configuration -that fails during CUDA graph capture is not an acceptable deployment profile. +`PersonaPlexCode2Wav` runs eager (`enforce_eager: true` in the deploy) and +maintains an independent streaming Mimi decoder for every active request, +keyed by the Stage 1 request id and released by `on_requests_finished`. Each +Stage 0 segment emits de-delayed agent codebooks; Stage 1 decodes only the new +code frames and emits only the new PCM suffix. Connector chunk boundaries +retain the final raw code frame needed to de-delay the next chunk. ### Data-plane projector -`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into -model-neutral native results: +`PersonaPlexDataPlaneSession` is the `CumulativeAudioTextDataPlane` of +`model_executor/common/duplex/data_plane.py` with a 24 kHz default rate. It +keeps one audio/text cursor per request so cumulative Stage 1 output cannot +replay old audio, and yields one internal result per new suffix: ```python { "stage_role": "tts", + "is_listen": False, "data_plane_request_id": request_id, "text": text_delta, "audio_data": encoded_audio_delta, @@ -192,82 +189,74 @@ model-neutral native results: "sample_rate_hz": 24000, "audio_duration_ms": delta_duration, "end_of_turn": False, + "runtime_impl": "scheduler_data_plane", + ... } ``` -It owns per-request audio and text cursors so a cumulative output cannot replay -old audio. The generic projector turns each native result into the Realtime -`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and -`response.output_text.delta` for text) under one `response_id`; the full -mapping is the name map in the -[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide. -PersonaPlex keeps one visible response open while continuous output arrives. -Session close or cancellation terminates that response through the generic -Realtime lifecycle; a codec segment finishing is not a conversational turn -boundary. PersonaPlex advertises `supports_barge_in=false` and -`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`, -and `session.resume` are rejected on this model. +The runner turns each result into the Realtime +`response.output_audio.delta` / `response.output_audio_transcript.delta` pair +under one `response_id`. PersonaPlex keeps one visible response open while +continuous output arrives; session close or cancellation terminates it through +the generic Realtime lifecycle. A codec segment finishing is not a +conversational turn boundary. ## Error and lifecycle contracts -- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice - basename, or changed format fails before scheduler submission. +- Unsupported sample rate, malformed base64, non-finite PCM, a partial frame + at the plan, an invalid voice basename or a changed format fails before + scheduler submission, as a typed `error` event. - Append is prepare/submit/commit. Failure rolls back the exact reserved PCM bytes and does not advance the model frame cursor. -- A repeated `operation_id` must not encode or submit the same frame twice. -- Input iterator exceptions execute the same cleanup as explicit close. -- Cancellation does not release the engine lease until the stage request and - any in-flight codec operation are actually finished. -- A bounded cleanup timeout returns a cleanup error and keeps the session in - the closing admission set; it must not make the slot available while work - still mutates shared state. -- New sessions cannot observe the previous voice, persona, PCM tail, Mimi - convolution state, or Talker delayed code frame. - -(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path, -which followed the same exception-safe rule, have been removed from the tree.) +- A repeated `(epoch, seq)` identity does not encode or submit the same frame + twice. +- Cancellation advances the epoch; the aborted request's Stage 0 state is + released by the next-epoch append or by `on_requests_finished`, whichever + comes first. +- Close releases the admission slot only once stage cleanup succeeded; the + reaper retries a failed cleanup while the slot stays held. +- New sessions cannot observe a previous voice, persona, PCM tail, Mimi + convolution state or Talker delayed code frame. ## Testing and acceptance -### Contract tests - -Tests first cover: - -- pipeline registration enables the control plane and selects both PersonaPlex - adapters; -- the capability payload is two-session, 80 ms, append-only, and honest; -- PCM reservation commit/rollback, partial-frame flush, invalid input, and - operation idempotency; -- runtime prompt fields and exact token budgeting; -- first-append voice/persona prefill followed by live user codes; -- later appends retain per-session Mimi state and do not replay prefill; -- interleaved Stage 0 and Stage 1 work preserves independent codec histories; -- output projection emits only audio/text deltas; -- close, exception, timeout, and reconnect cleanup; -- ordinary non-duplex imports do not load PersonaPlex modules. - -### Remote H20 validation - -Validation runs in an isolated remote worktree using the ModelScope -`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency. - -The required evidence is: - -1. default `personaplex.yaml` reaches ready without a local eager override; -2. `/health` returns 200; -3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`, - and two-session admission; -4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and - text deltas; -5. microphone input continues during assistant output without cancelling the - scheduler request; -6. two paced sessions simultaneously produce independent non-empty audio, and - a third session is rejected with `resource_exhausted`; -7. closing either session frees only its scheduler request and GPU codec state, - after which a replacement session can use a different persona without state - leakage; -8. malformed input returns one typed error and does not poison the next append; -9. all focused unit tests and `git diff --check` pass. +### CPU contract tests + +- `tests/model_executor/models/personaplex/duplex/test_plugin.py`: pipeline + binding and plugin load, honest capabilities, private keys, voice/persona + resolution and caching, immutable updates, greedy sampling, one-slot-plus- + prefill planning, frame validation, the 24 kHz silence unit, and the + CPU-checked helpers of the E2E driver. +- `tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py`: + first-append prefill, depformer teacher forcing, causal user-frame delays, + idempotent retries, independent encoders per session, capacity, epoch + restart and late finish of a superseded request. +- `tests/engine/duplex/test_session_runner_personaplex.py`: the session + runner with the real plugin -- one submission per frame, prefill on the + first append only, half frames buffered, wrong rate refused, cumulative + Code2Wav output projected as 24 kHz deltas, cancel restarting the epoch, + close aborting the request. +- `tests/model_executor/executor_common/`: the shared toolbox (PCM helpers, payload + validation, fixed-frame buffer, cumulative data plane, request-output + readers). + +### GPU validation + +`tests/e2e/online_serving/personaplex_realtime_duplex.py` (wrapped by +`tests/e2e/online_serving/test_personaplex_duplex.py` when +`PERSONAPLEX_MODEL_PATH` is set) drives the default `personaplex.yaml` +deployment and requires: + +1. the server reaches ready without a local eager override; `/health` is 200; +2. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80` + and two-session admission, with server-allocated session ids; +3. paced 24 kHz PCM appends produce finite, non-silent, whole-frame 24 kHz + audio deltas and text deltas; +4. two paced sessions simultaneously produce independent non-empty audio, and + a third session is refused with `resource_exhausted`; +5. closing one session frees only its scheduler request and codec state, after + which a replacement session with a different persona is admitted with a + fresh id and no state leakage. Audio that is empty, all zero, non-finite, or only a protocol `listen` event is not a successful end-to-end result. diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index 1ac585526f5..96d33381999 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -107,6 +107,7 @@ th { | `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — | | `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — | | `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) | +| `PersonaPlexTalkerForConditionalGeneration` | PersonaPlex (full duplex, 24 kHz speech in/out over `/v1/realtime?duplex=1`) | `nvidia/personaplex-7b-v1` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/PersonaPlex.md) | | `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — | | `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — | | `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — | diff --git a/docs/serving/realtime_duplex_api.md b/docs/serving/realtime_duplex_api.md index f2c26d70e31..7454d15f4ba 100644 --- a/docs/serving/realtime_duplex_api.md +++ b/docs/serving/realtime_duplex_api.md @@ -384,9 +384,9 @@ with no OpenAI counterpart. The event vocabulary is uniform, but several surfaces are gated by the `capabilities` object the server returns in `session.created`; a client must -branch on those flags rather than on the model name. MiniCPM-o 4.5 is the -only model on the plugin contract today; the other two columns record what -their integrations advertise once the follow-up PRs port them: +branch on those flags rather than on the model name. MiniCPM-o 4.5 and +PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column +records what its integration advertises once the follow-up PR ports it: | Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface | | --- | --- | --- | --- | --- | @@ -401,7 +401,11 @@ their integrations advertise once the follow-up PRs port them: Everything else in the catalogue — session lifecycle, heartbeat and event acknowledgement, append/commit/clear, the response envelope, playback acknowledgement, and the error envelope — behaves identically for every -model. +model. Two PersonaPlex specifics follow from its capabilities rather than from +special-casing: a model with `supports_client_commit=false` auto-responds +without `extra_body.auto_response`, and `response.cancel` / +`output_audio_buffer.clear` restart its conversation context (a new Stage 0 +request replays the voice/persona prefill). ### Compatibility with the OpenAI Realtime protocol diff --git a/examples/online_serving/personaplex/README.md b/examples/online_serving/personaplex/README.md index f03b46908ce..54b45c357c3 100644 --- a/examples/online_serving/personaplex/README.md +++ b/examples/online_serving/personaplex/README.md @@ -2,15 +2,15 @@ Serve [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1) (a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine -through the unified duplex serving stack (`/v1/duplex` and `/v1/realtime?duplex=1`). +through the unified full-duplex framework (`/v1/realtime?duplex=1`, alias `/v1/duplex`). > Requires a GPU and Hugging Face access to the gated repo > (`HF_TOKEN` with access to `nvidia/personaplex-7b-v1`). ## Start the server -The default `vllm_omni/deploy/personaplex.yaml` enables the engine-owned -full-duplex control plane (`session_mode: duplex`): +The default `vllm_omni/deploy/personaplex.yaml` is a duplex deployment +(`session_mode: duplex`, two sessions per replica): ```bash HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \ @@ -19,18 +19,32 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser --deploy-config vllm_omni/deploy/personaplex.yaml ``` -This exposes: +This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI +Realtime session protocol projected onto vLLM-Omni duplex sessions (client API and +wire protocol: [`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)). +There is no `/v1/chat/completions` route: PersonaPlex answers speech only. -- `WS /v1/duplex` — the native duplex session dialect - (`session.create` / `input_audio_buffer.append` / `response.output_audio.delta` ...); -- `WS /v1/realtime?duplex=1` — the same sessions projected onto the - OpenAI Realtime event vocabulary (client API and wire protocol: - [`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)). +PersonaPlex is a pure-lockstep model: every session is native duplex, audio flows +continuously in both directions in 80 ms frames, the model decides when to speak, +and there are no client commits or external turn signals +(`supports_client_commit=false`, `supports_external_turn_signal=false`). A session +therefore auto-responds without any vendor flag. -PersonaPlex is a pure-lockstep model: every session on a PersonaPlex deployment is -native duplex (`is_enabled()` is unconditionally true), audio flows continuously in -both directions, and there are no client commits or external turn signals -(`supports_client_commit=false`, `supports_external_turn_signal=false`). +## Talk to it + +With the client library and the PersonaPlex preset (24 kHz `pcm_f32le` in, bundled +voice prompt, persona text): + +```python +from vllm_omni.clients.duplex import DuplexClient +from vllm_omni.clients.personaplex import create_duplex_session_config + +cfg = create_duplex_session_config(voice="NATF2.pt", persona="You are a concise assistant.") +async with DuplexClient("ws://127.0.0.1:8000/v1/realtime?duplex=1", model="/path/to/personaplex-7b-v1", config=cfg) as c: + await c.stream_pcm(pcm_f32le_24k) # keep streaming; the model speaks while it listens +``` + +Voice and persona are fixed for the session (`session.update` cannot change them). ## Validate the serving path @@ -45,9 +59,11 @@ python tests/e2e/online_serving/personaplex_realtime_duplex.py \ --output-dir /tmp/personaplex-realtime-duplex ``` -The unified endpoint advertises `supports_barge_in=false`: overlapping speech -is native model behavior, but destructive output interruption and model-state -rewind have not been validated for PersonaPlex. +The endpoint advertises `supports_barge_in=false`: overlapping speech is native +model behaviour, but destructive output interruption and model-state rewind have +not been validated for PersonaPlex. `response.cancel` and +`output_audio_buffer.clear` restart the model's conversation context (a fresh +Stage 0 request replays the voice/persona prefill). ## Notes @@ -56,8 +72,8 @@ rewind have not been validated for PersonaPlex. regardless of engine speed. On localhost it is smooth. - The earlier standalone Moshi-web compatibility server (browser client at `/`, binary WS protocol at `/api/chat`, raw-PCM `/v1/audio/duplex`) was demo-only - and has been removed; use the unified endpoints above. -- Session config (voice / persona / sampling) is passed per session via - `extra_body`; see - `vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py`. + and has been removed; use the unified endpoint above. +- The model plugin, worker-side lockstep runtime and input framing live in + `vllm_omni/model_executor/models/personaplex/duplex/`; design notes in + [`docs/design/fullduplex-personaplex.md`](../../../docs/design/fullduplex-personaplex.md). - Full runbook: [`recipes/NVIDIA/PersonaPlex.md`](../../../recipes/NVIDIA/PersonaPlex.md). diff --git a/recipes/NVIDIA/PersonaPlex.md b/recipes/NVIDIA/PersonaPlex.md index 6f9e8544fda..65c6453ff62 100644 --- a/recipes/NVIDIA/PersonaPlex.md +++ b/recipes/NVIDIA/PersonaPlex.md @@ -25,10 +25,10 @@ matching the reference implementation frame for frame on the golden replays used as the acceptance gate. PersonaPlex is the first Moshi-class (pure-lockstep) model on the vLLM-Omni -full-duplex serving stack: it plugs into the generic `/v1/duplex` handler -through the standard plugin seams (`duplex_serving_adapter` / -`duplex_runtime_extension` in its `pipeline.py`) with a model-specific -package at `vllm_omni/model_executor/models/personaplex/duplex/`. +unified full-duplex framework: its `pipeline.py` declares one +`duplex_plugin` (`PersonaPlexDuplexPlugin`), and the model-specific package at +`vllm_omni/model_executor/models/personaplex/duplex/` holds the plugin, the +worker-side lockstep Stage 0 runtime and the 80 ms input framing. ## References @@ -114,16 +114,18 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser --deploy-config vllm_omni/deploy/personaplex.yaml ``` -This exposes `WS /v1/duplex` (native duplex dialect) and -`WS /v1/realtime?duplex=1` (OpenAI Realtime projection; client API and wire -protocol in [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md)). -Voice and persona are set per session via `extra_body`. +This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI +Realtime session protocol, client API and wire vocabulary in +[`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md). +Voice (`voice`, a bundled `.pt` basename) and persona (`instructions`) are set +in `session.update`; the model takes no client commits and serves no +`/v1/chat/completions` route. #### Verification ```bash -# GPU-free contract tests (stage0 runtime + unified serving adapter) -pytest tests/model_executor/models/personaplex/duplex/ -q +# GPU-free contract tests (plugin, stage0 runtime, runner scenario) +pytest tests/model_executor/models/personaplex/duplex/ tests/engine/duplex/test_session_runner_personaplex.py -q # GPU e2e: paced 24 kHz PCM over /v1/realtime?duplex=1, two concurrent # sessions, overflow admission, slot recycling, non-silent output diff --git a/tests/e2e/online_serving/personaplex_realtime_duplex.py b/tests/e2e/online_serving/personaplex_realtime_duplex.py index 78f01254899..05fc2c30a5c 100644 --- a/tests/e2e/online_serving/personaplex_realtime_duplex.py +++ b/tests/e2e/online_serving/personaplex_realtime_duplex.py @@ -11,7 +11,6 @@ import json import math import time -import uuid import wave from collections.abc import Awaitable, Callable, Sequence from pathlib import Path @@ -125,10 +124,11 @@ def _input_identity( return {"path": str(resolved), "sha256": actual} -def _realtime_url(base_url: str, model: str, session_id: str) -> str: +def _realtime_url(base_url: str, model: str) -> str: + """The duplex Realtime URL; the session id is allocated by the server, never chosen here.""" parts = urlsplit(base_url) query = dict(parse_qsl(parts.query, keep_blank_values=True)) - query.update(duplex="1", model=model, autostart="0", session_id=session_id) + query.update(duplex="1", model=model, autostart="0") return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment)) @@ -197,16 +197,27 @@ def _audio_bytes(client: RawRealtimeProbe) -> bytes: return b"".join(chunk for _, chunk in _validated_audio_chunks(client) if chunk) +def _session_id(created: dict[str, object]) -> str | None: + """The server-allocated id announced in ``session.created``.""" + session = created.get("session") + if isinstance(session, dict): + for key in ("id", "session_id"): + value = session.get(key) + if isinstance(value, str) and value: + return value + value = created.get("session_id") + return value if isinstance(value, str) and value else None + + async def _open_session( args: argparse.Namespace, *, - session_id: str, persona: str, expect_error: bool = False, ) -> tuple[RawRealtimeProbe, dict[str, object]]: - client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id)) + client = RawRealtimeProbe(_realtime_url(args.url, args.model)) await client.__aenter__() - await client.send(_session_update(args, session_id=session_id, persona=persona)) + await client.send(_session_update(args, persona=persona)) event_type = "error" if expect_error else "session.created" await wait_for( lambda: client.events.count(event_type) > 0, @@ -216,11 +227,10 @@ async def _open_session( return client, _events(client, event_type)[-1] -def _session_update(args: argparse.Namespace, *, session_id: str, persona: str) -> dict[str, object]: +def _session_update(args: argparse.Namespace, *, persona: str) -> dict[str, object]: return { "type": "session.update", "session": { - "session_id": session_id, "model": args.model, "modalities": ["audio", "text"], "input_audio_format": "pcm_f32le", @@ -426,13 +436,11 @@ async def run(args: argparse.Namespace) -> dict[str, object]: output_dir = Path(args.output_dir) output_dir.mkdir(parents=True, exist_ok=True) - ids = {name: f"personaplex-{name}-{uuid.uuid4().hex}" for name in ("primary", "secondary", "replacement")} - primary, created = await _open_session(args, session_id=ids["primary"], persona=args.persona) - secondary, secondary_created = await _open_session( - args, - session_id=ids["secondary"], - persona=args.secondary_persona, - ) + primary, created = await _open_session(args, persona=args.persona) + secondary, secondary_created = await _open_session(args, persona=args.secondary_persona) + ids = {"primary": _session_id(created), "secondary": _session_id(secondary_created)} + if not ids["primary"] or not ids["secondary"] or ids["primary"] == ids["secondary"]: + raise AssertionError(f"server did not allocate distinct session ids: {ids}") capabilities = _capabilities(created) if _capabilities(secondary_created) != capabilities: raise AssertionError("concurrent sessions returned different capabilities") @@ -446,12 +454,7 @@ async def run(args: argparse.Namespace) -> dict[str, object]: if any(capabilities.get(key) != value for key, value in expected_capabilities.items()): raise AssertionError(f"unexpected PersonaPlex capabilities: {capabilities}") - overflow, error = await _open_session( - args, - session_id=f"personaplex-overflow-{uuid.uuid4().hex}", - persona=args.persona, - expect_error=True, - ) + overflow, error = await _open_session(args, persona=args.persona, expect_error=True) error_body = error.get("error") overflow_code = error_body.get("code") if isinstance(error_body, dict) else error.get("code") await overflow.__aexit__(None, None, None) @@ -492,11 +495,10 @@ async def run(args: argparse.Namespace) -> dict[str, object]: _save(output_dir, "primary", primary, primary_audio) await _close_session(primary, timeout_s=args.timeout_s) - replacement, replacement_created = await _open_session( - args, - session_id=ids["replacement"], - persona=args.replacement_persona, - ) + replacement, replacement_created = await _open_session(args, persona=args.replacement_persona) + ids["replacement"] = _session_id(replacement_created) + if not ids["replacement"] or ids["replacement"] in {ids["primary"], ids["secondary"]}: + raise AssertionError(f"replacement session id was not freshly allocated: {ids}") if _capabilities(replacement_created) != capabilities: raise AssertionError("replacement session returned different capabilities") continuation_frames, replacement_frames = await asyncio.gather( @@ -694,8 +696,9 @@ async def _run_load_session( ready: asyncio.Future[None], start: asyncio.Future[float], ) -> dict[str, object]: - session_id = f"personaplex-load-{index}-{uuid.uuid4().hex}" - client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id), close_timeout_s=args.cleanup_timeout_s) + # Label only: the server allocates the real session id (read back from session.created). + session_id = f"personaplex-load-{index}" + client = RawRealtimeProbe(_realtime_url(args.url, args.model), close_timeout_s=args.cleanup_timeout_s) sends: list[tuple[float, float, float]] = [] error: str | None = None cleanup_error: str | None = None @@ -706,10 +709,9 @@ async def _run_load_session( try: async with client: try: - await asyncio.wait_for( - client.send(_session_update(args, session_id=session_id, persona=args.persona)), args.timeout_s - ) + await asyncio.wait_for(client.send(_session_update(args, persona=args.persona)), args.timeout_s) await _wait_load_event(client, "session.created", args.timeout_s) + session_id = _session_id(_events(client, "session.created")[-1]) or session_id capabilities = _capabilities(_events(client, "session.created")[-1]) if ( capabilities.get("chunk_period_ms") != 80 diff --git a/tests/e2e/online_serving/test_personaplex_duplex.py b/tests/e2e/online_serving/test_personaplex_duplex.py new file mode 100644 index 00000000000..8423fd1ddf9 --- /dev/null +++ b/tests/e2e/online_serving/test_personaplex_duplex.py @@ -0,0 +1,84 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +"""GPU coverage for PersonaPlex on the unified full-duplex framework. + +Boots the PersonaPlex deploy (``vllm_omni/deploy/personaplex.yaml``) and runs +the strict Realtime driver (``personaplex_realtime_duplex.py``): two paced +24 kHz sessions, admission overflow, per-session slot recycling, audible +whole-frame output. The checkpoint is gated, so the test runs only when +``PERSONAPLEX_MODEL_PATH`` points at a local copy of ``nvidia/personaplex-7b-v1``. +""" + +from __future__ import annotations + +import asyncio +import os +from pathlib import Path + +import pytest + +from tests.e2e.online_serving import personaplex_realtime_duplex as driver +from tests.helpers.mark import hardware_test +from tests.helpers.runtime import OmniServerParams +from tests.helpers.stage_config import get_deploy_config_path + +pytestmark = pytest.mark.omni + +MODEL_PATH = os.environ.get("PERSONAPLEX_MODEL_PATH", "") +DEPLOY_CONFIG = get_deploy_config_path("personaplex.yaml") + +SERVER_PARAMS = [ + pytest.param( + OmniServerParams( + model=MODEL_PATH or "nvidia/personaplex-7b-v1", + stage_config_path=DEPLOY_CONFIG, + use_stage_cli=False, + server_args=["--trust-remote-code"], + ), + id="two-stage-single-gpu", + ) +] + +requires_checkpoint = pytest.mark.skipif( + not MODEL_PATH or not Path(MODEL_PATH).is_dir(), + reason="set PERSONAPLEX_MODEL_PATH to a local nvidia/personaplex-7b-v1 checkout", +) + + +def _speech_wav() -> Path: + """Real speech (the MiniCPM-o asset, 16 kHz; the driver resamples to 24 kHz). + + PersonaPlex answers speech, not tones: a synthetic signal yields a silent + reply and the driver's audibility floor rightly fails it. + """ + return Path(__file__).resolve().parents[2] / "assets" / "minicpmo_4_5" / "response_required_16k.wav" + + +@requires_checkpoint +@pytest.mark.advanced_model +@hardware_test(res={"cuda": "H100"}, num_cards=1) +@pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True) +def test_personaplex_realtime_duplex_sessions(omni_server, tmp_path: Path) -> None: + input_wav = _speech_wav() + args = driver.parse_args( + [ + "--url", + f"ws://{omni_server.host}:{omni_server.port}/v1/realtime?duplex=1", + "--model", + MODEL_PATH, + "--input-wav", + str(input_wav), + "--output-dir", + str(tmp_path / "out"), + ] + ) + + result = asyncio.run(driver.run(args)) + + sessions = {name: result[name] for name in ("primary", "secondary", "replacement")} + assert all(isinstance(session, dict) for session in sessions.values()) + ids = {name: session["session_id"] for name, session in sessions.items() if isinstance(session, dict)} + assert ids["primary"] + assert ids["secondary"] != ids["primary"] + assert ids["replacement"] not in {ids["primary"], ids["secondary"]} diff --git a/tests/engine/duplex/test_duplex_plugin.py b/tests/engine/duplex/test_duplex_plugin.py index f45b625e792..7ac2dfa5dc6 100644 --- a/tests/engine/duplex/test_duplex_plugin.py +++ b/tests/engine/duplex/test_duplex_plugin.py @@ -5,6 +5,7 @@ from __future__ import annotations +import base64 from dataclasses import fields from pathlib import Path from typing import Any @@ -16,10 +17,13 @@ from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig from vllm_omni.engine.duplex.contracts import DuplexAppendPlan from vllm_omni.engine.duplex.plugin import ( + DefaultDuplexModelSessionState, DuplexDataPlane, + DuplexDataPlaneContext, DuplexModelPlugin, DuplexModelSessionState, DuplexRuntimeConfigError, + PcmAppendBuffer, coerce_int, load_duplex_plugin, payload_turn_id, @@ -322,3 +326,119 @@ def test_pipeline_config_binds_one_duplex_plugin_path() -> None: for source in framework_sources if "completed_append_cache_size" in source.read_text(encoding="utf-8") ] == [] + + +# --------------------------------------------------------------------------- # +# Plugin defaults shared by every model # +# --------------------------------------------------------------------------- # + + +def test_default_silence_unit_is_16k_zeros_and_follows_the_plugin_attributes() -> None: + plugin = FakePlugin(_encode_audio) + + unit = plugin.silence_unit_payload() + + assert unit["type"] == "audio" + assert unit["format"] == "pcm_f32le" + assert unit["sample_rate_hz"] == 16000 + assert base64.b64decode(unit["audio"]) == bytes(16000 * 4) + + class FramePlugin(FakePlugin): + silence_continuation_samples = 1920 + silence_continuation_sample_rate_hz = 24000 + + frame_unit = FramePlugin(_encode_audio).silence_unit_payload() + assert frame_unit["sample_rate_hz"] == 24000 + assert len(base64.b64decode(frame_unit["audio"])) == 1920 * 4 + + +def test_default_extra_body_validation_rejects_the_private_keys_naming_the_plugin() -> None: + class PrivateKeysPlugin(FakePlugin): + plugin_id = "keyed" + private_runtime_config_keys = frozenset({"secret_a", "secret_b"}) + + def validate_client_extra_body(self, extra_body: object) -> None: + DuplexModelPlugin.validate_client_extra_body(self, extra_body) + + plugin = PrivateKeysPlugin(_encode_audio) + plugin.validate_client_extra_body(None) + plugin.validate_client_extra_body({"auto_response": True}) + with pytest.raises( + DuplexRuntimeConfigError, match="keyed runtime configuration is server-owned: secret_a, secret_b" + ): + plugin.validate_client_extra_body({"secret_b": 1, "secret_a": 2}) + + +def test_default_data_plane_context_is_the_framework_dataclass() -> None: + class DefaultContextPlugin(FakePlugin): + def data_plane_context(self, **kwargs): + return DuplexModelPlugin.data_plane_context(self, **kwargs) + + context = DefaultContextPlugin(_encode_audio).data_plane_context( + epoch=2, + turn_id=3, + active_response_turn_id=1, + active_response_id="resp", + auto_responds=True, + response_format="pcm16", + speed=1.5, + modalities=("audio",), + ) + + assert isinstance(context, DuplexDataPlaneContext) + assert context == DuplexDataPlaneContext( + epoch=2, + turn_id=3, + active_response_turn_id=1, + active_response_id="resp", + auto_responds=True, + response_format="pcm16", + speed=1.5, + modalities=("audio",), + ) + + +def test_default_session_state_implements_the_shared_transitions() -> None: + class Buffer(PcmAppendBuffer): + pending_byte_count = 0 + + def clear(self) -> None: ... + + def clear_force_listen(self) -> None: ... + + def has_pending(self) -> bool: + return False + + def has_reserved(self) -> bool: + return False + + def prepare_append(self, payload, *, operation_id, chunk_period_ms, allow_emit): + return None + + def prepare_commit(self, *, operation_id, chunk_period_ms): + raise NotImplementedError + + def flush(self, *, chunk_period_ms): + return None + + state = DefaultDuplexModelSessionState(audio_buffer=Buffer()) + + assert isinstance(state, DuplexModelSessionState) + assert state.committed_audio_reserved_bytes == 0 and state.continuation_units == 0 + state.retain_committed_audio({"audio": "a"}, operation_id="op-1", reserved_bytes=10) + state.retain_committed_audio({"audio": "b"}, operation_id="op-2", reserved_bytes=5) + state.deferred_response_create = True + assert state.committed_audio_payload == {"audio": "b"} + assert state.committed_audio_operation_id == "op-2" + assert state.clear_committed_audio() == 15 + assert state.committed_audio_payload is None + assert state.committed_audio_operation_id is None + assert state.deferred_response_create is False + + state.continuation_owner_id = "response:r" + state.continuation_units = 3 + state.pending_silence_owner_id = "response:r" + state.clear_continuation() + assert state.continuation_owner_id is None + assert state.continuation_units == 0 + assert state.pending_silence_task is None and state.pending_silence_owner_id is None diff --git a/tests/engine/duplex/test_session_runner_personaplex.py b/tests/engine/duplex/test_session_runner_personaplex.py new file mode 100644 index 00000000000..6d835803c02 --- /dev/null +++ b/tests/engine/duplex/test_session_runner_personaplex.py @@ -0,0 +1,245 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +"""The session runner driven by the real PersonaPlex plugin. + +Same harness as ``test_session_runner.py`` (recording stage port, typed +commands through the manager, stage outputs pushed by hand), plugin swapped: +this pins the lockstep contract -- one Stage 0 submission per 80 ms frame, +prefill slots only on the first append of an epoch, auto-response implied by +``supports_client_commit=False``, cumulative Code2Wav output projected as +24 kHz deltas, and a cancel restarting the epoch. +""" + +from __future__ import annotations + +import asyncio +import base64 +import struct +from types import SimpleNamespace +from typing import Any + +import numpy as np +import pytest + +from tests.engine.duplex.test_session_runner import ( + SESSION_ID, + Harness, + RecordingStagePort, + _fake_encode_audio, + close_harness, + find, + types, +) +from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig +from vllm_omni.engine.duplex import commands +from vllm_omni.engine.duplex.config import DuplexSessionConfig +from vllm_omni.engine.duplex.messages import DuplexControlResultMessage, OpenDuplexSessionMessage +from vllm_omni.engine.duplex.session.manager import DuplexSessionManager +from vllm_omni.model_executor.models.personaplex.duplex import stage0 +from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE +from vllm_omni.model_executor.models.personaplex.duplex.plugin import PersonaPlexDuplexPlugin + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + +PREFILL_SLOTS = 4 + + +@pytest.fixture(autouse=True) +def _fake_prefill(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(stage0, "personaplex_prefill_slots", lambda model_path, voice, persona: PREFILL_SLOTS) + + +async def open_personaplex_harness(*, extra_body: dict[str, object] | None = None) -> Harness: + plugin = PersonaPlexDuplexPlugin(_fake_encode_audio) + port = RecordingStagePort(stage_count=2) + output: asyncio.Queue[Any] = asyncio.Queue() + results: asyncio.Queue[Any] = asyncio.Queue() + manager = DuplexSessionManager( + plugin=plugin, + stage_port=port, + output_sink=output, + result_sink=results, + runtime_config=DuplexSessionRuntimeConfig(), + model_config=SimpleNamespace(model="/models/personaplex-7b-v1"), + ) + config = DuplexSessionConfig( + model="nvidia/personaplex-7b-v1", + modalities=["audio", "text"], + instructions="Be concise.", + voice="NATF2.pt", + extra_body=dict(extra_body or {}), + ) + await manager.handle(OpenDuplexSessionMessage(control_id="c-open", session_id=SESSION_ID, session_config=config)) + result = await asyncio.wait_for(results.get(), timeout=2.0) + assert isinstance(result, DuplexControlResultMessage) and result.ok, result + harness = Harness(manager=manager, port=port, output=output, results=results, runner=manager.runners[SESSION_ID]) + await harness.settle() + return harness + + +def frame(samples: int = FRAME_SIZE, *, value: float = 0.05) -> commands.AppendAudio: + return commands.AppendAudio( + audio=struct.pack(f"<{samples}f", *([value] * samples)), + format="pcm_f32le", + sample_rate_hz=SAMPLE_RATE, + is_speech=True, + ) + + +def code2wav_output(request_id: str, *, samples: int, text: str) -> SimpleNamespace: + """A cumulative Stage 1 (Code2Wav) output the way the orchestrator hands it to the runner.""" + return SimpleNamespace( + request_id=request_id, + finished=False, + outputs=[SimpleNamespace(text=text, token_ids=[], multimodal_output={})], + multimodal_output={"audio": np.zeros(samples, dtype=np.float32), "sr": SAMPLE_RATE}, + ) + + +def submitted_duplex(port: RecordingStagePort) -> list[dict[str, object]]: + return [dict(sub.prompt["model_intermediate_buffer"]["duplex"]) for sub in port.submissions] + + +@pytest.mark.asyncio +async def test_open_advertises_the_lockstep_capabilities_and_reserves_stage0() -> None: + h = await open_personaplex_harness() + try: + created = find(h.events, "session.created") + capabilities = created.session["capabilities"] + assert capabilities["chunk_period_ms"] == 80 + assert capabilities["supports_client_commit"] is False + assert capabilities["supports_barge_in"] is False + # Not on the wire: the api server reads it from the plugin to decide the chat route. + assert h.session.capabilities.supports_chat_completions is False + assert [context.request_id for context in h.port.ensured] == [h.stage0_request_id(epoch=0)] + assert h.session.runtime_config["personaplex_prefill_slots"] == PREFILL_SLOTS + assert h.session.runtime_config["personaplex_voice_prompt"] == "NATF2.pt" + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_a_session_without_the_auto_response_flag_still_auto_responds() -> None: + h = await open_personaplex_harness(extra_body={}) + try: + assert h.runner.out.auto_responds() is True + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_each_frame_is_one_stage0_submission_with_prefill_only_on_the_first() -> None: + h = await open_personaplex_harness() + try: + await h.run(frame()) + await h.run(frame()) + await h.run(frame()) + + assert len(h.port.submissions) == 3 + assert [len(sub.prompt["prompt_token_ids"]) for sub in h.port.submissions] == [1 + PREFILL_SLOTS, 1, 1] + duplex = submitted_duplex(h.port) + assert [item["seq"] for item in duplex] == [1, 2, 3] + assert all(item["epoch"] == 0 for item in duplex) + assert all(item["session_id"] == SESSION_ID for item in duplex) + assert all("incarnation" not in item for item in duplex) + assert all(sub.context.request_id == h.stage0_request_id(epoch=0) for sub in h.port.submissions) + assert [sub.already_submitted for sub in h.port.submissions] == [False, True, True] + assert h.port.submissions[0].context.stage_sampling_params.max_tokens == 1 + assert not [event for event in h.events if event.type == "error"] + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_half_frames_are_buffered_until_a_whole_frame_exists() -> None: + h = await open_personaplex_harness() + try: + await h.run(frame(FRAME_SIZE // 2)) + assert h.port.submissions == [] + assert h.runner.model_state.audio_buffer.pending_byte_count == FRAME_SIZE * 2 + + await h.run(frame(FRAME_SIZE // 2)) + assert len(h.port.submissions) == 1 + assert h.runner.model_state.audio_buffer.pending_byte_count == 0 + payload = submitted_duplex(h.port)[0]["payload"] + assert isinstance(payload, dict) + assert len(base64.b64decode(payload["audio"])) == FRAME_SIZE * 4 + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_a_wrong_rate_append_is_refused_without_a_submission() -> None: + h = await open_personaplex_harness() + try: + events = await h.run( + commands.AppendAudio(audio=b"\x00" * 64, format="pcm_f32le", sample_rate_hz=16000, is_speech=None) + ) + + error = find(events, "error") + assert error.code == "bad_event" + assert "24000" in error.message + assert h.port.submissions == [] + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_cumulative_code2wav_output_becomes_24k_audio_and_text_deltas() -> None: + h = await open_personaplex_harness() + try: + await h.run(frame()) + request_id = h.stage0_request_id(epoch=0) + + first = await h.deliver_and_settle(code2wav_output(request_id, samples=FRAME_SIZE, text="he"), stage_id=1) + second = await h.deliver_and_settle( + code2wav_output(request_id, samples=2 * FRAME_SIZE, text="hello"), stage_id=1 + ) + + assert types(first)[0] == "response.created" + delta = find(first, "response.output_audio.delta") + assert delta.delta == f"wav-{FRAME_SIZE}" + assert delta.sample_rate_hz == SAMPLE_RATE + assert find(first, "response.output_audio_transcript.delta").delta == "he" + assert h.session.active_response_id is not None + + [delta_2] = [event for event in second if event.type == "response.output_audio.delta"] + assert delta_2.delta == f"wav-{FRAME_SIZE}" + assert find(second, "response.output_audio_transcript.delta").delta == "llo" + assert not [event for event in second if event.type == "response.created"] + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_cancel_restarts_the_epoch_and_the_next_frame_replays_the_prefill() -> None: + h = await open_personaplex_harness() + try: + await h.run(frame()) + old_request_id = h.stage0_request_id(epoch=0) + await h.deliver_and_settle(code2wav_output(old_request_id, samples=FRAME_SIZE, text="he"), stage_id=1) + + cancelled = await h.run(commands.CancelResponse()) + await h.run(frame()) + + assert h.session.epoch == 1 + assert "audio.cancelled" in types(cancelled) or "response.done" in types(cancelled) + assert ([old_request_id], True) in h.port.cleanups + assert h.port.submissions[-1].context.request_id == h.stage0_request_id(epoch=1) + assert submitted_duplex(h.port)[-1]["seq"] == 1 + assert len(h.port.submissions[-1].prompt["prompt_token_ids"]) == 1 + PREFILL_SLOTS + finally: + await close_harness(h) + + +@pytest.mark.asyncio +async def test_close_aborts_the_stage0_request() -> None: + h = await open_personaplex_harness() + await h.run(frame()) + request_id = h.stage0_request_id(epoch=0) + + await h.run(commands.CloseSession(reason="client_close")) + + assert h.port.cleanups == [([request_id], True)] + await close_harness(h) diff --git a/tests/engine/test_duplex_import_boundary.py b/tests/engine/test_duplex_import_boundary.py index 1da5131827f..7b44fc61cf4 100644 --- a/tests/engine/test_duplex_import_boundary.py +++ b/tests/engine/test_duplex_import_boundary.py @@ -65,6 +65,7 @@ def test_turn_based_stack_has_no_duplex_vocabulary() -> None: "vllm_omni.entrypoints.duplex", "vllm_omni.entrypoints.duplex_omni", "vllm_omni.clients", + "vllm_omni.model_executor.common.duplex", "vllm_omni.model_executor.models.minicpmo_4_5.duplex", "vllm_omni.model_executor.models.nemotron_voicechat.duplex", "vllm_omni.model_executor.models.personaplex.duplex", @@ -104,6 +105,7 @@ def test_duplex_stack_loads_the_framework_but_not_the_model_plugins() -> None: forbidden_prefixes = ( "vllm_omni.clients", + "vllm_omni.model_executor.common.duplex", "vllm_omni.model_executor.models.minicpmo_4_5.duplex", "vllm_omni.model_executor.models.nemotron_voicechat.duplex", "vllm_omni.model_executor.models.personaplex.duplex", diff --git a/tests/entrypoints/duplex/test_duplex_warmup.py b/tests/entrypoints/duplex/test_duplex_warmup.py new file mode 100644 index 00000000000..dc931da0c89 --- /dev/null +++ b/tests/entrypoints/duplex/test_duplex_warmup.py @@ -0,0 +1,53 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +"""The startup warmup sends the model's own silence unit, not a fixed 16 kHz one.""" + +from __future__ import annotations + +import base64 + +import pytest + +from vllm_omni.entrypoints.duplex.warmup import warmup_silence_unit + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +class _Plugin: + silence_continuation_samples = 1920 + + def silence_unit_payload(self) -> dict[str, object]: + return { + "type": "audio", + "audio": base64.b64encode(bytes(1920 * 4)).decode("ascii"), + "format": "pcm_f32le", + "sample_rate_hz": 24000, + } + + +class _LegacyPlugin: + silence_continuation_samples = 8000 + + +def test_the_plugin_unit_is_used_verbatim() -> None: + unit = warmup_silence_unit(_Plugin()) + + assert unit["format"] == "pcm_f32le" + assert unit["sample_rate_hz"] == 24000 + assert len(base64.b64decode(str(unit["audio"]))) == 1920 * 4 + + +def test_a_plugin_without_the_hook_falls_back_to_16k_zeros_of_its_sample_count() -> None: + unit = warmup_silence_unit(_LegacyPlugin()) + + assert unit["sample_rate_hz"] == 16000 + assert unit["format"] == "pcm_f32le" + assert len(base64.b64decode(str(unit["audio"]))) == 8000 * 4 + + +def test_no_plugin_means_the_minicpm_sized_default() -> None: + unit = warmup_silence_unit(None) + + assert unit["sample_rate_hz"] == 16000 + assert len(base64.b64decode(str(unit["audio"]))) == 16000 * 4 diff --git a/tests/model_executor/executor_common/__init__.py b/tests/model_executor/executor_common/__init__.py new file mode 100644 index 00000000000..4244ce4b4e1 --- /dev/null +++ b/tests/model_executor/executor_common/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project diff --git a/tests/model_executor/executor_common/audio/__init__.py b/tests/model_executor/executor_common/audio/__init__.py new file mode 100644 index 00000000000..4244ce4b4e1 --- /dev/null +++ b/tests/model_executor/executor_common/audio/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project diff --git a/tests/model_executor/executor_common/audio/test_pcm.py b/tests/model_executor/executor_common/audio/test_pcm.py new file mode 100644 index 00000000000..98d27b5fee6 --- /dev/null +++ b/tests/model_executor/executor_common/audio/test_pcm.py @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +import base64 + +import numpy as np +import pytest + +from vllm_omni.model_executor.common.audio.pcm import ( + PCM_F32LE_BYTES_PER_SAMPLE, + decode_pcm_f32le_base64, + pcm_f32le_sample_count, + pcm_f32le_samples, +) + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +def _b64(samples: np.ndarray) -> str: + return base64.b64encode(np.ascontiguousarray(samples, dtype=" None: + samples = np.arange(4, dtype=np.float32) + + raw = decode_pcm_f32le_base64(_b64(samples)) + + assert raw == samples.tobytes() + assert pcm_f32le_sample_count(raw) == 4 + assert PCM_F32LE_BYTES_PER_SAMPLE == 4 + + +def test_decode_accepts_an_empty_chunk() -> None: + assert decode_pcm_f32le_base64("") == b"" + assert pcm_f32le_sample_count(b"") == 0 + + +@pytest.mark.parametrize( + ("encoded", "match"), + [ + (None, "must be base64 pcm_f32le"), + (b"bytes", "must be base64 pcm_f32le"), + ("not*base64", "not valid base64"), + (base64.b64encode(b"\x00\x00\x00").decode("ascii"), "divisible by four"), + (_b64(np.array([0.0, np.nan], dtype=np.float32)), "must be finite"), + (_b64(np.array([np.inf], dtype=np.float32)), "must be finite"), + ], +) +def test_decode_rejects_malformed_audio_naming_the_model(encoded: object, match: str) -> None: + with pytest.raises(ValueError, match=match) as excinfo: + decode_pcm_f32le_base64(encoded, model="TestModel") + + assert str(excinfo.value).startswith("TestModel ") + + +def test_samples_are_a_writable_contiguous_float32_copy() -> None: + raw = np.arange(3, dtype=" SimpleNamespace: + return SimpleNamespace( + request_id=request_id, + outputs=[SimpleNamespace(text=text, multimodal_output={})], + multimodal_output={"model_outputs": np.arange(samples, dtype=np.float32), "sr": sr}, + finished=False, + ) + + +class _TestDataPlane(CumulativeAudioTextDataPlane): + stage_role = "codec" + default_sample_rate_hz = 16000 + + +def test_is_a_duplex_data_plane_and_projects_cumulative_output_as_deltas() -> None: + plane = CumulativeAudioTextDataPlane(_encode_audio) + plane.begin_request("req") + context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text")) + + projected = list( + plane.project( + {"data_plane_outputs": [_output("req", text="he", samples=4), _output("req", text="hello", samples=6)]}, + context=context, + ) + ) + + assert isinstance(plane, DuplexDataPlane) + assert [item["text"] for item in projected] == ["he", "llo"] + assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"] + assert [item["audio_duration_ms"] for item in projected] == [0, 0] + assert all(item["sample_rate_hz"] == 24000 for item in projected) + assert all(item["data_plane_request_id"] == "req" for item in projected) + assert all(item["is_listen"] is False and item["end_of_turn"] is False for item in projected) + assert projected[0]["stage_role"] == "tts" + assert projected[0]["runtime_impl"] == "scheduler_data_plane" + assert projected[0]["audio_format"] == "wav" + + +def test_subclass_constants_and_default_sample_rate_reach_the_projection() -> None: + plane = _TestDataPlane(_encode_audio) + output = SimpleNamespace( + request_id="req", + outputs=[SimpleNamespace(text="", multimodal_output={})], + multimodal_output={"audio": np.zeros(160, dtype=np.float32)}, + finished=False, + ) + + [item] = list(plane.project({"data_plane_outputs": [output]})) + + assert item["stage_role"] == "codec" + assert item["sample_rate_hz"] == 16000 + assert item["audio_duration_ms"] == 10 + + +def test_nothing_new_yields_nothing_and_non_results_are_ignored() -> None: + plane = CumulativeAudioTextDataPlane(_encode_audio) + same = _output("req", text="hi", samples=4) + + assert len(list(plane.project({"data_plane_outputs": [same, same]}))) == 1 + assert list(plane.project("not a dict")) == [] + assert list(plane.project({"data_plane_outputs": None})) == [] + + +def test_terminal_and_close_bookkeeping() -> None: + plane = CumulativeAudioTextDataPlane(_encode_audio) + session_request = duplex_resource_request_id(DuplexFence("sid", epoch=0), "stage0") + plane.begin_request(session_request) + plane.begin_request("other-req") + + assert plane.is_terminal(None) is False + assert plane.is_terminal(session_request) is False + plane.mark_terminal(session_request) + assert plane.is_terminal(session_request) is True + plane.begin_request(session_request) + assert plane.is_terminal(session_request) is False + + plane.close_session("sid", active_request_id="other-req") + assert plane._requests == {} + + plane.mark_terminal("late") + plane.close_stream("late") + assert plane.is_terminal("late") is False diff --git a/tests/model_executor/executor_common/duplex/test_payload.py b/tests/model_executor/executor_common/duplex/test_payload.py new file mode 100644 index 00000000000..c04683d13c2 --- /dev/null +++ b/tests/model_executor/executor_common/duplex/test_payload.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +import base64 + +import numpy as np +import pytest + +from vllm_omni.model_executor.common.duplex.payload import ( + decode_pcm_f32le_payload, + payload_audio, + payload_sample_count, +) + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +def _payload(samples: int, *, sample_rate_hz: int = 24000, fmt: str = "pcm_f32le", key: str = "audio") -> dict: + audio = base64.b64encode(np.zeros(samples, dtype=" None: + raw = decode_pcm_f32le_payload(_payload(1920), sample_rate_hz=24000, exact_samples=1920) + + assert len(raw) == 1920 * 4 + + +def test_decode_accepts_the_older_data_key() -> None: + assert len(decode_pcm_f32le_payload(_payload(8, key="data"), sample_rate_hz=24000)) == 32 + assert payload_audio({"data": "x"}) == "x" + assert payload_audio("not a mapping") is None + + +@pytest.mark.parametrize( + ("payload", "match"), + [ + ("frame", "must be a mapping"), + (_payload(1920, fmt="pcm16"), "format must be pcm_f32le"), + (_payload(1920, sample_rate_hz=16000), "sample_rate_hz must be 24000"), + (_payload(1919), "exactly 1920 samples"), + ({"format": "pcm_f32le", "sample_rate_hz": 24000, "audio": 12}, "must be base64"), + ], +) +def test_decode_rejects_payloads_that_do_not_match_the_model_unit(payload: object, match: str) -> None: + with pytest.raises(ValueError, match=match) as excinfo: + decode_pcm_f32le_payload(payload, sample_rate_hz=24000, exact_samples=1920, model="TestModel") + + assert str(excinfo.value).startswith("TestModel ") + + +def test_decode_without_exact_samples_accepts_any_whole_number_of_samples() -> None: + assert len(decode_pcm_f32le_payload(_payload(7), sample_rate_hz=24000)) == 28 + + +def test_payload_sample_count_is_lenient() -> None: + assert payload_sample_count(_payload(16000, sample_rate_hz=16000)) == 16000 + assert payload_sample_count(_payload(16000, fmt="pcm16")) is None + assert payload_sample_count({"format": "pcm_f32le", "audio": "not*base64"}) is None + assert payload_sample_count(None) is None diff --git a/tests/model_executor/executor_common/duplex/test_pcm_buffer.py b/tests/model_executor/executor_common/duplex/test_pcm_buffer.py new file mode 100644 index 00000000000..f4312b586c5 --- /dev/null +++ b/tests/model_executor/executor_common/duplex/test_pcm_buffer.py @@ -0,0 +1,185 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +import base64 + +import numpy as np +import pytest + +from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation +from vllm_omni.model_executor.common.duplex.pcm_buffer import ( + FixedFramePcmAppendBuffer, + FixedFramePcmAppendReservation, +) + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + +FRAME = 1920 +RATE = 24000 +PERIOD = 80 + + +def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = RATE, fmt: str = "pcm_f32le") -> dict[str, object]: + samples = np.ascontiguousarray(samples, dtype=" FixedFramePcmAppendBuffer: + return FixedFramePcmAppendBuffer(sample_rate_hz=RATE, frame_samples=FRAME, chunk_period_ms=PERIOD, model="Test") + + +def _decode(payload: dict[str, object]) -> np.ndarray: + return np.frombuffer(base64.b64decode(str(payload["audio"])), dtype=" None: + buffer = _buffer() + + assert isinstance(buffer, PcmAppendBuffer) + assert buffer.frame_bytes == FRAME * 4 + reservation = buffer.prepare_append( + _pcm_payload(np.arange(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True + ) + assert isinstance(reservation, FixedFramePcmAppendReservation) + assert isinstance(reservation, PcmAppendReservation) + + +def test_one_whole_frame_is_reserved_transactionally() -> None: + buffer = _buffer() + + reservation = buffer.prepare_append( + _pcm_payload(np.arange(FRAME)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True + ) + + assert reservation is not None + assert reservation.active + assert reservation.byte_count == FRAME * 4 + assert reservation.payload is not None and reservation.payload["sample_rate_hz"] == RATE + assert _decode(reservation.payload).tolist() == list(range(FRAME)) + assert buffer.pending_byte_count == 0 + assert buffer.has_reserved() and not buffer.has_pending() + + reservation.rollback() + + assert not reservation.active + assert buffer.pending_byte_count == FRAME * 4 + assert buffer.has_pending() and not buffer.has_reserved() + + +def test_partial_chunks_accumulate_until_a_whole_frame_exists() -> None: + buffer = _buffer() + half = _pcm_payload(np.zeros(FRAME // 2)) + + assert buffer.prepare_append(half, operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True) is None + assert buffer.pending_byte_count == FRAME * 2 + reservation = buffer.prepare_append(half, operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True) + + assert reservation is not None + assert reservation.byte_count == FRAME * 4 + assert buffer.pending_byte_count == 0 + + +def test_appends_are_not_emitted_while_emission_is_disallowed() -> None: + buffer = _buffer() + + assert ( + buffer.prepare_append( + _pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=False + ) + is None + ) + assert buffer.pending_byte_count == FRAME * 4 + + +def test_commit_flushes_a_padded_final_frame_and_an_empty_buffer_reserves_nothing() -> None: + buffer = _buffer() + buffer.prepare_append(_pcm_payload(np.ones(10)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=False) + + reservation = buffer.prepare_commit(operation_id="commit", chunk_period_ms=PERIOD) + + assert reservation.payload is not None + assert reservation.payload["final"] is True + decoded = _decode(reservation.payload) + assert decoded.size == FRAME + assert decoded[:10].tolist() == [1.0] * 10 and not decoded[10:].any() + assert reservation.byte_count == 40 + reservation.commit() + assert not buffer.has_reserved() + + empty = buffer.prepare_commit(operation_id="commit-2", chunk_period_ms=PERIOD) + assert empty.payload is None and empty.byte_count == 0 + empty.commit() + assert buffer.flush(chunk_period_ms=PERIOD) is None + + +def test_rolling_back_an_earlier_reservation_restores_later_ones_in_order() -> None: + buffer = _buffer() + first = buffer.prepare_append( + _pcm_payload(np.full(FRAME, 1.0)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True + ) + second = buffer.prepare_append( + _pcm_payload(np.full(FRAME, 2.0)), operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True + ) + assert first is not None and second is not None + + first.rollback() + + assert not first.active and not second.active + assert buffer.pending_byte_count == 2 * FRAME * 4 + replay = buffer.prepare_append( + _pcm_payload(np.zeros(0)), operation_id="op-3", chunk_period_ms=PERIOD, allow_emit=True + ) + assert replay is not None and _decode(replay.payload)[0] == 1.0 + + +def test_clear_drops_pending_audio_and_deactivates_reservations() -> None: + buffer = _buffer() + reservation = buffer.prepare_append( + _pcm_payload(np.zeros(FRAME + 8)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True + ) + assert reservation is not None and buffer.pending_byte_count == 32 + + buffer.clear() + buffer.clear_force_listen() + + assert not reservation.active + assert buffer.pending_byte_count == 0 + reservation.rollback() # no-op after clear + assert buffer.pending_byte_count == 0 + + +@pytest.mark.parametrize( + ("payload", "match"), + [ + (_pcm_payload(np.zeros(8), sample_rate_hz=16000), "sample_rate_hz must be 24000"), + (_pcm_payload(np.zeros(8), fmt="pcm16"), "format must be pcm_f32le"), + (_pcm_payload(np.array([0.0, np.nan])), "finite"), + ], +) +def test_malformed_audio_is_rejected_and_nothing_is_buffered(payload: dict[str, object], match: str) -> None: + buffer = _buffer() + + with pytest.raises(ValueError, match=match): + buffer.prepare_append(payload, operation_id="op", chunk_period_ms=PERIOD, allow_emit=True) + + assert buffer.pending_byte_count == 0 + + +def test_chunk_period_and_duplicate_operations_are_refused() -> None: + buffer = _buffer() + + with pytest.raises(ValueError, match="chunk_period_ms must be 80"): + buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=1000, allow_emit=True) + buffer.prepare_append(_pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True) + with pytest.raises(ValueError, match="duplicate active operation_id"): + buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True) + + +def test_constructor_validates_its_parameters() -> None: + with pytest.raises(ValueError, match="positive"): + FixedFramePcmAppendBuffer(sample_rate_hz=0, frame_samples=FRAME, chunk_period_ms=PERIOD) diff --git a/tests/model_executor/executor_common/test_request_outputs.py b/tests/model_executor/executor_common/test_request_outputs.py new file mode 100644 index 00000000000..e0c044acf42 --- /dev/null +++ b/tests/model_executor/executor_common/test_request_outputs.py @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +from types import SimpleNamespace + +import numpy as np +import pytest +import torch + +from vllm_omni.model_executor.common.request_outputs import ( + audio_sample_count, + audio_value, + coerce_int, + coerce_int_list, + first_completion, + multimodal_output, + sample_rate_hz, + slice_audio_delta, + text_delta, + text_value, + unwrap_request_output, +) + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + + +def test_unwrap_prefers_the_wrapped_request_output_and_its_first_completion() -> None: + completion = SimpleNamespace(text="hi", multimodal_output={}) + inner = SimpleNamespace(request_id="req", outputs=[completion, SimpleNamespace(text="second")]) + wrapper = SimpleNamespace(request_output=inner) + + output, first = unwrap_request_output(wrapper) + + assert output is inner + assert first is completion + assert first_completion(SimpleNamespace(outputs=[])) is None + assert first_completion(SimpleNamespace()) is None + + +def test_multimodal_output_takes_the_first_non_empty_mapping_as_a_copy() -> None: + completion = SimpleNamespace(multimodal_output={"audio": [1.0]}) + output = SimpleNamespace(multimodal_output={}) + + result = multimodal_output(output, completion) + + assert result == {"audio": [1.0]} + assert result is not completion.multimodal_output + assert multimodal_output(SimpleNamespace(multimodal_output=None), None) == {} + + +@pytest.mark.parametrize( + ("value", "expected"), + [ + (4, 4), + ("11", 11), + (2.9, 2), + (True, 1), + (None, None), + ("x", None), + (object(), None), + ([1], None), + (torch.tensor([7]), 7), + (torch.tensor(9), 9), + (torch.tensor([], dtype=torch.long), None), + (np.array([5], dtype=np.int32), 5), + (np.array([], dtype=np.int32), None), + ], +) +def test_coerce_int_handles_scalars_strings_tensors_and_arrays(value: object, expected: int | None) -> None: + assert coerce_int(value) == expected + + +def test_coerce_int_list_flattens_tensors_and_drops_non_integers() -> None: + assert coerce_int_list(torch.tensor([[1, 2], [3, 4]])) == [1, 2, 3, 4] + assert coerce_int_list(np.array([5, 6])) == [5, 6] + assert coerce_int_list([1, "2", "x", None, torch.tensor(3)]) == [1, 2, 3] + assert coerce_int_list(None) == [] + assert coerce_int_list("12") == [] + + +def test_audio_value_reads_the_known_keys_and_unwraps_a_single_item_list() -> None: + assert audio_value({"model_outputs": [np.zeros(2)]}).shape == (2,) + assert audio_value({"latent": "x"}) == "x" + assert audio_value({"audio": [1, 2]}) == [1, 2] + assert audio_value({}) is None + + +def test_text_value_prefers_multimodal_text_then_llm_text_then_the_completion() -> None: + completion = SimpleNamespace(text="from completion") + assert text_value({"text": "mm"}, completion) == "mm" + assert text_value({"llm_output_text": "llm"}, completion) == "llm" + assert text_value({}, completion) == "from completion" + assert text_value({"text": ""}, None) == "" + + +@pytest.mark.parametrize( + ("text", "previous", "expected"), + [("", "he", ""), ("he", "he", ""), ("hello", "he", "llo"), ("new", "old", "new"), ("abc", "", "abc")], +) +def test_text_delta(text: str, previous: str, expected: str) -> None: + assert text_delta(text, previous) == expected + + +def test_slice_audio_delta_returns_only_new_samples_for_cumulative_audio() -> None: + cumulative = np.arange(6, dtype=np.float32) + + assert slice_audio_delta(cumulative, 4).tolist() == [4.0, 5.0] + assert slice_audio_delta(cumulative, 6) is None + assert slice_audio_delta(cumulative, 0).tolist() == cumulative.tolist() + # A shorter waveform than the cursor means the stream restarted: send it whole. + assert slice_audio_delta(np.arange(3, dtype=np.float32), 4).tolist() == [0.0, 1.0, 2.0] + assert slice_audio_delta(None, 2) is None + assert slice_audio_delta(np.zeros(0, dtype=np.float32), 0) is None + + +def test_slice_audio_delta_keeps_tensors_as_contiguous_tensors() -> None: + delta = slice_audio_delta(torch.arange(6, dtype=torch.float32).reshape(2, 3), 4) + + assert isinstance(delta, torch.Tensor) + assert delta.is_contiguous() + assert delta.tolist() == [4.0, 5.0] + assert audio_sample_count(torch.zeros(2, 3)) == 6 + assert audio_sample_count("not audio") is None + + +@pytest.mark.parametrize( + ("multimodal", "expected"), + [ + ({"sr": 16000}, 16000), + ({"sample_rate_hz": 22050}, 22050), + ({"sr": [48000]}, 48000), + ({"sr": np.array([8000])}, 8000), + ({"sr": "bad"}, 24000), + ({"sr": 0}, 24000), + ({}, 24000), + ], +) +def test_sample_rate_hz_falls_back_to_the_default(multimodal: dict, expected: int) -> None: + assert sample_rate_hz(multimodal, default=24000) == expected diff --git a/tests/model_executor/models/personaplex/duplex/conftest.py b/tests/model_executor/models/personaplex/duplex/conftest.py deleted file mode 100644 index d49e8c61721..00000000000 --- a/tests/model_executor/models/personaplex/duplex/conftest.py +++ /dev/null @@ -1,21 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project - -"""PersonaPlex duplex is not on the duplex plugin framework yet. - -RFC vllm-omni#7181 splits the port: this PR lands the unified framework with -MiniCPM-o 4.5, and the follow-up PR moves PersonaPlex onto ``DuplexModelPlugin``. -Until then the modules under test still import the pre-framework duplex -runtime / serving adapter that this PR removed, so collecting them would fail -at import time and abort the whole pytest session. The tests themselves are -unchanged and come back with the port. - -Listed per file rather than globbed, so the coverage loss is visible and shrinks -file by file as the port lands. -""" - -#: ``test_unified_runtime`` imports ``engine.duplex.runtime`` and -#: ``entrypoints.duplex.runtime_adapter`` directly. ``test_stage0_runtime`` fails -#: transitively: ``personaplex.duplex.runtime_extension`` imports ``DuplexFence`` -#: from ``engine.duplex.messages``, which no longer exports it. -collect_ignore = ["test_stage0_runtime.py", "test_unified_runtime.py"] diff --git a/tests/model_executor/models/personaplex/duplex/test_plugin.py b/tests/model_executor/models/personaplex/duplex/test_plugin.py new file mode 100644 index 00000000000..56a90e4b5f5 --- /dev/null +++ b/tests/model_executor/models/personaplex/duplex/test_plugin.py @@ -0,0 +1,489 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +"""PersonaPlex on the unified full-duplex framework: the plugin contract and the E2E driver helpers.""" + +from __future__ import annotations + +import argparse +import base64 +from pathlib import Path +from types import SimpleNamespace +from typing import cast + +import numpy as np +import pytest +from vllm.sampling_params import SamplingParams + +from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver +from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy +from vllm_omni.engine.duplex.config import DuplexSessionConfig +from vllm_omni.engine.duplex.contracts import DuplexFence +from vllm_omni.engine.duplex.plugin import ( + DefaultDuplexModelSessionState, + DuplexDataPlaneContext, + DuplexRuntimeConfigError, + load_duplex_plugin, + validate_duplex_plugin_sampling, +) +from vllm_omni.model_executor.models.personaplex.duplex import stage0 +from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA, FRAME_SIZE, SAMPLE_RATE +from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession +from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer +from vllm_omni.model_executor.models.personaplex.duplex.plugin import ( + PRIVATE_RUNTIME_CONFIG_KEYS, + PersonaPlexDuplexPlugin, + PersonaPlexSessionState, +) +from vllm_omni.model_executor.models.personaplex.pipeline import PERSONAPLEX_PIPELINE + +pytestmark = [pytest.mark.core_model, pytest.mark.cpu] + +DEPLOY_PATH = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml" + + +def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = SAMPLE_RATE) -> dict[str, object]: + samples = np.ascontiguousarray(samples, dtype=" PersonaPlexDuplexPlugin: + return PersonaPlexDuplexPlugin(_encode_audio) + + +# --------------------------------------------------------------------------- # +# Pipeline binding # +# --------------------------------------------------------------------------- # + + +def test_pipeline_declares_the_duplex_plugin_and_no_pre_framework_wiring() -> None: + assert PERSONAPLEX_PIPELINE.duplex_plugin == ( + "vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin" + ) + assert PERSONAPLEX_PIPELINE.duplex_runtime_extension is None + assert PERSONAPLEX_PIPELINE.duplex_serving_adapter is None + assert PERSONAPLEX_PIPELINE.duplex_control_enabled is False + + plugin = load_duplex_plugin(PERSONAPLEX_PIPELINE.duplex_plugin, _encode_audio) + + assert isinstance(plugin, PersonaPlexDuplexPlugin) + assert plugin.plugin_id == "personaplex" + assert isinstance(plugin.data_plane, PersonaPlexDataPlaneSession) + validate_duplex_plugin_sampling(plugin, sampling_defaults=(SamplingParams(), SamplingParams())) + + +@pytest.mark.parametrize( + ("engine_arg", "expected"), + [("skip_tokenizer_init", True), ("enable_prefix_caching", False)], +) +def test_personaplex_stage_engine_args(engine_arg: str, expected: bool) -> None: + stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, load_deploy_config(DEPLOY_PATH)) + + assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [expected, expected] + + +def test_personaplex_deploy_is_duplex_and_propagates_capacity_to_all_model_stages() -> None: + deploy = load_deploy_config(DEPLOY_PATH) + stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy) + + assert deploy.session_mode == "duplex" + assert deploy.duplex_session.max_sessions == 2 + assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2] + assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"] + + +# --------------------------------------------------------------------------- # +# Session policy # +# --------------------------------------------------------------------------- # + + +def test_capabilities_are_honest() -> None: + single = _plugin().capabilities(max_sessions=1) + multi = _plugin().capabilities(max_sessions=2) + + assert multi.as_dict()["implementation_level"] == "model_native_duplex" + assert multi.as_dict()["input_modes"] == ["append_audio_chunk"] + assert multi.chunk_period_ms == 80 + assert single.supports_multi_session is False + assert multi.supports_multi_session is True + assert multi.supports_multi_session_same_replica is True + assert multi.supports_barge_in is False + assert multi.supports_client_commit is False + assert multi.supports_external_turn_signal is False + assert multi.supports_session_resume is False + assert multi.supports_audio_truncate is False + assert multi.supports_chat_completions is False + assert multi.session_admission_mode == "engine_managed" + + +def test_session_state_is_the_framework_default_with_the_personaplex_buffer() -> None: + state = _plugin().create_session_state() + + assert isinstance(state, PersonaPlexSessionState) + assert isinstance(state, DefaultDuplexModelSessionState) + assert isinstance(state.audio_buffer, PersonaPlexPcmAppendBuffer) + state.retain_committed_audio({"audio": ""}, operation_id="op", reserved_bytes=8) + assert state.clear_committed_audio() == 8 + assert state.committed_audio_payload is None + + +def test_private_runtime_keys_are_rejected_in_extra_body() -> None: + plugin = _plugin() + + assert PRIVATE_RUNTIME_CONFIG_KEYS == { + "personaplex_prefill_slots", + "personaplex_model_path", + "personaplex_voice_prompt", + "personaplex_persona", + } + plugin.validate_client_extra_body({"auto_response": True}) + plugin.validate_client_extra_body(None) + with pytest.raises(DuplexRuntimeConfigError, match="personaplex_persona") as excinfo: + plugin.validate_client_extra_body({"personaplex_persona": "x", "auto_response": True}) + assert excinfo.value.code == "invalid_duplex_runtime_config" + + +@pytest.mark.asyncio +async def test_prepare_runtime_config_resolves_voice_and_prefill_slots_once(monkeypatch) -> None: + calls: list[tuple[str, str, str]] = [] + + def fake_prefill_slots(model_path: str, voice: str, persona: str) -> int: + calls.append((model_path, voice, persona)) + return 42 + + monkeypatch.setattr(stage0, "personaplex_prefill_slots", fake_prefill_slots) + plugin = _plugin() + model_config = SimpleNamespace(model="/models/personaplex") + + runtime_config = await plugin.prepare_runtime_config( + DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."), + model_config=model_config, + ) + again = await plugin.prepare_runtime_config( + DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."), + model_config=model_config, + ) + defaults = await plugin.prepare_runtime_config(DuplexSessionConfig(), model_config=model_config) + + assert runtime_config == { + "personaplex_model_path": "/models/personaplex", + "personaplex_voice_prompt": "NATM1.pt", + "personaplex_persona": "Be brief.", + "personaplex_prefill_slots": 42, + } + assert again == runtime_config + assert defaults["personaplex_voice_prompt"] == "NATF2.pt" + assert defaults["personaplex_persona"] == DEFAULT_PERSONA + # The voice bundle and tokenizer are read once per (model, voice, persona). + assert calls == [ + ("/models/personaplex", "NATM1.pt", "Be brief."), + ("/models/personaplex", "NATF2.pt", DEFAULT_PERSONA), + ] + + +@pytest.mark.asyncio +async def test_prepare_runtime_config_reports_prefill_failures_as_runtime_config_errors(monkeypatch) -> None: + def failing_prefill_slots(model_path: str, voice: str, persona: str) -> int: + raise FileNotFoundError("voices.tgz missing") + + monkeypatch.setattr(stage0, "personaplex_prefill_slots", failing_prefill_slots) + + with pytest.raises(DuplexRuntimeConfigError, match="voices.tgz missing") as excinfo: + await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=SimpleNamespace(model="/m")) + assert excinfo.value.code == "prefill_unavailable" + + with pytest.raises(DuplexRuntimeConfigError, match="model path") as excinfo: + await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=None) + assert excinfo.value.code == "model_path_unavailable" + + +@pytest.mark.parametrize("voice", ["../NATF2.pt", "voices/NATF2.pt", "NATF2.wav", "/abs/NATF2.pt"]) +@pytest.mark.asyncio +async def test_prepare_runtime_config_rejects_voices_that_are_not_bundled_basenames(voice: str) -> None: + with pytest.raises(DuplexRuntimeConfigError, match="bundled .pt basename") as excinfo: + await _plugin().prepare_runtime_config( + DuplexSessionConfig(voice=voice), model_config=SimpleNamespace(model="/m") + ) + assert excinfo.value.code == "invalid_voice" + + +def test_runtime_config_update_rejects_changed_persona() -> None: + plugin = _plugin() + short_persona = "You are a helpful assistant." + current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"} + + with pytest.raises(DuplexRuntimeConfigError, match="persona") as excinfo: + plugin.runtime_config_for_update(DuplexSessionConfig(instructions="You are a pirate."), current) + assert excinfo.value.code == "persona_update_unsupported" + + unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(instructions=short_persona), current) + assert unchanged["personaplex_persona"] == short_persona + assert unchanged is not current + + +def test_runtime_config_update_rejects_changed_voice() -> None: + plugin = _plugin() + current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"} + + with pytest.raises(DuplexRuntimeConfigError, match="voice") as excinfo: + plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATM1.pt"), current) + assert excinfo.value.code == "voice_update_unsupported" + + unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATF2.pt"), current) + assert unchanged["personaplex_voice_prompt"] == "NATF2.pt" + + +def test_data_plane_context_is_the_framework_default() -> None: + context = _plugin().data_plane_context( + epoch=1, + turn_id=2, + active_response_turn_id=None, + active_response_id="resp", + auto_responds=True, + response_format="pcm16", + speed=None, + modalities=("audio",), + ) + + assert isinstance(context, DuplexDataPlaneContext) + assert context.response_format == "pcm16" and context.epoch == 1 + + +# --------------------------------------------------------------------------- # +# Engine policy # +# --------------------------------------------------------------------------- # + + +def test_sampling_params_are_greedy_one_token_on_stage0_only() -> None: + defaults = (SamplingParams(temperature=0.8, top_k=10, max_tokens=10), SamplingParams(max_tokens=1024)) + + configured = _plugin().configure_sampling_params(runtime_config={}, defaults=defaults) + + assert configured[0].temperature == 0.0 + assert configured[0].top_k == 1 + assert configured[0].max_tokens == 1 + assert configured[1] is defaults[1] + assert defaults[0].max_tokens == 10 + assert _plugin().configure_sampling_params(runtime_config={}, defaults=()) == () + + +def test_plan_append_reserves_one_slot_plus_prefill_on_the_first_seq() -> None: + plugin = _plugin() + fence = DuplexFence("session", epoch=3, turn_id=1) + common = { + "request_id": "req", + "fence": fence, + "session_config": {"instructions": "Be concise."}, + "runtime_config": {"personaplex_prefill_slots": 4}, + "turn_seq": 1, + "payload": _pcm_payload(np.zeros(FRAME_SIZE, np.float32)), + "final": False, + "sampling_params": SamplingParams(), + } + + first = plugin.plan_append(seq=1, **common) + second = plugin.plan_append(seq=2, **common) + + assert len(first.prompt["prompt_token_ids"]) == 5 + assert len(second.prompt["prompt_token_ids"]) == 1 + duplex = first.prompt["model_intermediate_buffer"]["duplex"] + assert duplex["session_id"] == "session" + assert duplex["epoch"] == 3 + assert duplex["turn_id"] == 1 + assert duplex["seq"] == 1 + assert duplex["data_plane"] is True + assert duplex["fence"] == fence + assert duplex["scheduler_token_budget"] == 5 + assert duplex["runtime_config"] == {"personaplex_prefill_slots": 4} + assert "incarnation" not in duplex + assert first.prompt["model_intermediate_buffer"]["global_request_id"] == ["session"] + + +@pytest.mark.parametrize( + ("payload", "match"), + [ + (_pcm_payload(np.zeros(FRAME_SIZE - 1, np.float32)), "exactly 1920 samples"), + (_pcm_payload(np.zeros(FRAME_SIZE, np.float32), sample_rate_hz=16000), "sample_rate_hz must be 24000"), + ({"format": "pcm16", "sample_rate_hz": SAMPLE_RATE, "audio": ""}, "format must be pcm_f32le"), + ("frame", "must be a mapping"), + ], +) +def test_plan_append_rejects_anything_but_one_24k_frame(payload: object, match: str) -> None: + with pytest.raises(ValueError, match=match): + _plugin().plan_append( + request_id="req", + fence=DuplexFence("session"), + session_config={}, + runtime_config={}, + seq=1, + turn_seq=1, + payload=payload, + final=False, + sampling_params=None, + ) + + +def test_plan_append_rejects_a_malformed_prefill_slot_count() -> None: + with pytest.raises(ValueError, match="personaplex_prefill_slots"): + _plugin().plan_append( + request_id="req", + fence=DuplexFence("session"), + session_config={}, + runtime_config={"personaplex_prefill_slots": "many"}, + seq=1, + turn_seq=1, + payload=_pcm_payload(np.zeros(FRAME_SIZE, np.float32)), + final=False, + sampling_params=None, + ) + + +def test_decide_output_never_decides() -> None: + assert ( + _plugin().decide_output( + stage_id=0, + final_stage_id=1, + segment_finished=True, + segment_token_ids=(1,), + segment_output_metadata={}, + output=SimpleNamespace(outputs=[]), + ) + is None + ) + + +def test_silence_unit_is_one_24k_frame_that_plan_append_accepts() -> None: + plugin = _plugin() + + unit = plugin.silence_unit_payload() + + assert unit["format"] == "pcm_f32le" + assert unit["sample_rate_hz"] == SAMPLE_RATE + assert len(base64.b64decode(unit["audio"])) == FRAME_SIZE * 4 + plan = plugin.plan_append( + request_id="req", + fence=DuplexFence("session"), + session_config={}, + runtime_config={}, + seq=7, + turn_seq=7, + payload=unit, + final=False, + sampling_params=None, + ) + assert len(plan.prompt["prompt_token_ids"]) == 1 + + +# --------------------------------------------------------------------------- # +# E2E driver helpers (CPU-checked parts of the GPU driver) # +# --------------------------------------------------------------------------- # + + +def _audio_client(frame_count: int = 10, *, voiced_frames: int = 0) -> e2e_driver.RawRealtimeProbe: + """A probe double carrying only the collected events the driver's result builder reads.""" + frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype=" None: + silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype=" None: + partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype=" None: + args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5) + + with pytest.raises(AssertionError, match="audible speech"): + e2e_driver._session_result(_audio_client(frame_count), input_frames=frame_count, args=args, minimum_chunks=1) + + +def test_e2e_driver_uses_absolute_audible_floor() -> None: + args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5) + + e2e_driver._session_result(_audio_client(200, voiced_frames=5), input_frames=200, args=args, minimum_chunks=1) + + +def test_e2e_driver_defaults_replacement_to_full_input() -> None: + args = e2e_driver.parse_args(["--model", "/model", "--input-wav", "/input.wav"]) + + assert args.replacement_frames == 0 + assert args.min_voiced_frames == 5 + + +def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None: + input_wav = tmp_path / "input.wav" + input_wav.write_bytes(b"fixture-wav") + + identity = e2e_driver._input_identity( + input_wav, + expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304", + ) + + assert identity == { + "path": str(input_wav.resolve()), + "sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304", + } + + +def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None: + input_wav = tmp_path / "input.wav" + input_wav.write_bytes(b"fixture-wav") + + with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"): + e2e_driver._input_identity(input_wav, expected_sha256="0" * 64) + + +def test_e2e_driver_lets_the_server_allocate_the_session_id() -> None: + url = e2e_driver._realtime_url("ws://127.0.0.1:8099/v1/realtime?duplex=1", "nvidia/personaplex-7b-v1") + + assert "session_id=" not in url + assert "duplex=1" in url and "autostart=0" in url diff --git a/tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py b/tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py index b585399b724..61c6e6c1b0f 100644 --- a/tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py +++ b/tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py @@ -14,6 +14,7 @@ ) from vllm_omni.model_executor.models.personaplex.duplex.stage0 import ( PersonaPlexStage0DuplexRuntime, + PersonaPlexStage0PreparedAppend, ) from vllm_omni.model_executor.models.personaplex.personaplex_talker import ( PersonaPlexTalkerForConditionalGeneration, @@ -80,13 +81,12 @@ def _build_frame_embed( return torch.full((1, 4), value, device=device) -def _duplex_info(*, seq: int, session_id: str = "session", incarnation: int = 1): +def _duplex_info(*, seq: int, session_id: str = "session", epoch: int = 0): pcm = np.zeros(1920, dtype=" PersonaPlexStage0DuplexRuntime: voice_embeddings = torch.arange(8, dtype=torch.float32).reshape(2, 1, 1, 4) - codec_args = {"codec": codecs[0]} + codec_args: dict[str, object] = {"codec": codecs[0]} if len(codecs) > 1: available = iter(codecs) codec_args = { @@ -248,15 +248,20 @@ def test_decoded_pcm_is_writable_for_torch_zero_copy() -> None: def _prepare_two_sessions( runtime: PersonaPlexStage0DuplexRuntime, -) -> tuple[object, object, object, object]: +) -> tuple[ + PersonaPlexStage0PreparedAppend, + PersonaPlexStage0PreparedAppend, + PersonaPlexStage0PreparedAppend, + PersonaPlexStage0PreparedAppend, +]: first_1 = runtime.prepare_append(_duplex_info(seq=1), prompt_len=18) second_1 = runtime.prepare_append( - _duplex_info(seq=1, session_id="other", incarnation=2), + _duplex_info(seq=1, session_id="other"), prompt_len=18, ) first_2 = runtime.prepare_append(_duplex_info(seq=2), prompt_len=19) second_2 = runtime.prepare_append( - _duplex_info(seq=2, session_id="other", incarnation=2), + _duplex_info(seq=2, session_id="other"), prompt_len=19, ) return first_1, second_1, first_2, second_2 @@ -281,7 +286,7 @@ def test_stage0_session_capacity_fails_before_codec_state_is_shared() -> None: with pytest.raises(RuntimeError, match="capacity 2"): runtime.prepare_append( - _duplex_info(seq=1, session_id="third", incarnation=3), + _duplex_info(seq=1, session_id="third"), prompt_len=18, ) @@ -292,12 +297,55 @@ def test_close_session_resets_and_reuses_released_codec() -> None: runtime = _runtime(first_codec, second_codec) _prepare_two_sessions(runtime) - runtime.close_session("session", 1) + runtime.close_session("session", 0) assert first_codec.reset_calls == 1 assert second_codec.reset_calls == 0 replacement = runtime.prepare_append( - _duplex_info(seq=1, session_id="replacement", incarnation=4), + _duplex_info(seq=1, session_id="replacement"), prompt_len=18, ) assert replacement.user_codes[:, 0].tolist() == [3] + + +def test_a_new_epoch_replays_the_prefill_and_recycles_the_codec() -> None: + codec = _FakeCodec() + runtime = _runtime(codec) + runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0") + runtime.prepare_append(_duplex_info(seq=2), prompt_len=19, request_id="req-e0") + + # A cancel advanced the fence: the next append is seq 1 of epoch 1 on a + # fresh Stage 0 request, so the voice/persona prefill is replayed and the + # earlier epoch's lockstep state is released first. + restarted = runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1") + + assert restarted.prefill_applied is True + assert restarted.prompt_offset == 0 + assert restarted.user_codes.shape == (1, 8) + assert list(runtime.sessions) == [("session", 1)] + assert runtime.request_sessions == {"req-e1": ("session", 1)} + assert codec.reset_calls == 1 + assert runtime.sessions[("session", 1)].codec is codec + + +def test_a_late_finish_of_the_old_epoch_request_does_not_close_the_new_state() -> None: + runtime = _runtime(_FakeCodec()) + runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0") + runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1") + + runtime.close_request("req-e0") + + assert list(runtime.sessions) == [("session", 1)] + runtime.close_request("req-e1") + assert runtime.sessions == {} + + +def test_stage0_capacity_counts_live_epochs_not_superseded_ones() -> None: + runtime = _runtime(_FakeCodec(), _FakeCodec()) + runtime.prepare_append(_duplex_info(seq=1), prompt_len=18) + runtime.prepare_append(_duplex_info(seq=1, session_id="other"), prompt_len=18) + + # Restarting one session must not need a third codec. + runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18) + + assert sorted(runtime.sessions) == [("other", 0), ("session", 1)] diff --git a/tests/model_executor/models/personaplex/duplex/test_unified_runtime.py b/tests/model_executor/models/personaplex/duplex/test_unified_runtime.py deleted file mode 100644 index e4f9ad9b1fc..00000000000 --- a/tests/model_executor/models/personaplex/duplex/test_unified_runtime.py +++ /dev/null @@ -1,420 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project - -import base64 -from pathlib import Path -from types import SimpleNamespace - -import numpy as np -import pytest -from vllm.sampling_params import SamplingParams -from vllm_omni.engine.duplex.runtime import ( - DuplexInputMode, -) -from vllm_omni.entrypoints.duplex.runtime_adapter import ( - ServingRuntimeConfigError, -) - -from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver -from vllm_omni.config.stage_config import ( - load_deploy_config, - merge_pipeline_deploy, -) -from vllm_omni.engine.duplex.messages import DuplexFence -from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA -from vllm_omni.model_executor.models.personaplex.duplex.data_plane import ( - PersonaPlexDataPlaneContext, - PersonaPlexDataPlaneSession, -) -from vllm_omni.model_executor.models.personaplex.duplex.input import ( - PersonaPlexPcmAppendBuffer, -) -from vllm_omni.model_executor.models.personaplex.duplex.runtime_extension import ( - PersonaPlexDuplexRuntimeExtension, -) -from vllm_omni.model_executor.models.personaplex.duplex.serving_adapter import ( - PersonaPlexServingRuntimeAdapter, -) -from vllm_omni.model_executor.models.personaplex.pipeline import ( - PERSONAPLEX_PIPELINE, -) - -pytestmark = [pytest.mark.core_model, pytest.mark.cpu] - - -def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = 24000) -> dict[str, object]: - samples = np.ascontiguousarray(samples, dtype=" SimpleNamespace: - frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype=" None: - silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype=" None: - partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype=" None: - args = SimpleNamespace( - max_frame_deficit=4, - voiced_frame_rms_threshold=1e-3, - min_voiced_frames=5, - ) - - with pytest.raises(AssertionError, match="audible speech"): - e2e_driver._session_result( - _audio_client(frame_count), - input_frames=frame_count, - args=args, - minimum_chunks=1, - ) - - -def test_e2e_driver_uses_absolute_audible_floor() -> None: - args = SimpleNamespace( - max_frame_deficit=4, - voiced_frame_rms_threshold=1e-3, - min_voiced_frames=5, - ) - - e2e_driver._session_result( - _audio_client(200, voiced_frames=5), - input_frames=200, - args=args, - minimum_chunks=1, - ) - - -def test_e2e_driver_defaults_replacement_to_full_input() -> None: - args = e2e_driver.parse_args( - [ - "--model", - "/model", - "--input-wav", - "/input.wav", - ] - ) - - assert args.replacement_frames == 0 - assert args.min_voiced_frames == 5 - - -def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None: - input_wav = tmp_path / "input.wav" - input_wav.write_bytes(b"fixture-wav") - - identity = e2e_driver._input_identity( - input_wav, - expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304", - ) - - assert identity == { - "path": str(input_wav.resolve()), - "sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304", - } - - -def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None: - input_wav = tmp_path / "input.wav" - input_wav.write_bytes(b"fixture-wav") - - with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"): - e2e_driver._input_identity(input_wav, expected_sha256="0" * 64) - - -@pytest.mark.parametrize( - ("attribute", "expected"), - [ - ( - "duplex_runtime_extension", - "vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension", - ), - ( - "duplex_serving_adapter", - "vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter", - ), - ], -) -def test_personaplex_pipeline_registers_unified_duplex_components( - attribute: str, - expected: str, -) -> None: - assert PERSONAPLEX_PIPELINE.duplex_control_enabled is True - assert getattr(PERSONAPLEX_PIPELINE, attribute) == expected - - -@pytest.mark.parametrize( - ("engine_arg", "expected"), - [ - ("skip_tokenizer_init", True), - ("enable_prefix_caching", False), - ], -) -def test_personaplex_stage_engine_args( - engine_arg: str, - expected: bool, -) -> None: - deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml" - stages = merge_pipeline_deploy( - PERSONAPLEX_PIPELINE, - load_deploy_config(deploy_path), - ) - - assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [ - expected, - expected, - ] - - -def test_personaplex_duplex_capacity_is_propagated_to_all_model_stages() -> None: - deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml" - deploy = load_deploy_config(deploy_path) - stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy) - - assert deploy.duplex_session.max_sessions == 2 - assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2] - assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"] - - -def test_personaplex_capabilities_are_honest() -> None: - single = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=1) - multi = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=2) - - assert multi.implementation_level == "model_native_duplex" - assert multi.input_modes == ["append_audio_chunk"] - assert multi.chunk_period_ms == 80 - assert single.supports_multi_session is False - assert multi.supports_multi_session is True - assert multi.supports_multi_session_same_replica is True - assert multi.supports_barge_in is False - - -def test_personaplex_runtime_config_update_rejects_changed_persona() -> None: - adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None) - short_persona = "You are a helpful assistant." - longer_persona = ( - "You are now a swashbuckling pirate captain who speaks entirely in " - "nautical slang, references buried treasure constantly, and never " - "breaks character no matter what the user asks." - ) - current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"} - - with pytest.raises(ServingRuntimeConfigError, match="persona"): - adapter.runtime_config_for_update( - SimpleNamespace(instructions=longer_persona, extra_body={}), - current, - ) - - unchanged = adapter.runtime_config_for_update( - SimpleNamespace(instructions=short_persona, extra_body={}), - current, - ) - assert unchanged["personaplex_persona"] == short_persona - - -def test_personaplex_runtime_config_update_rejects_changed_voice() -> None: - adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None) - current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"} - - with pytest.raises(ServingRuntimeConfigError, match="voice"): - adapter.runtime_config_for_update( - SimpleNamespace(instructions=None, voice="OtherVoice.pt", extra_body={}), - current, - ) - - unchanged = adapter.runtime_config_for_update( - SimpleNamespace(instructions=None, voice="NATF2.pt", extra_body={}), - current, - ) - assert unchanged["personaplex_voice_prompt"] == "NATF2.pt" - - -def test_personaplex_rejects_native_duplex_opt_out_flag() -> None: - # PersonaPlex is always model-native; the per-session opt-in knob is - # server-owned there, under both the canonical name and its deprecated - # model-prefixed alias. - adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None) - - with pytest.raises(ServingRuntimeConfigError, match="native_duplex"): - adapter.validate_client_extra_body({"native_duplex": False}) - with pytest.raises(ServingRuntimeConfigError, match="minicpmo45_native_duplex"): - adapter.validate_client_extra_body({"minicpmo45_native_duplex": False}) - - -def test_pcm_buffer_emits_one_80ms_frame_transactionally() -> None: - buffer = PersonaPlexPcmAppendBuffer() - reservation = buffer.prepare_append( - _pcm_payload(np.arange(1920, dtype=np.float32)), - operation_id="op-1", - chunk_period_ms=80, - allow_emit=True, - ) - - assert reservation is not None - assert reservation.byte_count == 1920 * 4 - assert buffer.pending_byte_count == 0 - reservation.rollback() - assert buffer.pending_byte_count == 1920 * 4 - - -def test_pcm_buffer_rejects_changed_sample_rate() -> None: - buffer = PersonaPlexPcmAppendBuffer() - assert ( - buffer.prepare_append( - _pcm_payload(np.zeros(100, np.float32)), - operation_id="op-1", - chunk_period_ms=80, - allow_emit=False, - ) - is None - ) - - with pytest.raises(ValueError, match="sample_rate"): - buffer.prepare_append( - _pcm_payload(np.zeros(100, np.float32), sample_rate_hz=16000), - operation_id="op-2", - chunk_period_ms=80, - allow_emit=False, - ) - - -def test_pcm_buffer_rejects_non_finite_samples() -> None: - buffer = PersonaPlexPcmAppendBuffer() - - with pytest.raises(ValueError, match="finite"): - buffer.prepare_append( - _pcm_payload(np.array([0.0, np.nan], np.float32)), - operation_id="op-1", - chunk_period_ms=80, - allow_emit=False, - ) - - -def test_runtime_extension_builds_one_frame_scheduler_append() -> None: - extension = PersonaPlexDuplexRuntimeExtension() - defaults = ( - SamplingParams(temperature=0.8, top_k=10, max_tokens=10), - SamplingParams(max_tokens=1024), - ) - - configured = extension.configure_sampling_params(runtime_config={}, defaults=defaults) - assert configured[0].temperature == 0.0 - assert configured[0].top_k == 1 - assert configured[0].max_tokens == 1 - assert configured[1] is defaults[1] - - plan = extension.plan_append( - request_id="req", - fence=DuplexFence("session", incarnation=2, epoch=3), - session_config={"instructions": "Be concise."}, - runtime_config={"personaplex_prefill_slots": 4}, - seq=1, - turn_seq=1, - mode=DuplexInputMode.APPEND_AUDIO_CHUNK, - payload=_pcm_payload(np.zeros(1920, np.float32)), - final=False, - sampling_params=configured[0], - ) - assert len(plan.prompt["prompt_token_ids"]) == 5 - duplex = plan.prompt["model_intermediate_buffer"]["duplex"] - assert duplex["session_id"] == "session" - assert duplex["incarnation"] == 2 - assert duplex["epoch"] == 3 - assert duplex["seq"] == 1 - assert duplex["data_plane"] is True - - -def test_projector_emits_only_cumulative_audio_and_text_suffixes() -> None: - encoded_sizes: list[int] = [] - - def encode_audio(audio, _sample_rate, _response_format, _speed): - size = int(np.asarray(audio, dtype=np.float32).size) - encoded_sizes.append(size) - return f"audio-{size}" - - projector = PersonaPlexDataPlaneSession(encode_audio) - projector.begin_request("req") - context = PersonaPlexDataPlaneContext(response_format="wav") - - first = SimpleNamespace( - request_id="req", - outputs=[SimpleNamespace(text="he", multimodal_output={})], - multimodal_output={"model_outputs": np.arange(4, dtype=np.float32), "sr": 24000}, - finished=False, - ) - second = SimpleNamespace( - request_id="req", - outputs=[SimpleNamespace(text="hello", multimodal_output={})], - multimodal_output={"model_outputs": np.arange(6, dtype=np.float32), "sr": 24000}, - finished=False, - ) - - projected = list( - projector.project( - {"data_plane_outputs": [first, second]}, - context=context, - ) - ) - - assert encoded_sizes == [4, 2] - assert [item["text"] for item in projected] == ["he", "llo"] - assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"] - assert all(item["sample_rate_hz"] == 24000 for item in projected) - assert all(item["end_of_turn"] is False for item in projected) diff --git a/vllm_omni/config/stage_config.py b/vllm_omni/config/stage_config.py index 00ebda453cc..0fe7efe4f57 100644 --- a/vllm_omni/config/stage_config.py +++ b/vllm_omni/config/stage_config.py @@ -324,10 +324,10 @@ class PipelineConfig: duplex_plugin: str | None = None # Preserve legacy turn deployments when adding an optional duplex plugin. default_session_mode: str | None = None - # Legacy duplex wiring of the models that are not ported to the plugin - # framework yet (PersonaPlex, Nemotron VoiceChat). Nothing reads them: a - # pipeline that only declares these is served turn-based. Each field goes - # away with the follow-up PR that ports its model to ``duplex_plugin``. + # Legacy duplex wiring of the model that is not ported to the plugin + # framework yet (Nemotron VoiceChat, RFC vllm-omni#7181 PR 4). Nothing + # reads them: a pipeline that only declares these is served turn-based. + # The fields go away with the PR that ports it to ``duplex_plugin``. duplex_runtime_extension: str | None = None duplex_serving_adapter: str | None = None duplex_control_enabled: bool = False diff --git a/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py b/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py index 5e262ff1e01..6d7a2a7d359 100644 --- a/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py +++ b/vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py @@ -37,6 +37,7 @@ QwenImageTransformer2DModel, ) from vllm_omni.diffusion.models.qwen_image.rope_utils import txt_seq_lens_from_embeds +from vllm_omni.diffusion.offloader.config import OffloadStrategy, resolve_offload_strategy from vllm_omni.diffusion.profiler.diffusion_pipeline_profiler import DiffusionPipelineProfilerMixin from vllm_omni.diffusion.utils.prompt_utils import ( validate_prompt_sequence_lengths, @@ -341,7 +342,7 @@ def __init__( # do not share VRAM with DiT construction (#7555). The DiT follows the # loader's default-device context (CUDA for online / AutoRound INT under # offload, CPU for layerwise / unquantized HSDP defer). - cpu_offload = bool(getattr(self.od_config, "enable_cpu_offload", False)) + cpu_offload = resolve_offload_strategy(self.od_config) is OffloadStrategy.MODEL_LEVEL enc_vae_device = torch.device("cpu") if cpu_offload else self.device self.text_encoder = self.text_encoder.to(enc_vae_device) self.vae = from_pretrained_with_prefetch( diff --git a/vllm_omni/engine/duplex/plugin.py b/vllm_omni/engine/duplex/plugin.py index 4fb23977b77..26402f108ba 100644 --- a/vllm_omni/engine/duplex/plugin.py +++ b/vllm_omni/engine/duplex/plugin.py @@ -16,9 +16,12 @@ from abc import ABC, abstractmethod from collections.abc import Callable, Iterable, Mapping from dataclasses import dataclass +from functools import lru_cache from importlib import import_module from typing import TYPE_CHECKING +import pybase64 as base64 + from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig from vllm_omni.engine.duplex.contracts import ( DuplexAppendPlan, @@ -219,6 +222,20 @@ def clear_continuation(self) -> None: self.silence_deadline_monotonic = None +@dataclass(frozen=True, slots=True) +class DuplexDataPlaneContext: + """Session state the runner hands a data plane to project one stage output.""" + + epoch: int = 0 + turn_id: int = 0 + active_response_turn_id: int | None = None + active_response_id: str | None = None + auto_responds: bool = False + response_format: str = "wav" + speed: float | None = None + modalities: tuple[str, ...] = () + + class DuplexDataPlane(ABC): """Projects raw stage outputs of one model into internal duplex events.""" @@ -277,12 +294,29 @@ class DuplexModelPlugin(ABC): private_runtime_config_keys: frozenset[str] = frozenset() #: Samples per silence unit the runner appends to keep a model turn going. silence_continuation_samples: int = 16000 + #: Sample rate of that unit: the runner submits it through ``plan_append`` + #: exactly like client audio, so it must be a unit the model accepts. + silence_continuation_sample_rate_hz: int = 16000 data_plane: DuplexDataPlane def __init__(self, encode_audio: EncodeAudio) -> None: # Constructor-only: concrete plugins hand the encoder to their data plane. del encode_audio + def silence_unit_payload(self) -> dict[str, object]: + """One silence unit as an append payload (``pcm_f32le`` zeros). + + Used by the runner's turn continuation and by the startup warmup; a + model whose unit is not plain zero PCM overrides it. + """ + samples = int(self.silence_continuation_samples) + return { + "type": "audio", + "audio": _silence_pcm_f32le_base64(samples), + "format": "pcm_f32le", + "sample_rate_hz": int(self.silence_continuation_sample_rate_hz), + } + # ---- engine policy (was DuplexRuntimeExtension) ---- @abstractmethod @@ -430,8 +464,15 @@ def create_session_state(self) -> DuplexModelSessionState: ... @abstractmethod def capabilities(self, *, max_sessions: int) -> DuplexCapabilities: ... - @abstractmethod - def validate_client_extra_body(self, extra_body: object) -> None: ... + def validate_client_extra_body(self, extra_body: object) -> None: + """Refuse client ``extra_body`` keys the server owns (``private_runtime_config_keys``).""" + if not isinstance(extra_body, Mapping): + return + private_keys = sorted(self.private_runtime_config_keys.intersection(extra_body)) + if private_keys: + raise DuplexRuntimeConfigError( + f"{self.plugin_id} runtime configuration is server-owned: " + ", ".join(private_keys) + ) @abstractmethod async def prepare_runtime_config( @@ -445,7 +486,6 @@ def runtime_config_for_update( current: Mapping[str, object], ) -> dict[str, object]: ... - @abstractmethod def data_plane_context( self, *, @@ -457,7 +497,18 @@ def data_plane_context( response_format: str, speed: float | None, modalities: tuple[str, ...], - ) -> object: ... + ) -> object: + """The context handed to ``data_plane.project``; the default is the generic dataclass.""" + return DuplexDataPlaneContext( + epoch=epoch, + turn_id=turn_id, + active_response_turn_id=active_response_turn_id, + active_response_id=active_response_id, + auto_responds=auto_responds, + response_format=response_format, + speed=speed, + modalities=modalities, + ) # Optional hook: build the runtime config patch for a function-call output # item. Plugins without tools keep the default (no change). @@ -515,6 +566,11 @@ def validate_duplex_plugin_sampling(plugin: DuplexModelPlugin, *, sampling_defau ) +@lru_cache(maxsize=8) +def _silence_pcm_f32le_base64(samples: int) -> str: + return base64.b64encode(bytes(max(0, samples) * 4)).decode("ascii") + + def payload_turn_id(payload: object) -> int | None: if not isinstance(payload, Mapping): return None @@ -537,6 +593,7 @@ def coerce_int(value: object) -> int | None: __all__ = [ "DefaultDuplexModelSessionState", "DuplexDataPlane", + "DuplexDataPlaneContext", "DuplexModelPlugin", "DuplexModelSessionState", "DuplexRuntimeConfigError", diff --git a/vllm_omni/engine/duplex/session/emitter.py b/vllm_omni/engine/duplex/session/emitter.py index 6ccfcc142ab..759bd6c2762 100644 --- a/vllm_omni/engine/duplex/session/emitter.py +++ b/vllm_omni/engine/duplex/session/emitter.py @@ -121,7 +121,14 @@ def emit(self, payload: dict[str, object]) -> None: # ------------------------------------------------------------------ # def auto_responds(self) -> bool: - """Whether the session answers committed input without a ``response.create``.""" + """Whether the session answers committed input without a ``response.create``. + + A model that takes no client commits (``supports_client_commit`` off, a + lockstep model) can only auto-respond; for every other model the + client opts in through ``extra_body.auto_response``. + """ + if not self._ctx.session.capabilities.supports_client_commit: + return True extra = getattr(self._ctx.session.config, "extra_body", None) if not isinstance(extra, dict): return False diff --git a/vllm_omni/engine/duplex/session/model_channel.py b/vllm_omni/engine/duplex/session/model_channel.py index 1c2d67c5885..e2bccf9b795 100644 --- a/vllm_omni/engine/duplex/session/model_channel.py +++ b/vllm_omni/engine/duplex/session/model_channel.py @@ -29,7 +29,6 @@ from dataclasses import replace from typing import TYPE_CHECKING, Protocol -import pybase64 as base64 from vllm.logger import init_logger from vllm_omni.engine.duplex.config import DuplexSessionState @@ -83,7 +82,6 @@ class ModelChannel: """Appends out to the model, events back from it, for one session.""" # One MiniCPM model unit (1 s at 16 kHz) is the compatibility default. - _SILENCE_UNIT_PAYLOAD_AUDIO = base64.b64encode(bytes(16000 * 4)).decode("ascii") _RESPONSE_MAX_CONTINUATION_UNITS = 8 _AUTO_RESPONSE_MAX_CONTINUATION_UNITS = 64 @@ -1210,13 +1208,8 @@ def _attach_runtime_metadata( # ------------------------------------------------------------------ # def silence_unit_payload(self) -> dict[str, object]: - samples = int(self._ctx.plugin.silence_continuation_samples) - audio = ( - self._SILENCE_UNIT_PAYLOAD_AUDIO - if samples == 16000 - else base64.b64encode(bytes(samples * 4)).decode("ascii") - ) - return {"type": "audio", "audio": audio, "format": "pcm_f32le", "sample_rate_hz": 16000} + """One silence unit in the model's own format: the plugin defines it (rate and length).""" + return dict(self._ctx.plugin.silence_unit_payload()) def response_continuations_remaining(self, response_id: str) -> bool: model_state = self._ctx.model_state diff --git a/vllm_omni/entrypoints/duplex/warmup.py b/vllm_omni/entrypoints/duplex/warmup.py index 14c098735a5..4625313b1ad 100644 --- a/vllm_omni/entrypoints/duplex/warmup.py +++ b/vllm_omni/entrypoints/duplex/warmup.py @@ -94,6 +94,26 @@ def _warmup_jpeg_b64() -> str: return base64.b64encode(buf.getvalue()).decode("ascii") +_DEFAULT_SILENCE_SAMPLES = 16000 +_DEFAULT_SILENCE_SAMPLE_RATE_HZ = 16000 + + +def warmup_silence_unit(plugin: object | None) -> dict[str, object]: + """The append the silent-frames warmup sends: the plugin's silence unit, or the 16 kHz default.""" + factory = getattr(plugin, "silence_unit_payload", None) + if callable(factory): + unit = dict(factory()) + if isinstance(unit.get("audio"), str) and isinstance(unit.get("format"), str): + unit.setdefault("sample_rate_hz", _DEFAULT_SILENCE_SAMPLE_RATE_HZ) + return unit + samples = int(getattr(plugin, "silence_continuation_samples", _DEFAULT_SILENCE_SAMPLES)) + return { + "audio": base64.b64encode(bytes(samples * 4)).decode("ascii"), + "format": "pcm_f32le", + "sample_rate_hz": _DEFAULT_SILENCE_SAMPLE_RATE_HZ, + } + + async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None: """Run one throwaway realtime session before real clients are admitted. @@ -121,9 +141,14 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None: model_name = served else: model_name = args.model - # One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin). - frame_samples = int(getattr(plugin, "silence_continuation_samples", 16000)) - native_append = plugin.capabilities(max_sessions=1).supports_core_resumable_request if plugin else True + # One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin): + # the unit's format, rate and length are the model's, not a fixed 16 kHz. + unit = warmup_silence_unit(plugin) + frame_samples = len(base64.b64decode(str(unit["audio"]))) // 4 + capabilities = getattr(plugin, "capabilities", None) + native_append = ( + bool(capabilities(max_sessions=1).supports_core_resumable_request) if callable(capabilities) else True + ) from vllm_omni.clients.duplex import build_realtime_url url = ( @@ -163,6 +188,10 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None: "turn_detection": None, "extra_body": {"auto_response": native_append}, } + if kind != "video_turn": + session["input_audio_format"] = unit["format"] + session["sample_rate_hz"] = unit["sample_rate_hz"] + session["audio"] = {"input": {"sample_rate_hz": unit["sample_rate_hz"]}} if kind == "video_turn": # The default AURA prompt answers with <|silent|> unless the user # asked for something. This session is thrown away; force one short @@ -205,9 +234,17 @@ async def _recv_until(predicate, timeout_s: float) -> bool: ) sent = 1 else: - silence = base64.b64encode(bytes(frame_samples * 4)).decode("ascii") while sent < warmup_frames: - await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence})) + await ws.send( + json.dumps( + { + "type": "input_audio_buffer.append", + "audio": unit["audio"], + "format": unit["format"], + "sample_rate_hz": unit["sample_rate_hz"], + } + ) + ) sent += 1 await asyncio.sleep(0.08) if not native_append: @@ -251,4 +288,5 @@ async def _recv_until(predicate, timeout_s: float) -> bool: "_warmup_duplex_realtime", "lookup_duplex_plugin", "startup_warmup_kind", + "warmup_silence_unit", ] diff --git a/vllm_omni/model_executor/common/__init__.py b/vllm_omni/model_executor/common/__init__.py new file mode 100644 index 00000000000..eff43e48d43 --- /dev/null +++ b/vllm_omni/model_executor/common/__init__.py @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Model-neutral helpers shared between model packages. + +``audio`` and ``request_outputs`` carry no duplex vocabulary and may be used +from any model. ``duplex`` builds on the duplex framework contracts and is +imported only from a model's ``duplex`` package. +""" diff --git a/vllm_omni/model_executor/common/audio/__init__.py b/vllm_omni/model_executor/common/audio/__init__.py new file mode 100644 index 00000000000..531f06a7218 --- /dev/null +++ b/vllm_omni/model_executor/common/audio/__init__.py @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Generic audio helpers shared between model packages.""" diff --git a/vllm_omni/model_executor/common/audio/pcm.py b/vllm_omni/model_executor/common/audio/pcm.py new file mode 100644 index 00000000000..a5cb0a3189f --- /dev/null +++ b/vllm_omni/model_executor/common/audio/pcm.py @@ -0,0 +1,54 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Little-endian float32 PCM (``pcm_f32le``) helpers. + +Plain audio plumbing every model that takes raw PCM needs: decode a base64 +chunk, check it is whole finite samples, count and materialise them. No +duplex vocabulary lives here. +""" + +from __future__ import annotations + +import binascii + +import numpy as np +import pybase64 as base64 + +PCM_F32LE_BYTES_PER_SAMPLE = 4 + + +def decode_pcm_f32le_base64(encoded: object, *, model: str = "audio") -> bytes: + """Decode base64 ``pcm_f32le`` into raw bytes, refusing partial or non-finite samples. + + ``model`` names the caller in error messages so a client sees which model + rejected its audio. + """ + if not isinstance(encoded, str): + raise ValueError(f"{model} audio must be base64 pcm_f32le") + try: + raw = base64.b64decode(encoded, validate=True) + except (binascii.Error, ValueError) as exc: + raise ValueError(f"{model} audio is not valid base64") from exc + if len(raw) % PCM_F32LE_BYTES_PER_SAMPLE: + raise ValueError(f"{model} pcm_f32le byte length must be divisible by four") + samples = np.frombuffer(raw, dtype=" int: + return len(raw) // PCM_F32LE_BYTES_PER_SAMPLE + + +def pcm_f32le_samples(raw: bytes) -> np.ndarray: + """The samples as a writable, contiguous float32 array (safe for zero-copy tensor views).""" + return np.ascontiguousarray(np.frombuffer(raw, dtype=" None: + self._encode_audio = encode_audio + self._requests: dict[str, _RequestCursor] = {} + + # ---- request / session bookkeeping ---- + + def begin_request(self, request_id: str) -> None: + self._requests.setdefault(request_id, _RequestCursor()).terminal = False + + def is_terminal(self, request_id: str | None) -> bool: + if request_id is None: + return False + state = self._requests.get(request_id) + return state is not None and state.terminal + + def mark_terminal(self, request_id: str) -> None: + self._requests.setdefault(request_id, _RequestCursor()).terminal = True + + def close_stream(self, request_id: str) -> None: + self._requests.pop(request_id, None) + + def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None: + if active_request_id is not None: + self._requests.pop(active_request_id, None) + for request_id in list(self._requests): + if duplex_resource_request_belongs_to_session(request_id, session_id): + self._requests.pop(request_id, None) + + # ---- projection ---- + + def project(self, result: object, *, context: object | None = None) -> Iterator[dict[str, object]]: + if not isinstance(result, dict): + return + outputs = result.get("data_plane_outputs") + if not isinstance(outputs, list): + return + if not isinstance(context, DuplexDataPlaneContext): + context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text")) + for output in outputs: + projected = self._project_output(output, context=context) + if projected is not None: + yield projected + + def _project_output(self, output: object, *, context: DuplexDataPlaneContext) -> dict[str, object] | None: + output, completion = unwrap_request_output(output) + request_id = getattr(output, "request_id", None) + if not isinstance(request_id, str) or not request_id: + request_id = None + state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor() + multimodal = multimodal_output(output, completion) + audio = audio_value(multimodal) + audio_delta = slice_audio_delta(audio, state.audio_samples) + total_samples = audio_sample_count(audio) + if total_samples is not None: + state.audio_samples = total_samples + + text = text_value(multimodal, completion) + delta_text = text_delta(text, state.text) + if text: + state.text = text + + rate = sample_rate_hz(multimodal, default=self.default_sample_rate_hz) + encoded = self._encode_audio(audio_delta, rate, context.response_format, context.speed) + if not encoded and not delta_text: + return None + delta_samples = audio_sample_count(audio_delta) or 0 + return { + "supported": True, + "stage_role": self.stage_role, + "is_listen": False, + "data_plane_request_id": request_id, + "text": delta_text, + "audio_data": encoded or "", + "audio_format": context.response_format, + "sample_rate_hz": rate, + "audio_duration_ms": round(delta_samples * 1000 / max(1, rate)), + "end_of_turn": False, + "uses_model_runner_scheduler": self.uses_model_runner_scheduler, + "runner_kv_backed": self.runner_kv_backed, + "runtime_impl": self.runtime_impl, + "owned_runtime": self.owned_runtime, + } + + +__all__ = ["CumulativeAudioTextDataPlane"] diff --git a/vllm_omni/model_executor/common/duplex/payload.py b/vllm_omni/model_executor/common/duplex/payload.py new file mode 100644 index 00000000000..2483ac49783 --- /dev/null +++ b/vllm_omni/model_executor/common/duplex/payload.py @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Validation of one duplex audio append payload. + +The runner hands a plugin the append as a mapping ``{format, sample_rate_hz, +audio}`` (``audio`` base64 ``pcm_f32le``). Models with a fixed input rate and +frame size validate it here; the PCM itself is decoded by +``vllm_omni.model_executor.common.audio.pcm``. +""" + +from __future__ import annotations + +from collections.abc import Mapping + +from vllm_omni.model_executor.common.audio.pcm import ( + decode_pcm_f32le_base64, + pcm_f32le_sample_count, +) + + +def payload_audio(payload: object) -> object | None: + """The encoded audio field of an append payload (``audio``, or the older ``data``).""" + if not isinstance(payload, Mapping): + return None + return payload.get("audio") or payload.get("data") + + +def decode_pcm_f32le_payload( + payload: object, + *, + sample_rate_hz: int, + exact_samples: int | None = None, + model: str = "duplex", +) -> bytes: + """Raw PCM of one append that must be ``pcm_f32le`` at ``sample_rate_hz``. + + ``exact_samples`` pins the unit length (a lockstep model takes exactly one + frame per append); ``None`` accepts any whole number of samples. + """ + if not isinstance(payload, Mapping): + raise ValueError(f"{model} duplex append payload must be a mapping") + if payload.get("format") != "pcm_f32le": + raise ValueError(f"{model} duplex append format must be pcm_f32le") + if payload.get("sample_rate_hz") != sample_rate_hz: + raise ValueError(f"{model} duplex append sample_rate_hz must be {sample_rate_hz}") + raw = decode_pcm_f32le_base64(payload_audio(payload), model=model) + if exact_samples is not None and pcm_f32le_sample_count(raw) != exact_samples: + raise ValueError(f"{model} duplex append must contain exactly {exact_samples} samples") + return raw + + +def payload_sample_count(payload: object) -> int | None: + """Sample count of a ``pcm_f32le`` append, or ``None`` when it is not one (lenient, for budgeting).""" + if not isinstance(payload, Mapping) or payload.get("format") != "pcm_f32le": + return None + try: + raw = decode_pcm_f32le_base64(payload_audio(payload)) + except ValueError: + return None + return pcm_f32le_sample_count(raw) + + +__all__ = ["decode_pcm_f32le_payload", "payload_audio", "payload_sample_count"] diff --git a/vllm_omni/model_executor/common/duplex/pcm_buffer.py b/vllm_omni/model_executor/common/duplex/pcm_buffer.py new file mode 100644 index 00000000000..aec8b557db8 --- /dev/null +++ b/vllm_omni/model_executor/common/duplex/pcm_buffer.py @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Transactional framing of client PCM into fixed model units. + +A frame-locked model (PersonaPlex: 1920 samples every 80 ms at 24 kHz) takes +exactly one frame per append. Clients send arbitrary chunks, so the session +runner buffers them here and takes one whole frame out at a time as a +*reservation*: committed once the stage accepted the append, rolled back +(the bytes go back to the front of the buffer) when it did not. +""" + +from __future__ import annotations + +import pybase64 as base64 + +from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation +from vllm_omni.model_executor.common.audio.pcm import ( + PCM_F32LE_BYTES_PER_SAMPLE, + decode_pcm_f32le_base64, +) + + +class FixedFramePcmAppendReservation(PcmAppendReservation): + __slots__ = ("_active", "_owner", "_raw", "operation_id", "payload") + + def __init__( + self, + *, + owner: FixedFramePcmAppendBuffer, + operation_id: str, + payload: dict[str, object] | None, + raw: bytes, + ) -> None: + self._owner = owner + self.operation_id = operation_id + self.payload = payload + self._raw = raw + self._active = True + + @property + def active(self) -> bool: + return self._active + + @property + def byte_count(self) -> int: + return len(self._raw) + + def commit(self) -> None: + self._owner._commit_reservation(self) + + def rollback(self) -> None: + self._owner._rollback_reservation(self) + + +class FixedFramePcmAppendBuffer(PcmAppendBuffer): + """Frame ``pcm_f32le`` at one sample rate into ``frame_samples`` units, transactionally.""" + + def __init__( + self, + *, + sample_rate_hz: int, + frame_samples: int, + chunk_period_ms: int, + model: str = "duplex", + ) -> None: + if sample_rate_hz <= 0 or frame_samples <= 0 or chunk_period_ms <= 0: + raise ValueError("FixedFramePcmAppendBuffer needs positive sample_rate_hz, frame_samples, chunk_period_ms") + self.sample_rate_hz = int(sample_rate_hz) + self.frame_samples = int(frame_samples) + self.chunk_period_ms = int(chunk_period_ms) + self.model = model + self._buffer = bytearray() + self._reservations: list[FixedFramePcmAppendReservation] = [] + + @property + def frame_bytes(self) -> int: + return self.frame_samples * PCM_F32LE_BYTES_PER_SAMPLE + + @property + def pending_byte_count(self) -> int: + return len(self._buffer) + + def has_pending(self) -> bool: + return bool(self._buffer) + + def has_reserved(self) -> bool: + return any(reservation.active for reservation in self._reservations) + + def clear_force_listen(self) -> None: + return + + def clear(self) -> None: + for reservation in self._reservations: + reservation._active = False + self._reservations.clear() + self._buffer.clear() + + def prepare_append( + self, + payload: dict[str, object], + *, + operation_id: str, + chunk_period_ms: int, + allow_emit: bool, + ) -> FixedFramePcmAppendReservation | None: + self._require_chunk_period(chunk_period_ms) + if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations): + raise ValueError(f"{self.model} duplicate active operation_id: {operation_id}") + self._buffer.extend(self._decode_payload(payload)) + if not allow_emit or len(self._buffer) < self.frame_bytes: + return None + return self._reserve_frame(payload, operation_id=operation_id, flush=False) + + def prepare_commit(self, *, operation_id: str, chunk_period_ms: int) -> FixedFramePcmAppendReservation: + self._require_chunk_period(chunk_period_ms) + if not self._buffer: + reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=None, raw=b"") + self._reservations.append(reservation) + return reservation + payload: dict[str, object] = { + "type": "audio", + "format": "pcm_f32le", + "sample_rate_hz": self.sample_rate_hz, + "audio": "", + "final": True, + } + return self._reserve_frame(payload, operation_id=operation_id, flush=True) + + def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None: + reservation = self.prepare_commit(operation_id=f"{self.model}-flush", chunk_period_ms=chunk_period_ms) + payload = reservation.payload + reservation.commit() + return payload + + # ------------------------------------------------------------------ # + # Internals # + # ------------------------------------------------------------------ # + + def _require_chunk_period(self, chunk_period_ms: int) -> None: + if chunk_period_ms != self.chunk_period_ms: + raise ValueError(f"{self.model} chunk_period_ms must be {self.chunk_period_ms}") + + def _decode_payload(self, payload: dict[str, object]) -> bytes: + if payload.get("format") != "pcm_f32le": + raise ValueError(f"{self.model} input format must be pcm_f32le") + if payload.get("sample_rate_hz") != self.sample_rate_hz: + raise ValueError(f"{self.model} sample_rate_hz must be {self.sample_rate_hz}") + return decode_pcm_f32le_base64(payload.get("audio"), model=self.model) + + def _reserve_frame( + self, + source_payload: dict[str, object], + *, + operation_id: str, + flush: bool, + ) -> FixedFramePcmAppendReservation: + frame_bytes = self.frame_bytes + consumed_bytes = min(len(self._buffer), frame_bytes) if flush else frame_bytes + raw = bytes(self._buffer[:consumed_bytes]) + del self._buffer[:consumed_bytes] + encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes) + payload = dict(source_payload) + payload["type"] = "audio" + payload["format"] = "pcm_f32le" + payload["sample_rate_hz"] = self.sample_rate_hz + payload["audio"] = base64.b64encode(encoded_raw).decode("ascii") + reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=payload, raw=raw) + self._reservations.append(reservation) + return reservation + + def _commit_reservation(self, reservation: FixedFramePcmAppendReservation) -> None: + if not reservation.active: + return + reservation._active = False + self._reservations.remove(reservation) + + def _rollback_reservation(self, reservation: FixedFramePcmAppendReservation) -> None: + if not reservation.active: + return + try: + index = self._reservations.index(reservation) + except ValueError: + reservation._active = False + return + # Later reservations were taken after this one: they go back too, in order. + rolled_back = self._reservations[index:] + restored = b"".join(item._raw for item in rolled_back if item.active) + self._buffer[:0] = restored + for item in rolled_back: + item._active = False + del self._reservations[index:] + + +__all__ = ["FixedFramePcmAppendBuffer", "FixedFramePcmAppendReservation"] diff --git a/vllm_omni/model_executor/common/request_outputs.py b/vllm_omni/model_executor/common/request_outputs.py new file mode 100644 index 00000000000..729768abc3a --- /dev/null +++ b/vllm_omni/model_executor/common/request_outputs.py @@ -0,0 +1,172 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""Reading a vLLM ``RequestOutput`` the way model-side projection code needs to. + +Stage outputs arrive as duck-typed objects (a ``RequestOutput``, an +``OmniRequestOutput`` wrapping one, or a test double): these helpers unwrap +them, read the ``multimodal_output`` mapping, coerce tensor scalars, and turn +cumulative audio/text into the delta since the last read. No duplex +vocabulary lives here. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import Any, cast + +import numpy as np + + +def unwrap_request_output(output: object) -> tuple[object, object | None]: + """Return ``(request_output, first_completion)`` for a stage output or a wrapper around one.""" + inner = getattr(output, "request_output", None) + if inner is not None and inner is not output: + output = inner + return output, first_completion(output) + + +def first_completion(output: object) -> object | None: + outputs = getattr(output, "outputs", None) + return outputs[0] if isinstance(outputs, list) and outputs else None + + +def multimodal_output(output: object, completion: object | None = None) -> dict[str, object]: + """The first non-empty ``multimodal_output`` mapping of the output or its completion, copied.""" + for candidate in ( + getattr(output, "multimodal_output", None), + getattr(completion, "multimodal_output", None) if completion is not None else None, + ): + if isinstance(candidate, Mapping) and candidate: + return dict(candidate) + return {} + + +def coerce_int(value: object) -> int | None: + """``int(value)`` for scalars, one-element tensors/arrays and numeric strings; ``None`` otherwise.""" + detach = getattr(value, "detach", None) + if callable(detach): + try: + flat = detach().cpu().reshape(-1) + if flat.numel() == 0: + return None + value = flat[0].item() + except (RuntimeError, TypeError, ValueError, IndexError): + return None + elif isinstance(value, np.ndarray): + if value.size == 0: + return None + value = value.reshape(-1)[0].item() + try: + return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item) + except (TypeError, ValueError): + return None + + +def coerce_int_list(value: object) -> list[int]: + if value is None: + return [] + if hasattr(value, "detach"): + try: + value = value.detach().cpu().reshape(-1).tolist() + except (RuntimeError, TypeError, ValueError): + return [] + elif isinstance(value, np.ndarray): + value = value.reshape(-1).tolist() + if not isinstance(value, (list, tuple)): + return [] + return [token_id for item in value if (token_id := coerce_int(item)) is not None] + + +def audio_value(multimodal: Mapping[str, object]) -> object | None: + """The audio carried by a ``multimodal_output`` (``audio`` / ``model_outputs`` / ``latent``).""" + value = next( + (multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal), + None, + ) + if isinstance(value, list) and len(value) == 1: + return value[0] + return value + + +def text_value(multimodal: Mapping[str, object], completion: object | None) -> str: + for candidate in ( + multimodal.get("text"), + multimodal.get("llm_output_text"), + getattr(completion, "text", None) if completion is not None else None, + ): + if isinstance(candidate, str) and candidate: + return candidate + return "" + + +def text_delta(text: str, previous: str) -> str: + """What ``text`` adds over ``previous`` when outputs are cumulative; the whole text on a restart.""" + if not text: + return "" + if text == previous: + return "" + if previous and text.startswith(previous): + return text[len(previous) :] + return text + + +def audio_sample_count(audio: object | None) -> int | None: + if audio is None: + return None + try: + import torch + + if isinstance(audio, torch.Tensor): + return int(audio.numel()) + except ImportError: + pass + try: + return int(np.asarray(audio, dtype=np.float32).size) + except (TypeError, ValueError): + return None + + +def slice_audio_delta(audio: object | None, offset: int) -> object | None: + """The samples of a cumulative ``audio`` past ``offset``; the whole audio when it restarted.""" + samples = audio_sample_count(audio) + if samples is None or samples <= 0: + return None + if offset <= 0 or samples < offset: + return audio + if samples == offset: + return None + try: + import torch + + if isinstance(audio, torch.Tensor): + return audio.reshape(-1)[offset:].contiguous() + except ImportError: + pass + return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:] + + +def sample_rate_hz(multimodal: Mapping[str, object], *, default: int) -> int: + value = multimodal.get("sr", multimodal.get("sample_rate_hz", default)) + if isinstance(value, list) and value: + value = value[0] + if hasattr(value, "item"): + try: + value = value.item() + except (RuntimeError, TypeError, ValueError): + value = default + return int(value) if isinstance(value, int | float) and value > 0 else default + + +__all__ = [ + "audio_sample_count", + "audio_value", + "coerce_int", + "coerce_int_list", + "first_completion", + "multimodal_output", + "sample_rate_hz", + "slice_audio_delta", + "text_delta", + "text_value", + "unwrap_request_output", +] diff --git a/vllm_omni/model_executor/models/minicpmo_4_5/duplex/data_plane.py b/vllm_omni/model_executor/models/minicpmo_4_5/duplex/data_plane.py index 7bf51abfd23..9fe0468e769 100644 --- a/vllm_omni/model_executor/models/minicpmo_4_5/duplex/data_plane.py +++ b/vllm_omni/model_executor/models/minicpmo_4_5/duplex/data_plane.py @@ -12,7 +12,7 @@ from vllm_omni.engine.duplex.contracts import ( duplex_resource_request_belongs_to_session, ) -from vllm_omni.engine.duplex.plugin import DuplexDataPlane +from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext from vllm_omni.outputs.duplex import get_duplex_output_decision logger = init_logger(__name__) @@ -21,17 +21,8 @@ @dataclass(frozen=True, slots=True) -class MiniCPMO45DataPlaneContext: - """Serving state needed to project one MiniCPM data-plane output.""" - - epoch: int = 0 - turn_id: int = 0 - active_response_turn_id: int | None = None - active_response_id: str | None = None - auto_responds: bool = False - response_format: str = "wav" - speed: float | None = None - modalities: tuple[str, ...] = () +class MiniCPMO45DataPlaneContext(DuplexDataPlaneContext): + """Serving state needed to project one MiniCPM data-plane output (the framework fields).""" @dataclass(slots=True) diff --git a/vllm_omni/model_executor/models/minicpmo_4_5/duplex/plugin.py b/vllm_omni/model_executor/models/minicpmo_4_5/duplex/plugin.py index 16203ecd578..b22e54f515e 100644 --- a/vllm_omni/model_executor/models/minicpmo_4_5/duplex/plugin.py +++ b/vllm_omni/model_executor/models/minicpmo_4_5/duplex/plugin.py @@ -18,7 +18,7 @@ from binascii import Error as BinasciiError from collections.abc import Mapping from copy import deepcopy -from typing import TYPE_CHECKING, Any, cast +from typing import TYPE_CHECKING import numpy as np from numpy.typing import NDArray @@ -40,6 +40,11 @@ reject_changed_runtime_value, reject_private_runtime_keys, ) +from vllm_omni.model_executor.common.duplex.payload import payload_sample_count as _duplex_pcm_sample_count +from vllm_omni.model_executor.common.request_outputs import coerce_int as _coerce_int +from vllm_omni.model_executor.common.request_outputs import coerce_int_list as _coerce_int_list +from vllm_omni.model_executor.common.request_outputs import first_completion as _first_completion +from vllm_omni.model_executor.common.request_outputs import multimodal_output as _multimodal_output from vllm_omni.model_executor.models.minicpmo_4_5.duplex.capabilities import ( minicpmo45_native_capabilities, ) @@ -56,7 +61,6 @@ ) if TYPE_CHECKING: - import torch from transformers import PreTrainedTokenizerBase from vllm.config import ModelConfig @@ -170,19 +174,6 @@ def _duplex_vision_tokens(payload: object, *, tile_pixels: int | None = None) -> return blocks * _DUPLEX_VISION_TOKENS_PER_FRAME -def _duplex_pcm_sample_count(payload: object) -> int | None: - if not isinstance(payload, dict): - return None - audio = payload.get("audio") or payload.get("data") - if payload.get("format") != "pcm_f32le" or not isinstance(audio, str): - return None - try: - raw = b64decode(audio, validate=True) - except (BinasciiError, ValueError): - return None - return len(raw) // 4 - - def duplex_payload_is_exact_chunks(payload: object) -> bool: sample_count = _duplex_pcm_sample_count(payload) return sample_count is not None and sample_count != 0 and sample_count % _DUPLEX_CHUNK_SAMPLES == 0 @@ -288,48 +279,6 @@ def build_duplex_data_plane_prompt( # ---- engine policy helpers: listen decision ---- -def _coerce_int(value: object) -> int | None: - detach = getattr(value, "detach", None) - if callable(detach): - try: - flat: torch.Tensor = detach().cpu().reshape(-1) - if flat.numel() == 0: - return None - value = flat[0].item() - except Exception: - return None - try: - return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item) - except (TypeError, ValueError): - return None - - -def _coerce_int_list(value: object) -> list[int]: - if value is None: - return [] - if hasattr(value, "detach"): - try: - value = value.detach().cpu().reshape(-1).tolist() - except Exception: - return [] - if not isinstance(value, (list, tuple)): - return [] - return [token_id for item in value if (token_id := _coerce_int(item)) is not None] - - -def _first_completion(output: object) -> object | None: - outputs = getattr(output, "outputs", None) - return outputs[0] if isinstance(outputs, list) and outputs else None - - -def _multimodal_output(output: object, completion: object | None) -> dict[str, object]: - metadata = getattr(output, "multimodal_output", None) - if isinstance(metadata, dict): - return metadata - metadata = getattr(completion, "multimodal_output", None) if completion is not None else None - return metadata if isinstance(metadata, dict) else {} - - def _special_token_ids(metadata: dict[str, object]) -> dict[str, int]: sources: list[object] = [metadata.get("special_token_ids"), metadata.get("meta")] sources.append( diff --git a/vllm_omni/model_executor/models/personaplex/duplex/__init__.py b/vllm_omni/model_executor/models/personaplex/duplex/__init__.py index d71a41c5d13..8e2439a7938 100644 --- a/vllm_omni/model_executor/models/personaplex/duplex/__init__.py +++ b/vllm_omni/model_executor/models/personaplex/duplex/__init__.py @@ -3,39 +3,22 @@ """PersonaPlex full-duplex integration. PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep -speech-to-speech model. This package plugs it into the generic duplex serving -stack through the standard plugin seams (``duplex_serving_adapter`` / -``duplex_runtime_extension`` dotted strings in the model's ``pipeline.py``): +speech-to-speech model. This package plugs it into the unified full-duplex +framework through the one seam the framework has, ``PipelineConfig.duplex_plugin``: -- :class:`PersonaPlexConfig` immutable session config (voice / persona / sampling) -- :class:`PersonaPlexServingRuntimeAdapter` the ``ServingRuntimeAdapter`` impl -- :class:`PersonaPlexDuplexRuntimeExtension` the engine ``DuplexRuntimeExtension`` -- :class:`PersonaPlexStage0DuplexRuntime` Stage 0 session state and prefill -- :class:`PersonaPlexPcmAppendBuffer` PCM input framing +- :class:`PersonaPlexDuplexPlugin` the ``DuplexModelPlugin`` (engine + session policy) +- :class:`PersonaPlexStage0DuplexRuntime` worker-side lockstep state and first-append prefill +- :class:`PersonaPlexPcmAppendBuffer` 80 ms PCM input framing """ -from .config import ( - PersonaPlexConfig, -) -from .input import ( - PersonaPlexPcmAppendBuffer, -) +from .input import PersonaPlexPcmAppendBuffer +from .plugin import PersonaPlexDuplexPlugin from .policy import PrefillStep -from .runtime_extension import ( - PersonaPlexDuplexRuntimeExtension, -) -from .serving_adapter import ( - PersonaPlexServingRuntimeAdapter, -) -from .stage0 import ( - PersonaPlexStage0DuplexRuntime, -) +from .stage0 import PersonaPlexStage0DuplexRuntime __all__ = [ - "PersonaPlexConfig", - "PersonaPlexDuplexRuntimeExtension", + "PersonaPlexDuplexPlugin", "PersonaPlexPcmAppendBuffer", - "PersonaPlexServingRuntimeAdapter", "PersonaPlexStage0DuplexRuntime", "PrefillStep", ] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/capabilities.py b/vllm_omni/model_executor/models/personaplex/duplex/capabilities.py new file mode 100644 index 00000000000..df7d38f93ed --- /dev/null +++ b/vllm_omni/model_executor/models/personaplex/duplex/capabilities.py @@ -0,0 +1,55 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project + +from __future__ import annotations + +from vllm_omni.engine.duplex.config import DuplexCapabilities +from vllm_omni.model_executor.models.personaplex.duplex.config import CHUNK_PERIOD_MS + + +def personaplex_capabilities(*, max_sessions: int = 1) -> DuplexCapabilities: + """What a PersonaPlex session advertises in ``session.created``. + + PersonaPlex is a pure-lockstep model: audio flows both ways at the codec + frame rate, the model itself decides when to speak, and there are no + client commits or external turn signals. Barge-in is native model + behaviour but destructive output interruption and model-state rewind are + not validated, so it is not advertised. No text seeding exists, so the + model cannot serve ``/v1/chat/completions``. + """ + supports_multi_session = max_sessions > 1 + return DuplexCapabilities( + supports_model_native_turn_policy=True, + supports_external_turn_signal=False, + supports_client_commit=False, + supports_barge_in=False, + supports_playback_ack=True, + supports_input_append=True, + supports_replace_latest_chunk=False, + supports_reencode_context=False, + supports_rollback_to_checkpoint=False, + supports_turn_commit_only=False, + supports_model_internal_state=True, + supports_stage_resumption=True, + supports_core_resumable_request=True, + supports_stage_connector_handoff=True, + supports_independent_io_streams=True, + supports_realtime_endpoint=True, + supports_multi_session=supports_multi_session, + supports_multi_session_same_replica=supports_multi_session, + supports_session_lease=True, + supports_session_resume=False, + session_admission_mode="engine_managed", + supports_audio_truncate=False, + supports_chat_completions=False, + requires_model_runner_kv=True, + requires_native_stage_role=True, + adapter_patterns=["scheduler_data_plane"], + signal_sources=["model_native", "client_event"], + stage_handoff_transport="scheduler_data_plane", + chunk_period_ms=CHUNK_PERIOD_MS, + target_barge_in_latency_ms=None, + ) + + +__all__ = ["personaplex_capabilities"] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/config.py b/vllm_omni/model_executor/models/personaplex/duplex/config.py index 13f807b3e11..fb27e7b5269 100644 --- a/vllm_omni/model_executor/models/personaplex/duplex/config.py +++ b/vllm_omni/model_executor/models/personaplex/duplex/config.py @@ -1,64 +1,24 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project -"""Configuration for the PersonaPlex full-duplex backend. +"""Constants of the PersonaPlex full-duplex integration. PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep full-duplex speech-to-speech model running at the Mimi codec frame rate -(12.5 Hz / 80 ms). One config drives both the offline driver and the duplex -adapter. Defaults mirror the PersonaPlex reference loop. +(12.5 Hz / 80 ms). The values are fixed by the pretrained checkpoint; they +live here so the rest of the package never hard-codes them. """ from __future__ import annotations -from dataclasses import dataclass - -# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE). Fixed by the -# pretrained checkpoint; exposed here only so the rest of the package never -# hard-codes them. +# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE). SAMPLE_RATE = 24000 FRAME_RATE = 12.5 FRAME_SIZE = int(SAMPLE_RATE / FRAME_RATE) # 1920 samples per 80 ms frame +CHUNK_PERIOD_MS = int(round(1000 / FRAME_RATE)) # 80 # Default assistant persona shipped with PersonaPlex (offline.py:335). DEFAULT_PERSONA = "You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way." +# Bundled voice prompt used when a session names none. +DEFAULT_VOICE = "NATF2.pt" - -@dataclass(frozen=True) -class PersonaPlexConfig: - """Immutable session configuration for a PersonaPlex conversation. - - Attributes: - hf_repo: HuggingFace repo holding the weights, Mimi codec and tokenizer. - voice_prompt: Voice-clone reference. Either a bundled basename - (``"NATF2.pt"`` / ``"NATM1.pt"`` from ``voices.tgz``) or a path to a - ``.pt`` embedding bundle or a reference ``.wav``. - persona: System role text; injected as `` ... `` into the - inner-monologue stream at session start. - device: Torch device for the backend (``"cuda"`` / ``"cpu"``). - cpu_offload: Offload LM layers to CPU when GPU memory is tight (needs - ``accelerate``). - batch_size: Concurrent conversation slots sharing one engine. ``1`` is - the single-session path; ``> 1`` enables elastic batching with - per-slot recycle for new callers. - - Note: the native stepper decodes greedily (argmax) for both the text head - and the depformer, so there are no sampling knobs here yet. Temperature / - top-k / seed fields will be added if and when a sampling path is wired in. - """ - - hf_repo: str = "nvidia/personaplex-7b-v1" - voice_prompt: str = "NATF2.pt" - persona: str = DEFAULT_PERSONA - - device: str = "cuda" - cpu_offload: bool = False - - batch_size: int = 1 - - @property - def sample_rate(self) -> int: - return SAMPLE_RATE - - @property - def frame_size(self) -> int: - return FRAME_SIZE +__all__ = ["CHUNK_PERIOD_MS", "DEFAULT_PERSONA", "DEFAULT_VOICE", "FRAME_RATE", "FRAME_SIZE", "SAMPLE_RATE"] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/data_plane.py b/vllm_omni/model_executor/models/personaplex/duplex/data_plane.py index c6c293c177f..9dba81f4176 100644 --- a/vllm_omni/model_executor/models/personaplex/duplex/data_plane.py +++ b/vllm_omni/model_executor/models/personaplex/duplex/data_plane.py @@ -1,228 +1,17 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""PersonaPlex output projection: cumulative Code2Wav audio and inner-monologue text into deltas.""" from __future__ import annotations -from collections.abc import Callable, Iterator, Mapping -from dataclasses import dataclass +from vllm_omni.model_executor.common.duplex.data_plane import CumulativeAudioTextDataPlane +from vllm_omni.model_executor.models.personaplex.duplex.config import SAMPLE_RATE -import numpy as np -from vllm_omni.engine.duplex.contracts import ( - duplex_resource_request_belongs_to_session, -) +class PersonaPlexDataPlaneSession(CumulativeAudioTextDataPlane): + """Project cumulative staged PersonaPlex output into Realtime deltas (24 kHz by default).""" -EncodeAudio = Callable[[object, int, str, float | None], str | None] + default_sample_rate_hz = SAMPLE_RATE -@dataclass(frozen=True, slots=True) -class PersonaPlexDataPlaneContext: - epoch: int = 0 - turn_id: int = 0 - active_response_turn_id: int | None = None - active_response_id: str | None = None - auto_responds: bool = True - response_format: str = "wav" - speed: float | None = None - modalities: tuple[str, ...] = ("audio", "text") - - -@dataclass(slots=True) -class _RequestCursor: - audio_samples: int = 0 - text: str = "" - terminal: bool = False - - -class PersonaPlexDataPlaneSession: - """Project cumulative staged PersonaPlex output into Realtime deltas.""" - - def __init__(self, encode_audio: EncodeAudio) -> None: - self._encode_audio = encode_audio - self._requests: dict[str, _RequestCursor] = {} - - def begin_request(self, request_id: str) -> None: - self._requests.setdefault(request_id, _RequestCursor()).terminal = False - - def is_terminal(self, request_id: str | None) -> bool: - if request_id is None: - return False - state = self._requests.get(request_id) - return state is not None and state.terminal - - def mark_terminal(self, request_id: str) -> None: - self._requests.setdefault(request_id, _RequestCursor()).terminal = True - - def close_stream(self, request_id: str) -> None: - self._requests.pop(request_id, None) - - def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None: - if active_request_id is not None: - self._requests.pop(active_request_id, None) - for request_id in list(self._requests): - if duplex_resource_request_belongs_to_session(request_id, session_id): - self._requests.pop(request_id, None) - - def project( - self, - result: object, - *, - context: PersonaPlexDataPlaneContext | None = None, - ) -> Iterator[dict[str, object]]: - if not isinstance(result, dict): - return - outputs = result.get("data_plane_outputs") - if not isinstance(outputs, list): - return - context = context or PersonaPlexDataPlaneContext() - for output in outputs: - projected = self._project_output(output, context=context) - if projected is not None: - yield projected - - def _project_output( - self, - output: object, - *, - context: PersonaPlexDataPlaneContext, - ) -> dict[str, object] | None: - output, completion = _unwrap_output(output) - request_id = getattr(output, "request_id", None) - if not isinstance(request_id, str) or not request_id: - request_id = None - state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor() - multimodal = _multimodal_output(output, completion) - audio = _audio_value(multimodal) - audio_delta = _slice_audio(audio, state.audio_samples) - audio_samples = _num_samples(audio) - if audio_samples is not None: - state.audio_samples = audio_samples - - text = _text_value(multimodal, completion) - text_delta = _text_delta(text, state.text) - if text: - state.text = text - - sample_rate_hz = _sample_rate(multimodal) - encoded = self._encode_audio(audio_delta, sample_rate_hz, context.response_format, context.speed) - if not encoded and not text_delta: - return None - delta_samples = _num_samples(audio_delta) or 0 - return { - "supported": True, - "stage_role": "tts", - "is_listen": False, - "data_plane_request_id": request_id, - "text": text_delta, - "audio_data": encoded or "", - "audio_format": context.response_format, - "sample_rate_hz": sample_rate_hz, - "audio_duration_ms": round(delta_samples * 1000 / max(1, sample_rate_hz)), - "end_of_turn": False, - "uses_model_runner_scheduler": True, - "runner_kv_backed": True, - "runtime_impl": "scheduler_data_plane", - "owned_runtime": False, - } - - -def _unwrap_output(output: object) -> tuple[object, object | None]: - inner = getattr(output, "request_output", None) - if inner is not None and inner is not output: - output = inner - outputs = getattr(output, "outputs", None) - completion = outputs[0] if isinstance(outputs, list) and outputs else None - return output, completion - - -def _multimodal_output(output: object, completion: object | None) -> dict[str, object]: - for candidate in ( - getattr(output, "multimodal_output", None), - getattr(completion, "multimodal_output", None) if completion is not None else None, - ): - if isinstance(candidate, Mapping) and candidate: - return dict(candidate) - return {} - - -def _audio_value(multimodal: dict[str, object]) -> object | None: - value = next( - (multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal), - None, - ) - if isinstance(value, list) and len(value) == 1: - return value[0] - return value - - -def _text_value(multimodal: dict[str, object], completion: object | None) -> str: - for candidate in ( - multimodal.get("text"), - multimodal.get("llm_output_text"), - getattr(completion, "text", None) if completion is not None else None, - ): - if isinstance(candidate, str) and candidate: - return candidate - return "" - - -def _text_delta(text: str, previous: str) -> str: - if not text: - return "" - if text == previous: - return "" - if previous and text.startswith(previous): - return text[len(previous) :] - return text - - -def _slice_audio(audio: object | None, offset: int) -> object | None: - samples = _num_samples(audio) - if samples is None or samples <= 0: - return None - if offset <= 0 or samples < offset: - return audio - if samples == offset: - return None - try: - import torch - - if isinstance(audio, torch.Tensor): - return audio.reshape(-1)[offset:].contiguous() - except Exception: - pass - return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:] - - -def _num_samples(audio: object | None) -> int | None: - if audio is None: - return None - try: - import torch - - if isinstance(audio, torch.Tensor): - return int(audio.numel()) - except Exception: - pass - try: - return int(np.asarray(audio, dtype=np.float32).size) - except Exception: - return None - - -def _sample_rate(multimodal: dict[str, object]) -> int: - value = multimodal.get("sr", multimodal.get("sample_rate_hz", 24000)) - if isinstance(value, list) and value: - value = value[0] - if hasattr(value, "item"): - try: - value = value.item() - except Exception: - value = 24000 - return int(value) if isinstance(value, int | float) else 24000 - - -__all__ = [ - "PersonaPlexDataPlaneContext", - "PersonaPlexDataPlaneSession", -] +__all__ = ["PersonaPlexDataPlaneSession"] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/input.py b/vllm_omni/model_executor/models/personaplex/duplex/input.py index e78d6b707cf..d568a49fe24 100644 --- a/vllm_omni/model_executor/models/personaplex/duplex/input.py +++ b/vllm_omni/model_executor/models/personaplex/duplex/input.py @@ -1,213 +1,27 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""PCM input framing for PersonaPlex: 24 kHz ``pcm_f32le`` into 1920-sample (80 ms) units.""" from __future__ import annotations -import base64 -import binascii +from vllm_omni.model_executor.common.duplex.pcm_buffer import FixedFramePcmAppendBuffer +from vllm_omni.model_executor.models.personaplex.duplex.config import ( + CHUNK_PERIOD_MS, + FRAME_SIZE, + SAMPLE_RATE, +) -import numpy as np -_SAMPLE_RATE_HZ = 24000 -_FRAME_SAMPLES = 1920 -_BYTES_PER_SAMPLE = 4 - - -class PersonaPlexPcmAppendReservation: - __slots__ = ( - "_active", - "_owner", - "_raw", - "operation_id", - "payload", - ) - - def __init__( - self, - *, - owner: PersonaPlexPcmAppendBuffer, - operation_id: str, - payload: dict[str, object] | None, - raw: bytes, - ) -> None: - self._owner = owner - self.operation_id = operation_id - self.payload = payload - self._raw = raw - self._active = True - - @property - def active(self) -> bool: - return self._active - - @property - def byte_count(self) -> int: - return len(self._raw) - - def commit(self) -> None: - self._owner._commit_reservation(self) - - def rollback(self) -> None: - self._owner._rollback_reservation(self) - - -class PersonaPlexPcmAppendBuffer: +class PersonaPlexPcmAppendBuffer(FixedFramePcmAppendBuffer): """Transactionally frame 24 kHz float PCM into PersonaPlex 80 ms units.""" def __init__(self) -> None: - self._buffer = bytearray() - self._sample_rate_hz: int | None = None - self._reservations: list[PersonaPlexPcmAppendReservation] = [] - - @property - def pending_byte_count(self) -> int: - return len(self._buffer) - - def has_pending(self) -> bool: - return bool(self._buffer) - - def has_reserved(self) -> bool: - return any(reservation.active for reservation in self._reservations) - - def clear_force_listen(self) -> None: - return - - def clear(self) -> None: - for reservation in self._reservations: - reservation._active = False - self._reservations.clear() - self._buffer.clear() - self._sample_rate_hz = None - - def prepare_append( - self, - payload: dict[str, object], - *, - operation_id: str, - chunk_period_ms: int, - allow_emit: bool, - ) -> PersonaPlexPcmAppendReservation | None: - if chunk_period_ms != 80: - raise ValueError("PersonaPlex chunk_period_ms must be 80") - if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations): - raise ValueError(f"PersonaPlex duplicate active operation_id: {operation_id}") - - raw = self._decode_payload(payload) - self._buffer.extend(raw) - if not allow_emit or len(self._buffer) < _FRAME_SAMPLES * _BYTES_PER_SAMPLE: - return None - return self._reserve_frame(payload, operation_id=operation_id, flush=False) - - def prepare_commit( - self, - *, - operation_id: str, - chunk_period_ms: int, - ) -> PersonaPlexPcmAppendReservation: - if chunk_period_ms != 80: - raise ValueError("PersonaPlex chunk_period_ms must be 80") - if not self._buffer: - reservation = PersonaPlexPcmAppendReservation( - owner=self, - operation_id=operation_id, - payload=None, - raw=b"", - ) - self._reservations.append(reservation) - return reservation - payload: dict[str, object] = { - "type": "audio", - "format": "pcm_f32le", - "sample_rate_hz": self._sample_rate_hz or _SAMPLE_RATE_HZ, - "audio": "", - "final": True, - } - return self._reserve_frame(payload, operation_id=operation_id, flush=True) - - def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None: - reservation = self.prepare_commit( - operation_id="personaplex-flush", - chunk_period_ms=chunk_period_ms, + super().__init__( + sample_rate_hz=SAMPLE_RATE, + frame_samples=FRAME_SIZE, + chunk_period_ms=CHUNK_PERIOD_MS, + model="PersonaPlex", ) - payload = reservation.payload - reservation.commit() - return payload - - def _decode_payload(self, payload: dict[str, object]) -> bytes: - if payload.get("format") != "pcm_f32le": - raise ValueError("PersonaPlex input format must be pcm_f32le") - sample_rate_hz = payload.get("sample_rate_hz") - if sample_rate_hz != _SAMPLE_RATE_HZ: - raise ValueError("PersonaPlex sample_rate_hz must be 24000") - if self._sample_rate_hz is not None and self._sample_rate_hz != sample_rate_hz: - raise ValueError("PersonaPlex sample_rate_hz changed within a session") - audio = payload.get("audio") - if not isinstance(audio, str): - raise ValueError("PersonaPlex audio must be base64 pcm_f32le") - try: - raw = base64.b64decode(audio, validate=True) - except (binascii.Error, ValueError) as exc: - raise ValueError("PersonaPlex audio is not valid base64") from exc - if len(raw) % _BYTES_PER_SAMPLE: - raise ValueError("PersonaPlex pcm_f32le byte length must be divisible by four") - samples = np.frombuffer(raw, dtype=" PersonaPlexPcmAppendReservation: - frame_bytes = _FRAME_SAMPLES * _BYTES_PER_SAMPLE - if flush: - consumed_bytes = min(len(self._buffer), frame_bytes) - else: - consumed_bytes = frame_bytes - raw = bytes(self._buffer[:consumed_bytes]) - del self._buffer[:consumed_bytes] - encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes) - payload = dict(source_payload) - payload["type"] = "audio" - payload["format"] = "pcm_f32le" - payload["sample_rate_hz"] = _SAMPLE_RATE_HZ - payload["audio"] = base64.b64encode(encoded_raw).decode("ascii") - reservation = PersonaPlexPcmAppendReservation( - owner=self, - operation_id=operation_id, - payload=payload, - raw=raw, - ) - self._reservations.append(reservation) - return reservation - - def _commit_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None: - if not reservation.active: - return - reservation._active = False - self._reservations.remove(reservation) - - def _rollback_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None: - if not reservation.active: - return - try: - index = self._reservations.index(reservation) - except ValueError: - reservation._active = False - return - rolled_back = self._reservations[index:] - restored = b"".join(item._raw for item in rolled_back if item.active) - self._buffer[:0] = restored - for item in rolled_back: - item._active = False - del self._reservations[index:] -__all__ = [ - "PersonaPlexPcmAppendBuffer", - "PersonaPlexPcmAppendReservation", -] +__all__ = ["PersonaPlexPcmAppendBuffer"] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/plugin.py b/vllm_omni/model_executor/models/personaplex/duplex/plugin.py new file mode 100644 index 00000000000..1d2fbac90af --- /dev/null +++ b/vllm_omni/model_executor/models/personaplex/duplex/plugin.py @@ -0,0 +1,269 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project +"""PersonaPlex full-duplex model plugin: engine policy and session policy in one class. + +PersonaPlex is a pure-lockstep model. Engine policy is one 80 ms frame per +resumable Stage 0 append with greedy one-token decoding; session policy is +the bundled voice prompt and the persona text, resolved once at open into the +server-owned runtime configuration the worker replays as its prefill. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Mapping +from copy import deepcopy +from dataclasses import dataclass, field +from pathlib import PurePath +from typing import TYPE_CHECKING + +from vllm.sampling_params import SamplingParams + +from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig +from vllm_omni.engine.duplex.contracts import ( + DuplexAppendPlan, + DuplexFence, + DuplexOutputDecision, +) +from vllm_omni.engine.duplex.plugin import ( + DefaultDuplexModelSessionState, + DuplexModelPlugin, + DuplexRuntimeConfigError, + EncodeAudio, + reject_changed_runtime_value, +) +from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload +from vllm_omni.model_executor.models.personaplex.duplex.capabilities import personaplex_capabilities +from vllm_omni.model_executor.models.personaplex.duplex.config import ( + DEFAULT_PERSONA, + DEFAULT_VOICE, + FRAME_SIZE, + SAMPLE_RATE, +) +from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession +from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer + +if TYPE_CHECKING: + from vllm.config import ModelConfig + +#: Runtime-config keys the server derives from the session fields; a client +#: cannot set them through ``extra_body``. +PRIVATE_RUNTIME_CONFIG_KEYS = frozenset( + { + "personaplex_prefill_slots", + "personaplex_model_path", + "personaplex_voice_prompt", + "personaplex_persona", + } +) + + +@dataclass +class PersonaPlexSessionState(DefaultDuplexModelSessionState): + """Per-session model state: the framework flags plus the 80 ms framing buffer.""" + + audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer) + + +def voice_name(value: object) -> str: + """A bundled ``.pt`` basename (``NATF2.pt``); paths are refused so the worker only reads the checkpoint.""" + voice = value if isinstance(value, str) and value else DEFAULT_VOICE + path = PurePath(voice) + if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts): + raise DuplexRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename", code="invalid_voice") + return voice + + +def persona_text(value: object) -> str: + return str(value) if isinstance(value, str) and value else DEFAULT_PERSONA + + +def prefill_slot_count(runtime_config: Mapping[str, object]) -> int: + raw = runtime_config.get("personaplex_prefill_slots", 0) + try: + return max(0, int(raw)) # type: ignore[arg-type] + except (TypeError, ValueError) as exc: + raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc + + +class PersonaPlexDuplexPlugin(DuplexModelPlugin): + """PersonaPlex-owned sampling policy, append planning, session state and output projection.""" + + plugin_id = "personaplex" + private_runtime_config_keys = PRIVATE_RUNTIME_CONFIG_KEYS + #: One codec frame: the runner keeps a model turn clocked with these when the client pauses. + silence_continuation_samples = FRAME_SIZE + silence_continuation_sample_rate_hz = SAMPLE_RATE + + def __init__(self, encode_audio: EncodeAudio) -> None: + super().__init__(encode_audio) + self.data_plane = PersonaPlexDataPlaneSession(encode_audio) + # The prefill length depends only on (model, voice, persona); the + # voice bundle and tokenizer are read once per distinct triple. + self._prefill_slots: dict[tuple[str, str, str], int] = {} + self._prefill_lock = asyncio.Lock() + + # ---- engine policy (the resumable Stage 0 request) ---- + + def configure_sampling_params( + self, + *, + runtime_config: dict[str, object], + defaults: tuple[object, ...], + ) -> tuple[object, ...]: + del runtime_config + if not defaults: + return defaults + configured = list(defaults) + stage0 = defaults[0] + if isinstance(stage0, SamplingParams): + # Greedy, one temporal token per frame: the native stepper takes + # argmax for the text head and the depformer. + stage0 = stage0.clone() + stage0.temperature = 0.0 + stage0.top_k = 1 + stage0.max_tokens = 1 + configured[0] = stage0 + return tuple(configured) + + def plan_append( + self, + *, + request_id: str, + fence: DuplexFence, + session_config: dict[str, object], + runtime_config: dict[str, object], + seq: int, + turn_seq: int, + payload: object, + final: bool, + sampling_params: object, + ) -> DuplexAppendPlan: + del sampling_params + decode_pcm_f32le_payload(payload, sample_rate_hz=SAMPLE_RATE, exact_samples=FRAME_SIZE, model="PersonaPlex") + normalized_payload = dict(payload) # type: ignore[call-overload] + # One scheduler slot per frame; the first append of an epoch also + # carries the voice/persona prefill (a new epoch is a new Stage 0 + # request with fresh KV, so the worker replays it). + prompt_slots = 1 + (prefill_slot_count(runtime_config) if seq <= 1 else 0) + return DuplexAppendPlan( + prompt={ + "prompt_token_ids": [0] * prompt_slots, + "model_intermediate_buffer": { + "request_id": request_id, + "global_request_id": [fence.session_id], + "duplex": { + "data_plane": True, + "fence": fence, + "session_id": fence.session_id, + "epoch": fence.epoch, + "turn_id": fence.turn_id, + "seq": seq, + "turn_seq": turn_seq, + "mode": "append_audio_chunk", + "payload": normalized_payload, + "final": final, + "session_config": dict(session_config), + "runtime_config": dict(runtime_config), + "scheduler_token_budget": prompt_slots, + }, + }, + } + ) + + def decide_output( + self, + *, + stage_id: int, + final_stage_id: int, + segment_finished: bool, + segment_token_ids: tuple[int, ...], + segment_output_metadata: dict[str, object], + output: object, + ) -> DuplexOutputDecision | None: + # Always-clocked model: what the client hears comes from the final + # stage data plane, never from a Stage 0 listen/speak decision. + del stage_id, final_stage_id, segment_finished, segment_token_ids, segment_output_metadata, output + return None + + # ---- session policy ---- + + def create_session_state(self) -> PersonaPlexSessionState: + return PersonaPlexSessionState() + + def capabilities(self, *, max_sessions: int) -> DuplexCapabilities: + return personaplex_capabilities(max_sessions=max_sessions) + + async def prepare_runtime_config( + self, config: DuplexSessionConfig, *, model_config: ModelConfig | None + ) -> dict[str, object]: + self.validate_client_extra_body(config.extra_body) + model_path = getattr(model_config, "model", None) + if not isinstance(model_path, str) or not model_path: + raise DuplexRuntimeConfigError("PersonaPlex model path is unavailable", code="model_path_unavailable") + voice = voice_name(config.voice) + persona = persona_text(config.instructions) + prefill_slots = await self._prefill_slots_for(model_path, voice, persona) + return { + "personaplex_model_path": model_path, + "personaplex_voice_prompt": voice, + "personaplex_persona": persona, + "personaplex_prefill_slots": prefill_slots, + } + + async def _prefill_slots_for(self, model_path: str, voice: str, persona: str) -> int: + key = (model_path, voice, persona) + cached = self._prefill_slots.get(key) + if cached is not None: + return cached + from vllm_omni.model_executor.models.personaplex.duplex.stage0 import personaplex_prefill_slots + + async with self._prefill_lock: + cached = self._prefill_slots.get(key) + if cached is not None: + return cached + try: + # Reads the voice bundle and the tokenizer: off the orchestrator loop. + slots = await asyncio.to_thread(personaplex_prefill_slots, model_path, voice, persona) + except Exception as exc: + raise DuplexRuntimeConfigError( + f"PersonaPlex voice/persona prefill could not be prepared: {exc}", + code="prefill_unavailable", + ) from exc + self._prefill_slots[key] = slots + return slots + + def runtime_config_for_update( + self, + config: DuplexSessionConfig, + current: Mapping[str, object], + ) -> dict[str, object]: + self.validate_client_extra_body(config.extra_body) + new_persona = persona_text(config.instructions) + reject_changed_runtime_value( + new_persona, + current.get("personaplex_persona"), + message="PersonaPlex persona (instructions) cannot be changed after the session is created", + code="persona_update_unsupported", + ) + new_voice = voice_name(config.voice) + reject_changed_runtime_value( + new_voice, + current.get("personaplex_voice_prompt"), + message="PersonaPlex voice cannot be changed after the session is created", + code="voice_update_unsupported", + ) + runtime_config = deepcopy(dict(current)) + runtime_config["personaplex_voice_prompt"] = new_voice + runtime_config["personaplex_persona"] = new_persona + return runtime_config + + +__all__ = [ + "PRIVATE_RUNTIME_CONFIG_KEYS", + "PersonaPlexDuplexPlugin", + "PersonaPlexSessionState", + "persona_text", + "prefill_slot_count", + "voice_name", +] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/runtime_extension.py b/vllm_omni/model_executor/models/personaplex/duplex/runtime_extension.py deleted file mode 100644 index a30ac41b94f..00000000000 --- a/vllm_omni/model_executor/models/personaplex/duplex/runtime_extension.py +++ /dev/null @@ -1,129 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project - -from __future__ import annotations - -import base64 -import binascii -from typing import Any - -from vllm.sampling_params import SamplingParams - -from vllm_omni.engine.duplex.messages import DuplexFence -from vllm_omni.engine.duplex.runtime import ( - DuplexAppendPlan, - DuplexInputMode, - DuplexOutputDecision, -) - -_FRAME_BYTES = 1920 * 4 - - -def _validated_frame_payload(payload: object) -> dict[str, object]: - if not isinstance(payload, dict): - raise ValueError("PersonaPlex duplex append payload must be a mapping") - if payload.get("format") != "pcm_f32le": - raise ValueError("PersonaPlex duplex append format must be pcm_f32le") - if payload.get("sample_rate_hz") != 24000: - raise ValueError("PersonaPlex duplex append sample_rate_hz must be 24000") - audio = payload.get("audio") - if not isinstance(audio, str): - raise ValueError("PersonaPlex duplex append audio must be base64 pcm_f32le") - try: - raw = base64.b64decode(audio, validate=True) - except (binascii.Error, ValueError) as exc: - raise ValueError("PersonaPlex duplex append audio is not valid base64") from exc - if len(raw) != _FRAME_BYTES: - raise ValueError("PersonaPlex duplex append must contain exactly 1920 samples") - return dict(payload) - - -class PersonaPlexDuplexRuntimeExtension: - """PersonaPlex policy for the engine-owned resumable duplex request.""" - - def configure_sampling_params( - self, - *, - runtime_config: dict[str, Any], - defaults: tuple[object, ...], - ) -> tuple[object, ...]: - del runtime_config - if not defaults: - return defaults - configured = list(defaults) - stage0 = defaults[0] - if isinstance(stage0, SamplingParams): - stage0 = stage0.clone() - stage0.temperature = 0.0 - stage0.top_k = 1 - stage0.max_tokens = 1 - configured[0] = stage0 - return tuple(configured) - - def plan_append( - self, - *, - request_id: str, - fence: DuplexFence, - session_config: dict[str, Any], - runtime_config: dict[str, Any], - seq: int, - turn_seq: int, - mode: DuplexInputMode, - payload: object, - final: bool, - sampling_params: object, - ) -> DuplexAppendPlan: - del sampling_params - if mode is not DuplexInputMode.APPEND_AUDIO_CHUNK: - raise ValueError(f"PersonaPlex does not support duplex input mode {mode.value!r}") - normalized_payload = _validated_frame_payload(payload) - prefill_slots = runtime_config.get("personaplex_prefill_slots", 0) - try: - prefill_slots = max(0, int(prefill_slots)) - except (TypeError, ValueError) as exc: - raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc - prompt_slots = 1 + (prefill_slots if seq <= 1 else 0) - return DuplexAppendPlan( - prompt={ - "prompt_token_ids": [0] * prompt_slots, - "model_intermediate_buffer": { - "request_id": request_id, - "global_request_id": [fence.session_id], - "duplex": { - "data_plane": True, - "fence": fence, - "session_id": fence.session_id, - "incarnation": fence.incarnation, - "epoch": fence.epoch, - "turn_id": fence.turn_id, - "response_seq": fence.response_seq, - "seq": seq, - "turn_seq": turn_seq, - "mode": mode.value, - "payload": normalized_payload, - "final": final, - "session_config": dict(session_config), - "runtime_config": dict(runtime_config), - "scheduler_token_budget": prompt_slots, - }, - }, - } - ) - - def decide_output( - self, - *, - stage_id: int, - final_stage_id: int, - segment_finished: bool, - segment_token_ids: tuple[int, ...], - segment_output_metadata: dict[str, Any], - output: object, - ) -> DuplexOutputDecision | None: - del stage_id, final_stage_id, segment_finished - del segment_token_ids, segment_output_metadata, output - return None - - -__all__ = ["PersonaPlexDuplexRuntimeExtension"] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py b/vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py deleted file mode 100644 index d27360fd055..00000000000 --- a/vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py +++ /dev/null @@ -1,253 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project - -from __future__ import annotations - -import asyncio -from collections.abc import Awaitable, Callable, Mapping -from copy import deepcopy -from dataclasses import dataclass, field -from pathlib import PurePath -from typing import Any - -from vllm_omni.entrypoints.duplex.protocol import ( - DuplexCapabilities, -) -from vllm_omni.entrypoints.duplex.runtime_adapter import ( - ServingRuntimeConfigError, - reject_changed_runtime_value, -) -from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA -from vllm_omni.model_executor.models.personaplex.duplex.data_plane import ( - PersonaPlexDataPlaneContext, - PersonaPlexDataPlaneSession, -) -from vllm_omni.model_executor.models.personaplex.duplex.input import ( - PersonaPlexPcmAppendBuffer, -) - -EncodeAudio = Callable[[object, int, str, float | None], str | None] - -_PRIVATE_RUNTIME_CONFIG_KEYS = frozenset( - { - "personaplex_prefill_slots", - "personaplex_model_path", - # PersonaPlex is always model-native; the per-session opt-in knob - # (canonical name and its deprecated alias) is server-owned here. - "native_duplex", - "minicpmo45_native_duplex", - } -) - - -@dataclass(slots=True) -class PersonaPlexServingSessionState: - audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer) - input_since_commit: bool = False - speech_since_commit: bool = False - native_context_locked: bool = False - committed_audio_payload: dict[str, object] | None = None - committed_audio_operation_id: str | None = None - committed_audio_reserved_bytes: int = 0 - deferred_response_create: bool = False - deferred_precreate_response: bool = False - data_plane_task: asyncio.Task[None] | None = None - data_plane_restart_requested: bool = False - continuation_owner_id: str | None = None - continuation_units: int = 0 - pending_silence_task: asyncio.Task[bool] | None = None - pending_silence_owner_id: str | None = None - silence_continuation_scheduler: Callable[..., Awaitable[bool]] | None = None - - def retain_committed_audio( - self, - payload: dict[str, object], - *, - operation_id: str | None, - reserved_bytes: int = 0, - ) -> None: - self.committed_audio_payload = payload - self.committed_audio_operation_id = operation_id - self.committed_audio_reserved_bytes += max(0, int(reserved_bytes)) - - def clear_committed_audio(self) -> int: - reserved_bytes = self.committed_audio_reserved_bytes - self.committed_audio_payload = None - self.committed_audio_operation_id = None - self.committed_audio_reserved_bytes = 0 - self.deferred_response_create = False - self.deferred_precreate_response = False - return reserved_bytes - - def clear_continuation(self) -> None: - self.continuation_owner_id = None - self.continuation_units = 0 - self.pending_silence_task = None - self.pending_silence_owner_id = None - - -class PersonaPlexServingRuntimeAdapter: - adapter_id = "personaplex" - clean_response_done_prefix = "" - interrupted_tts_prefix = "" - private_runtime_config_keys = _PRIVATE_RUNTIME_CONFIG_KEYS - - def __init__(self, encode_audio: EncodeAudio) -> None: - self.session_states: dict[str, PersonaPlexServingSessionState] = {} - self.data_plane = PersonaPlexDataPlaneSession(encode_audio) - - def create_session_state(self) -> PersonaPlexServingSessionState: - return PersonaPlexServingSessionState() - - def session_state(self, session_id: str) -> PersonaPlexServingSessionState: - return self.session_states.setdefault(session_id, self.create_session_state()) - - def remove_session_state(self, session_id: str) -> None: - self.session_states.pop(session_id, None) - - @staticmethod - def is_enabled(config: object) -> bool: - del config - return True - - @staticmethod - def capabilities(*, max_sessions: int) -> DuplexCapabilities: - supports_multi_session = max_sessions > 1 - return DuplexCapabilities( - supports_model_native_turn_policy=True, - supports_external_turn_signal=False, - supports_client_commit=False, - supports_barge_in=False, - supports_playback_ack=True, - supports_input_append=True, - supports_replace_latest_chunk=False, - supports_reencode_context=False, - supports_rollback_to_checkpoint=False, - supports_turn_commit_only=False, - supports_model_internal_state=True, - supports_stage_resumption=True, - supports_core_resumable_request=True, - supports_stage_connector_handoff=True, - supports_independent_io_streams=True, - supports_realtime_endpoint=True, - supports_multi_session=supports_multi_session, - supports_multi_session_same_replica=supports_multi_session, - supports_session_lease=True, - supports_session_resume=False, - session_admission_mode="engine_managed", - supports_audio_truncate=False, - requires_model_runner_kv=True, - requires_native_stage_role=True, - implementation_level="model_native_duplex", - adapter_patterns=["scheduler_data_plane"], - input_modes=["append_audio_chunk"], - signal_sources=["model_native", "client_event"], - stage_handoff_transport="scheduler_data_plane", - chunk_period_ms=80, - target_barge_in_latency_ms=None, - ) - - @staticmethod - def validate_client_extra_body(extra_body: object) -> None: - if not isinstance(extra_body, dict): - return - private = sorted(_PRIVATE_RUNTIME_CONFIG_KEYS.intersection(extra_body)) - if private: - raise ServingRuntimeConfigError("PersonaPlex runtime configuration is server-owned: " + ", ".join(private)) - - @classmethod - async def prepare_runtime_config( - cls, - config: object, - *, - model_config: Any, - ) -> dict[str, object]: - extra_body = getattr(config, "extra_body", None) - cls.validate_client_extra_body(extra_body) - model_path = getattr(model_config, "model", None) - if not isinstance(model_path, str) or not model_path: - raise ServingRuntimeConfigError("PersonaPlex model path is unavailable") - voice = cls._voice_name(getattr(config, "voice", None)) - instructions = getattr(config, "instructions", None) or DEFAULT_PERSONA - from vllm_omni.model_executor.models.personaplex.duplex.stage0 import ( - personaplex_prefill_slots, - ) - - try: - prefill_slots = await asyncio.to_thread( - personaplex_prefill_slots, - model_path, - voice, - str(instructions), - ) - except Exception as exc: - raise ServingRuntimeConfigError(f"PersonaPlex voice/persona prefill could not be prepared: {exc}") from exc - return { - "personaplex_model_path": model_path, - "personaplex_voice_prompt": voice, - "personaplex_persona": str(instructions), - "personaplex_prefill_slots": prefill_slots, - } - - @classmethod - def runtime_config_for_update( - cls, - config: object, - current: Mapping[str, object], - ) -> dict[str, object]: - cls.validate_client_extra_body(getattr(config, "extra_body", None)) - new_persona = str(getattr(config, "instructions", None) or DEFAULT_PERSONA) - reject_changed_runtime_value( - new_persona, - current.get("personaplex_persona"), - message="PersonaPlex persona (instructions) cannot be changed after the session is created", - code="persona_update_unsupported", - ) - new_voice = cls._voice_name(getattr(config, "voice", None)) - reject_changed_runtime_value( - new_voice, - current.get("personaplex_voice_prompt"), - message="PersonaPlex voice cannot be changed after the session is created", - code="voice_update_unsupported", - ) - runtime_config = deepcopy(dict(current)) - runtime_config["personaplex_voice_prompt"] = new_voice - runtime_config["personaplex_persona"] = new_persona - return runtime_config - - @staticmethod - def data_plane_context( - *, - epoch: int, - turn_id: int, - active_response_turn_id: int | None, - active_response_id: str | None, - auto_responds: bool, - response_format: str, - speed: float | None, - modalities: tuple[str, ...], - ) -> PersonaPlexDataPlaneContext: - return PersonaPlexDataPlaneContext( - epoch=epoch, - turn_id=turn_id, - active_response_turn_id=active_response_turn_id, - active_response_id=active_response_id, - auto_responds=auto_responds, - response_format=response_format, - speed=speed, - modalities=modalities, - ) - - @staticmethod - def _voice_name(value: object) -> str: - voice = value if isinstance(value, str) and value else "NATF2.pt" - path = PurePath(voice) - if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts): - raise ServingRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename") - return voice - - -__all__ = [ - "PersonaPlexServingRuntimeAdapter", - "PersonaPlexServingSessionState", -] diff --git a/vllm_omni/model_executor/models/personaplex/duplex/stage0.py b/vllm_omni/model_executor/models/personaplex/duplex/stage0.py index 58d476bc5b9..5deecafdb36 100644 --- a/vllm_omni/model_executor/models/personaplex/duplex/stage0.py +++ b/vllm_omni/model_executor/models/personaplex/duplex/stage0.py @@ -3,17 +3,19 @@ from __future__ import annotations -import base64 -import binascii import io import tarfile from collections.abc import Callable from dataclasses import dataclass, field +from functools import lru_cache from pathlib import Path from typing import Any import numpy as np +from vllm_omni.model_executor.common.audio.pcm import pcm_f32le_samples +from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload +from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE from vllm_omni.model_executor.models.personaplex.duplex.policy import ( AUDIO_SILENCE_FRAME_CNT, SILENCE_TOKENS, @@ -22,7 +24,7 @@ wrap_with_system_tags, ) -_FRAME_SAMPLES = 1920 +_FRAME_SAMPLES = FRAME_SIZE @dataclass(slots=True) @@ -37,8 +39,10 @@ class PersonaPlexStage0PreparedAppend: @dataclass(slots=True) class PersonaPlexStage0SessionState: + """Lockstep state of one (session, epoch): a new epoch is a new Stage 0 request with fresh KV.""" + session_id: str - incarnation: int + epoch: int user_codes: Any | None = None last_text_token: Any | None = None last_agent_codes: Any | None = None @@ -94,14 +98,29 @@ def load_personaplex_voice_state(model_path: str, voice: str) -> dict[str, Any]: raise FileNotFoundError(f"PersonaPlex bundled voice prompt {voice!r} was not found under {model_path!r}") -def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int: +@lru_cache(maxsize=4) +def _cached_tokenizer(model_path: str): + return load_personaplex_tokenizer(model_path) + + +@lru_cache(maxsize=16) +def _cached_voice_embedding_rows(model_path: str, voice: str) -> int: state = load_personaplex_voice_state(model_path, voice) embeddings = state.get("embeddings") if not hasattr(embeddings, "shape") or len(embeddings.shape) < 1: raise ValueError(f"PersonaPlex voice prompt {voice!r} has no embeddings") - tokenizer = load_personaplex_tokenizer(model_path) - persona_tokens = tokenizer(wrap_with_system_tags(persona)) if persona else [] - return int(embeddings.shape[0]) + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens) + return int(embeddings.shape[0]) + + +def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int: + """Scheduler slots the first append of a session needs for the voice + persona prefill. + + The voice bundle row count and the tokenizer are cached per model path: + they are constant, and this runs on every session open. + """ + voice_rows = _cached_voice_embedding_rows(model_path, voice) + persona_tokens = _cached_tokenizer(model_path)(wrap_with_system_tags(persona)) if persona else [] + return voice_rows + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens) class PersonaPlexStage0DuplexRuntime: @@ -147,19 +166,24 @@ def prepare_append( session_id = duplex.get("session_id") if not isinstance(session_id, str) or not session_id: raise ValueError("PersonaPlex duplex append requires session_id") - incarnation = _coerce_non_negative_int(duplex.get("incarnation"), "incarnation") epoch = _coerce_non_negative_int(duplex.get("epoch"), "epoch") seq = _coerce_positive_int(duplex.get("seq"), "seq") identity = (epoch, seq) - key = (session_id, incarnation) + key = (session_id, epoch) state = self.sessions.get(key) if state is None: + # A newer epoch supersedes the session's earlier lockstep state: + # the engine aborted that request, but its finish notification + # may still be in flight, so release it here rather than let the + # two epochs share the codec budget. + for stale_key in [k for k in self.sessions if k[0] == session_id and k[1] < epoch]: + self.close_session(*stale_key) if len(self.sessions) >= self.max_sessions: raise RuntimeError(f"PersonaPlex Stage 0 session capacity {self.max_sessions} is exhausted") state = PersonaPlexStage0SessionState( session_id=session_id, - incarnation=incarnation, + epoch=epoch, codec=self._acquire_codec(), ) self.sessions[key] = state @@ -291,7 +315,6 @@ def prepare_append( "stage0_prepared": True, "prefill_applied": first_append, "session_id": session_id, - "incarnation": incarnation, "epoch": epoch, "seq": seq, }, @@ -363,8 +386,8 @@ def close_request(self, request_id: str) -> None: if not state.request_ids: self.close_session(*key) - def close_session(self, session_id: str, incarnation: int) -> None: - key = (session_id, incarnation) + def close_session(self, session_id: str, epoch: int) -> None: + key = (session_id, epoch) state = self.sessions.pop(key, None) if state is None: return @@ -421,23 +444,13 @@ def _model_device_dtype(self): @staticmethod def _decode_pcm(payload: object) -> np.ndarray: - if not isinstance(payload, dict): - raise ValueError("PersonaPlex duplex payload must be a mapping") - if payload.get("format") != "pcm_f32le" or payload.get("sample_rate_hz") != 24000: - raise ValueError("PersonaPlex Stage 0 requires 24 kHz pcm_f32le") - audio = payload.get("audio") - if not isinstance(audio, str): - raise ValueError("PersonaPlex Stage 0 requires base64 audio") - try: - raw = base64.b64decode(audio, validate=True) - except (binascii.Error, ValueError) as exc: - raise ValueError("PersonaPlex Stage 0 audio is not valid base64") from exc - samples = np.frombuffer(raw, dtype=" int: diff --git a/vllm_omni/model_executor/models/personaplex/pipeline.py b/vllm_omni/model_executor/models/personaplex/pipeline.py index ac34bbd230d..f0c4f7b8586 100644 --- a/vllm_omni/model_executor/models/personaplex/pipeline.py +++ b/vllm_omni/model_executor/models/personaplex/pipeline.py @@ -2,9 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project """PersonaPlex pipeline: Talker (AR decode -> Mimi codebooks) -> Code2Wav (codebooks -> 24 kHz PCM). -PersonaPlex is a Moshi finetune (full-duplex speech-to-speech). For offline/batch -runs it is served as a 2-stage vllm-omni audio->audio pipeline, reusing the -Qwen3-TTS staged topology: +PersonaPlex is a Moshi finetune (full-duplex speech-to-speech), served as a +2-stage vllm-omni audio->audio pipeline that reuses the Qwen3-TTS staged +topology (online: one resumable Stage 0 request per duplex session, see +``duplex/plugin.py``): * Stage 0 (``personaplex``) is the AR talker: the Helium temporal transformer plus the depformer (both built by the lead). It emits the per-frame audio @@ -31,13 +32,8 @@ # Pipeline-level default; the code2wav stage overrides per-stage below. model_arch="PersonaPlexTalkerForConditionalGeneration", default_deploy_config_name="personaplex.yaml", - duplex_runtime_extension=( - "vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension" - ), - duplex_serving_adapter=( - "vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter" - ), - duplex_control_enabled=True, + # Served full duplex over /v1/realtime?duplex=1 by the unified framework. + duplex_plugin="vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin", stages=( StagePipelineConfig( stage_id=0,