Repository navigation
[Core] Give the OpenAI Realtime wire codec a reusable home (RFC #6592 P0a) - #7640
Conversation
|
This PR appears to belong to: docs/design/module/entrypoints.md. Module owners: @NickCao @alex-jw-brooks @linyueqian Routing: @NickCao via module of the changed files, module named in the PR description, semantic router, CODEOWNERS; @alex-jw-brooks via module of the changed files, module named in the PR description, semantic router; @linyueqian via module of the changed files, module named in the PR description, semantic router @psv666, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
|
I think this is more systematic and the boundary between realtime and duplex protocol is very clear |
There was a problem hiding this comment.
Reviewed this against the P0a scope in #6592 and the prerequisites described in #7223.
Moving the shared helpers into protocol/realtime makes sense after #7413, while session ownership and model execution should remain in the duplex engine.
I agree with @chickeyton that common Realtime message types and serialization should also be reusable, with duplex-specific protocol extensions separated from the common types. The current classes mix wire representation with engine-internal behavior—for example, UpdateSession.payload() produces a turn.signal mailbox message. The extraction should distinguish those responsibilities: share the wire types and serialization, while keeping internal command mapping in the engine, rather than moving the existing classes wholesale.
One non-blocking issue: the RealtimeProtocolCapabilities docstring says the defaults only support turn_detection: null, but validate_turn_detection=None actually skips validation and accepts other values, including semantic_vad. Please document this as "no turn-detection validation" and make clear that consumers must supply their own validator when restricting supported modes.
The extracted helpers still retain existing duplex assumptions, including 16 kHz append conversion and MiniCPM-specific frame validation. Capability restrictions also currently apply only to session validation. Existing duplex behavior should remain unchanged in P0a; the Qwen3-specific policies and validation across its input paths can be addressed in P0b/P1.
I found no blocking behavior regression in the code reviewed. The main design concern is the boundary between shared wire types and duplex-internal commands.
amy-why-3459
left a comment
amy-why-3459
left a comment
There was a problem hiding this comment.
P0a extraction looks right: after #7413 the codec cannot live under entrypoints/, and vllm_omni/protocol/realtime/ is the correct layer-neutral home. I would not expand this PR into protocol/duplex or a wholesale DuplexCommand move — UpdateSession.payload() is an engine mailbox (turn.signal), not a wire type.
@chickeyton I agree the OpenAI event types should be reusable, but I would not put DuplexCommand / DuplexEvent in protocol/duplex in this PR. Those classes mix wire fields with engine-internal payload() (e.g. UpdateSession → turn.signal). P0a’s job is the stateless codec; the follow-up is a protocol-level client-event decode that duplex maps onto DuplexCommand, rather than moving the mailbox types into protocol/.
Please keep this PR behavior-preserving and land three honesty fixes:
RealtimeProtocolCapabilitiesclass docstring contradicts the field, the tests, andvalidate_turn_detection=None(skips validation, includingsemantic_vad).DuplexSessionConfig.from_realtimestill calls the two validators directly;translate_realtime_commandalready usesvalidate_session_payload+DUPLEX_REALTIME_CAPABILITIES.ENTRY-INV-002asked for one session check.fullduplex.md#7 /entrypoints.md/ therealtime_commands.pymodule docstring over-claim that “parsing a client event” lives inprotocol/. Only the leaf helpers moved. Also fix the test path invllm_omni/protocol/__init__.py(tests/protocol/realtime/..., nottests/protocol/...).
The remaining P0a gap vs RFC #6592 is translate_realtime_command: the OpenAI event-type switch still returns DuplexCommand. That is fine not to do here, but please name it as the next extraction so P0b (Qwen3 GA) does not copy the parser. Follow-up shape: protocol-level decode → engine maps onto DuplexCommand.
16 kHz append conversion, MiniCPM video_frames, and session-only capability checks are acceptable P0a freeze; they are consumer policy for P0b, not a reason to reopen the codec in this PR.
|
|
||
| The defaults are the full set the codec knows how to decode and no turn | ||
| detection, which is the shape of a consumer that accepts every format it is | ||
| handed and only supports ``turn_detection: null``. |
There was a problem hiding this comment.
The class docstring says the defaults “only support turn_detection: null”, but validate_turn_detection=None skips the check. test_a_consumer_without_a_turn_detection_check_ignores_turn_detection already accepts semantic_vad, and the field comment is correct.
Please document this as “no turn-detection validation”; consumers that want to restrict modes must supply their own validator (as DUPLEX_REALTIME_CAPABILITIES does).
| normalize_turn_detection_session_payload, | ||
| validate_realtime_turn_detection, | ||
| ) | ||
| from vllm_omni.protocol.realtime.formats import validate_realtime_session_audio_formats |
There was a problem hiding this comment.
This path still does:
format_error = validate_realtime_session_audio_formats(payload)
turn_detection_error = validate_realtime_turn_detection(payload)while translate_realtime_command uses validate_session_payload(..., capabilities=DUPLEX_REALTIME_CAPABILITIES). Same order and codes today, but ENTRY-INV-002 just required one implementation.
Please route from_realtime through validate_session_payload and keep raising DuplexConfigError from the rejection (param="turn_detection" when that is the code). Then a later narrower capability object cannot drift between handshake and session.update.
| helpers on :class:`~vllm_omni.engine.duplex.realtime_events.RealtimeProjectionState`; | ||
| This module is the *duplex binding* of the shared Realtime codec. Parsing, | ||
| validating and decoding a client event is not duplex-specific and lives in | ||
| ``vllm_omni.protocol.realtime``; what lives here is the part that is: which |
There was a problem hiding this comment.
“Parsing, validating and decoding a client event … lives in vllm_omni.protocol.realtime” is ahead of the code. decode_audio_append and the session/item/format helpers moved; the event-type switch (session.update, response.create, turn.signal, …) is still translate_realtime_command and still returns DuplexCommand.
Please narrow this sentence to what actually moved, and call out the dispatcher as the leftover so P0b does not fork it. Same over-claim is in docs/design/fullduplex.md #7 and docs/design/module/entrypoints.md (“which client events are accepted”).
| the session state lives. That keeps one protocol implementation usable by more | ||
| than one runtime (see ``vllm_omni.protocol.realtime``). | ||
|
|
||
| Dependency rule, asserted by ``tests/protocol/test_protocol_import_boundary.py``: |
There was a problem hiding this comment.
This points at tests/protocol/test_protocol_import_boundary.py; the file added in this PR is tests/protocol/realtime/test_protocol_import_boundary.py (and that is the path entrypoints.md already uses).
|
Regarding resample: openai api doc says all inputs and outputs are 24kHz sample rate. But our models natively only accepts 16kHz (minicpm and nemotron) or 24kHz (personalplex), and outputs other sample rates, like 24kHz for minicpm but 22.05 kHz for nemotron. We should do the resampling from client rate (likely 24kHz, if using openai compatible client) to model native input rates, and from model native output rates to 24kHz in this layer. |
Please take a reference of |
…project#6592 P0a) RFC vllm-project#6592 P0a asks for the model-agnostic OpenAI Realtime protocol layer to be promoted out of the duplex code so a second Realtime surface can reuse it. PR vllm-project#7413 had already moved it out of MiniCPM and out of serving, but it landed inside vllm_omni/engine/duplex/ -- a package whose job is engine-resident duplex sessions. A non-duplex consumer could only reach the codec by importing the duplex engine, and duplex runtime dependencies had started leaking back into it: translate_realtime_command reached into engine/duplex/turn_detection (and so the Silero VAD backend) to validate session.update. Move the stateless, model- and runtime-agnostic half into a layer-neutral vllm_omni/protocol/realtime/: audio format negotiation, session-object readers, conversation-item rules, the append decoder, the PCM/G.711/WAV codec and the error envelope. RFC vllm-project#6592 proposes entrypoints/openai/realtime/ for this, which vllm-project#7413 rules out: it would put a protocol layer back in serving and make engine/duplex/session/runner.py import entrypoints. A shared layer that both the engine and a future serving-side surface depend on has to belong to neither, so it sits beside config/, inputs/ and outputs/. What stays where it is: * DuplexCommand / DuplexEvent keep vllm-project#7413's typed boundary. They also carry duplex extensions (Listen, Speak, OverlapDecision, TurnEvent) that do not belong in a shared GA vocabulary. * RealtimeProjectionState and DuplexSessionRunner stay engine-side. The projection is session state, and the engine owns session state. * DuplexModelPlugin stays the MiniCPM integration seam, untouched. engine/duplex/realtime_commands.py (887 -> 389 lines) becomes the duplex binding of the shared codec: which DuplexCommand a decoded event becomes, plus duplex_response_format and DUPLEX_REALTIME_CAPABILITIES. The names it used to define are re-exported from it so existing importers and tests keep working; engine/duplex/audio.py is a re-export shim for the same reason. There is one implementation of each behaviour, asserted by identity. RFC vllm-project#6592's RealtimeModelAdapter is three quarters already built: build_prompt is DuplexModelPlugin.plan_append plus the model's stage0, start_response is the session runner plus DuplexStagePort.submit, and cancel is the runner's control path plus DuplexDataPlane.close_stream. Adding an interface over those would be a second model integration framework. Only the fourth quarter, supports(turn_detection, formats), was missing and is genuinely protocol-layer work, so it lands as RealtimeProtocolCapabilities: two format sets and one validator hook that the session check consults. Duplex supplies DUPLEX_REALTIME_CAPABILITIES, which defers to the existing turn-detection validator through a lazy import, so the VAD backend is still not loaded until a session object arrives. Behaviour-preserving. Every moved function body is unchanged; the new code is the constructor lift out of build_append_audio and a session check that runs the same two validations in the same order with the same codes. A differential probe over 1882 observable points -- 1448 client-event translations across every event type and four default profiles (460 of them error paths), 38 session objects through the config and response-option paths, all 49 error codes rendered as envelopes, 57 audio conversions, a 28-step projection replay and the runner-side resolvers -- produces byte-identical output against a worktree pinned at the parent commit (md5 db675cc9e29c24344f4283bf6243439b). The existing duplex tests pass unchanged and no wire expectation was edited. The three new tests are structural only: the import direction, the capability seam, and identity assertions that a copy-and-fork cannot satisfy. Verified on this base: the focused duplex/protocol suites pass (578), mypy reports the same 46 pre-existing errors in the same 5 files as the parent commit with none in the new package, and the L3 merge gate "Omni . MiniCPM-o 4.5 Duplex Test" (tests/e2e/online_serving/test_minicpmo_4_5_duplex.py) passes 11/11 in 8m02s, collecting the same 11-of-12 as the parent so nothing is silently skipped. Review asked for the other half of the codec too (vllm-project#7640): chickeyton proposed a complete Realtime Command/Event vocabulary in protocol/realtime with a protocol/duplex layer for the extensions, and natureofnature added the constraint that makes it work -- the existing classes are not pure wire types, so share the wire types and serialization but keep the internal command mapping in the engine, rather than relocating the classes wholesale. Both are right, and the gap was real: extracting only the stateless helpers would have left P0b re-declaring ~40 event classes and their rendering, which is the larger half of "a second OpenAI Realtime codec". So the 42 events and 17 commands are split along that line: * protocol/realtime/events.py -- the 30 events OpenAI defines, the base class and the pure to_realtime() rendering. * protocol/duplex/events.py -- the 12 vLLM-Omni adds (listen/speak, playback.acknowledged, overlap.decision, turn.event, the session resume/replace/resync trio, session.closed/expired/heartbeat_ack, duplex.raw). * protocol/realtime/commands.py -- the 10 GA client commands, carrying the WIRE type and the decoded fields. * protocol/duplex/commands.py -- the 7 extensions (barge_in, playback.ack, input.cancel, input.text.append, session.heartbeat, session.close, turn.signal). * engine/duplex/commands.py -- the mailbox half stays: payload() renders the session runner's dictionary and its `type` is the runner's channel, which is genuinely not the client event. session.update and the three conversation.item.* commands all travel on `turn.signal`; that four-way collision is the concrete proof the halves needed separating rather than moving. Each concrete class pairs the engine half with its protocol twin, so payload() stays a method and isinstance holds against both. engine/duplex/events.py becomes a re-export shim that keeps `DuplexEvent` as the name 18 files already use (it is the same object as RealtimeEvent) and keeps DOMAIN_TERMINAL_EVENTS / MODEL_OUTPUT_EVENTS, which are the runner's epoch-filter policy over *internal* event types, not wire contract. Two defects review found are fixed: the RealtimeProtocolCapabilities docstring claimed the defaults only accept turn_detection: null when they actually skip turn-detection validation entirely, and the package now states plainly which duplex-shaped defaults it still carries (16 kHz resample target, the MiniCPM-o camera-frame contract) and that parameterizing them is P0b/P1. RFC vllm-project#6592 open question 8 asks who owns the promoted module. A new top-level package with no CODEOWNERS rule falls to the repository-wide fallback, which would make the wire codec's reviewers an accident rather than a choice, so this proposes an answer: entrypoints.md declares vllm_omni/protocol/** and its existing owners review it. Deciding what a client may put on the wire is that document's subject, `vllm_omni/errors.py` and `inputs/` / `outputs/` are already owned by contract documents rather than by a consumer, and two of those owners have history on the Qwen3 Realtime connection that P0b will target -- so the future second consumer has review rights on day one. Ownership is review scope, not a dependency: ENTRY-INV-002 states the import rule the boundary test enforces. It is deliberately not fullduplex.md, so a non-duplex consumer never needs duplex review to change the codec. Maintainers should reassign this if they would rather it graduate to its own module document now; the expectation recorded in the doc is that it graduates when P0b gives it a second consumer. No P0b or P1 behaviour: no Qwen3 adapter, prompt builder, profile selection, route dispatch, GA audio rate change, image/video input or VAD change. Signed-off-by: psv666 <2693925048@qq.com>
8ed36b5 to
c334b01
Compare
…project#6592 P0a) RFC vllm-project#6592 P0a asks for the model-agnostic OpenAI Realtime protocol layer to be promoted out of the duplex code so a second Realtime surface can reuse it. PR vllm-project#7413 had already moved it out of MiniCPM and out of serving, but it landed inside vllm_omni/engine/duplex/ -- a package whose job is engine-resident duplex sessions. A non-duplex consumer could only reach the codec by importing the duplex engine, and duplex runtime dependencies had started leaking back into it: translate_realtime_command reached into engine/duplex/turn_detection (and so the Silero VAD backend) to validate session.update. Move the stateless, model- and runtime-agnostic half into a layer-neutral vllm_omni/protocol/realtime/: audio format negotiation, session-object readers, conversation-item rules, the append decoder, the PCM/G.711/WAV codec and the error envelope. RFC vllm-project#6592 proposes entrypoints/openai/realtime/ for this, which vllm-project#7413 rules out: it would put a protocol layer back in serving and make engine/duplex/session/runner.py import entrypoints. A shared layer that both the engine and a future serving-side surface depend on has to belong to neither, so it sits beside config/, inputs/ and outputs/. What stays where it is: * DuplexCommand / DuplexEvent keep vllm-project#7413's typed boundary. They also carry duplex extensions (Listen, Speak, OverlapDecision, TurnEvent) that do not belong in a shared GA vocabulary. * RealtimeProjectionState and DuplexSessionRunner stay engine-side. The projection is session state, and the engine owns session state. * DuplexModelPlugin stays the MiniCPM integration seam, untouched. engine/duplex/realtime_commands.py (887 -> 389 lines) becomes the duplex binding of the shared codec: which DuplexCommand a decoded event becomes, plus duplex_response_format and DUPLEX_REALTIME_CAPABILITIES. The names it used to define are re-exported from it so existing importers and tests keep working; engine/duplex/audio.py is a re-export shim for the same reason. There is one implementation of each behaviour, asserted by identity. RFC vllm-project#6592's RealtimeModelAdapter is three quarters already built: build_prompt is DuplexModelPlugin.plan_append plus the model's stage0, start_response is the session runner plus DuplexStagePort.submit, and cancel is the runner's control path plus DuplexDataPlane.close_stream. Adding an interface over those would be a second model integration framework. Only the fourth quarter, supports(turn_detection, formats), was missing and is genuinely protocol-layer work, so it lands as RealtimeProtocolCapabilities: two format sets and one validator hook that the session check consults. Duplex supplies DUPLEX_REALTIME_CAPABILITIES, which defers to the existing turn-detection validator through a lazy import, so the VAD backend is still not loaded until a session object arrives. Behaviour-preserving. Every moved function body is unchanged; the new code is the constructor lift out of build_append_audio and a session check that runs the same two validations in the same order with the same codes. A differential probe over 1943 observable points -- 1448 client-event translations across every event type and four default profiles (460 of them error paths), 38 session objects through the config and response-option paths, all 49 error codes rendered as envelopes, 57 audio conversions, a 28-step projection replay and the runner-side resolvers -- produces byte-identical output against a worktree pinned at the parent commit (byte-identical in 14 of 15 sections; see the key-order note above). The existing duplex tests pass unchanged and no wire expectation was edited. The three new tests are structural only: the import direction, the capability seam, and identity assertions that a copy-and-fork cannot satisfy. Verified on this base: the focused duplex/protocol suites pass (578), mypy reports the same 46 pre-existing errors in the same 5 files as the parent commit with none in the new package, and the L3 merge gate "Omni . MiniCPM-o 4.5 Duplex Test" (tests/e2e/online_serving/test_minicpmo_4_5_duplex.py) passes 11/11 in 8m02s, collecting the same 11-of-12 as the parent so nothing is silently skipped. Review asked for the other half of the codec too (vllm-project#7640): chickeyton proposed a complete Realtime Command/Event vocabulary in protocol/realtime with a protocol/duplex layer for the extensions, and natureofnature added the constraint that makes it work -- the existing classes are not pure wire types, so share the wire types and serialization but keep the internal command mapping in the engine, rather than relocating the classes wholesale. Both are right, and the gap was real: extracting only the stateless helpers would have left P0b re-declaring ~40 event classes and their rendering, which is the larger half of "a second OpenAI Realtime codec". The split follows the three tiers `docs/serving/realtime_duplex_api.md` already defines, which @chickeyton pointed at -- Tier 1 identical to OpenAI, Tier 2 OpenAI names carrying our extensions, Tier 3 ours alone. A first pass used a two-bucket GA/omni taxonomy and got Tier 2 wrong: seven classes with OpenAI names but vLLM-Omni fields (`session.created` + `resume_token`, `input_audio_buffer.append` + `video_frames`, ...) landed in the package that claims to be the OpenAI surface. Now each Tier 2 class subclasses its Tier 1 twin and declares only what it adds, so `protocol/realtime` is honestly pure: * protocol/realtime -- 22 Tier-1 events, 8 Tier-1 commands, the base classes and the pure to_realtime() rendering, with OpenAI's fields and no others. * protocol/duplex -- 8 Tier-2 event subclasses, 2 Tier-2 command subclasses, 12 Tier-3 events, 7 Tier-3 commands, and our error-code vocabulary (the doc files that under Tier 3 too; `protocol/realtime` keeps only the envelope shape and OpenAI's three error classes). So the 42 events and 17 commands are split along that line: * protocol/realtime/events.py -- the 30 events OpenAI defines, the base class and the pure to_realtime() rendering. * protocol/duplex/events.py -- the 12 vLLM-Omni adds (listen/speak, playback.acknowledged, overlap.decision, turn.event, the session resume/replace/resync trio, session.closed/expired/heartbeat_ack, duplex.raw). * protocol/realtime/commands.py -- the 10 GA client commands, carrying the WIRE type and the decoded fields. * protocol/duplex/commands.py -- the 7 extensions (barge_in, playback.ack, input.cancel, input.text.append, session.heartbeat, session.close, turn.signal). * engine/duplex/commands.py -- the mailbox half stays: payload() renders the session runner's dictionary and its `type` is the runner's channel, which is genuinely not the client event. session.update and the three conversation.item.* commands all travel on `turn.signal`; that four-way collision is the concrete proof the halves needed separating rather than moving. Each concrete class pairs the engine half with its protocol twin, so payload() stays a method and isinstance holds against both. engine/duplex/events.py becomes a re-export shim that keeps `DuplexEvent` as the name 18 files already use (it is the same object as RealtimeEvent) and keeps DOMAIN_TERMINAL_EVENTS / MODEL_OUTPUT_EVENTS, which are the runner's epoch-filter policy over *internal* event types, not wire contract. Two defects review found are fixed: the RealtimeProtocolCapabilities docstring claimed the defaults only accept turn_detection: null when they actually skip turn-detection validation entirely, and the package now states plainly which duplex-shaped defaults it still carries (16 kHz resample target, the MiniCPM-o camera-frame contract) and that parameterizing them is P0b/P1. Two further review findings from @amy-why-3459 are fixed: `DuplexSessionConfig.from_realtime` called the two session validators directly while `translate_realtime_command` went through `validate_session_payload` + `DUPLEX_REALTIME_CAPABILITIES` -- two session checks, which ENTRY-INV-002 (this PR's own invariant) forbids; it now uses the same seam, and `RealtimeSessionRejection` carries `param` so the config path keeps reporting `param="turn_detection"`. The `tests/protocol/...` path in `vllm_omni/protocol/__init__.py` was also wrong. One deliberate, measured diff: splitting `response_id` out of `ResponseCreated` and `ResponseDone` as a Tier 2 extension moves it after `response` in the field order, so those two events render the same keys and values in a different order. The probe records JSON key order precisely so this is visible rather than hidden; 14 of its 15 sections are byte-identical and `key_order` differs only in those two entries. RFC vllm-project#6592 open question 8 asks who owns the promoted module. A new top-level package with no CODEOWNERS rule falls to the repository-wide fallback, which would make the wire codec's reviewers an accident rather than a choice, so this proposes an answer: entrypoints.md declares vllm_omni/protocol/** and its existing owners review it. Deciding what a client may put on the wire is that document's subject, `vllm_omni/errors.py` and `inputs/` / `outputs/` are already owned by contract documents rather than by a consumer, and two of those owners have history on the Qwen3 Realtime connection that P0b will target -- so the future second consumer has review rights on day one. Ownership is review scope, not a dependency: ENTRY-INV-002 states the import rule the boundary test enforces. It is deliberately not fullduplex.md, so a non-duplex consumer never needs duplex review to change the codec. Maintainers should reassign this if they would rather it graduate to its own module document now; the expectation recorded in the doc is that it graduates when P0b gives it a second consumer. No P0b or P1 behaviour: no Qwen3 adapter, prompt builder, profile selection, route dispatch, GA audio rate change, image/video input or VAD change. Signed-off-by: psv666 <2693925048@qq.com>
c334b01 to
2ae54ec
Compare
|
Redone per the three tiers. @chickeyton — you pointed at I had invented a two-bucket GA/omni taxonomy. Measuring it against the doc's tiers afterwards, seven classes with OpenAI names but vLLM-Omni fields had landed in the package that claims to be the OpenAI surface: That is exactly the failure the tier split exists to prevent, so Tier 2 now goes where your diagram put it — "extended realtime classes" in # protocol/realtime/events.py — Tier 1, OpenAI's fields and no others
class SessionCreated(RealtimeEvent):
session: Mapping[str, object] = field(default_factory=dict)
# protocol/duplex/events.py — Tier 2, same name, same wire type, additive only
class SessionCreated(realtime_events.SessionCreated):
attachment_generation: int | None = None
resume_token: str | None = NoneA test now enforces it both ways: Tier 2 must be a strict superset of its Tier 1 twin with the same I also moved the error-code vocabulary to Tier 3 ( @amy-why-3459 — your three fixes are all inYou and @chickeyton disagreed on whether to do the split here at all; the team settled on doing it. Your three concrete findings stand regardless and are fixed:
On your last point — @NickCao — resamplingNoted and recorded, not acted on. One deliberate diffSplitting Result: of the probe's 15 sections over 1943 observable points, 14 are byte-identical to the parent commit and Verification
The probe caught one real bug during the rework: TaxonomyINFO 09-17 11:52:06 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed. Events
Tier 1: 22 · Tier 2: 8 · Tier 3: 12 = 42 Commands
Tier 1: 8 · Tier 2: 2 · Tier 3: 7 = 17 Bold mailbox = the session runner's channel differs from the client event (4 of 17) — the reason the mailbox half stays engine-side. |
|
…project#6592 P0a) RFC vllm-project#6592 P0a asks for the model-agnostic OpenAI Realtime protocol layer to be promoted out of the duplex code so a second Realtime surface can reuse it. PR vllm-project#7413 had already moved it out of MiniCPM and out of serving, but it landed inside vllm_omni/engine/duplex/ -- a package whose job is engine-resident duplex sessions. A non-duplex consumer could only reach the codec by importing the duplex engine, and duplex runtime dependencies had started leaking back into it: translate_realtime_command reached into engine/duplex/turn_detection (and so the Silero VAD backend) to validate session.update. Move the stateless, model- and runtime-agnostic half into a layer-neutral vllm_omni/protocol/realtime/: audio format negotiation, session-object readers, conversation-item rules, the append decoder, the PCM/G.711/WAV codec and the error envelope. RFC vllm-project#6592 proposes entrypoints/openai/realtime/ for this, which vllm-project#7413 rules out: it would put a protocol layer back in serving and make engine/duplex/session/runner.py import entrypoints. A shared layer that both the engine and a future serving-side surface depend on has to belong to neither, so it sits beside config/, inputs/ and outputs/. What stays where it is: * DuplexCommand / DuplexEvent keep vllm-project#7413's typed boundary. They also carry duplex extensions (Listen, Speak, OverlapDecision, TurnEvent) that do not belong in a shared GA vocabulary. * RealtimeProjectionState and DuplexSessionRunner stay engine-side. The projection is session state, and the engine owns session state. * DuplexModelPlugin stays the MiniCPM integration seam, untouched. engine/duplex/realtime_commands.py (887 -> 389 lines) becomes the duplex binding of the shared codec: which DuplexCommand a decoded event becomes, plus duplex_response_format and DUPLEX_REALTIME_CAPABILITIES. The names it used to define are re-exported from it so existing importers and tests keep working; engine/duplex/audio.py is a re-export shim for the same reason. There is one implementation of each behaviour, asserted by identity. RFC vllm-project#6592's RealtimeModelAdapter is three quarters already built: build_prompt is DuplexModelPlugin.plan_append plus the model's stage0, start_response is the session runner plus DuplexStagePort.submit, and cancel is the runner's control path plus DuplexDataPlane.close_stream. Adding an interface over those would be a second model integration framework. Only the fourth quarter, supports(turn_detection, formats), was missing and is genuinely protocol-layer work, so it lands as RealtimeProtocolCapabilities: two format sets and one validator hook that the session check consults. Duplex supplies DUPLEX_REALTIME_CAPABILITIES, which defers to the existing turn-detection validator through a lazy import, so the VAD backend is still not loaded until a session object arrives. Behaviour-preserving. Every moved function body is unchanged; the new code is the constructor lift out of build_append_audio and a session check that runs the same two validations in the same order with the same codes. A differential probe over 1943 observable points -- 1448 client-event translations across every event type and four default profiles (460 of them error paths), 38 session objects through the config and response-option paths, all 49 error codes rendered as envelopes, 57 audio conversions, a 28-step projection replay and the runner-side resolvers -- produces byte-identical output against a worktree pinned at the parent commit (byte-identical in 14 of 15 sections; see the key-order note above). The existing duplex tests pass unchanged and no wire expectation was edited. The four new test files are structural only: the import direction, the capability seam, and identity assertions that a copy-and-fork cannot satisfy. Verified on this base: the focused duplex/protocol/fullduplex suites pass (779), mypy reports the same 46 pre-existing errors as the parent commit across every touched file with none introduced by the new packages, and the L3 merge gate "Omni . MiniCPM-o 4.5 Duplex Test" (tests/e2e/online_serving/test_minicpmo_4_5_duplex.py) passes 11/11 in 7m48s, collecting the same 11-of-12 as the parent so nothing is silently skipped. The Tier-2 split had introduced one new mypy-3.10 error (ErrorEvent.error.fget -- attribute access on a class yields the getter, not the property object); it is reached through __dict__ now, so the error set is byte-identical to the parent. Review asked for the other half of the codec too (vllm-project#7640): chickeyton proposed a complete Realtime Command/Event vocabulary in protocol/realtime with a protocol/duplex layer for the extensions, and natureofnature added the constraint that makes it work -- the existing classes are not pure wire types, so share the wire types and serialization but keep the internal command mapping in the engine, rather than relocating the classes wholesale. Both are right, and the gap was real: extracting only the stateless helpers would have left P0b re-declaring ~40 event classes and their rendering, which is the larger half of "a second OpenAI Realtime codec". The split follows the three tiers `docs/serving/realtime_duplex_api.md` already defines, which @chickeyton pointed at -- Tier 1 identical to OpenAI, Tier 2 OpenAI names carrying our extensions, Tier 3 ours alone. A first pass used a two-bucket GA/omni taxonomy and got Tier 2 wrong: seven classes with OpenAI names but vLLM-Omni fields (`session.created` + `resume_token`, `input_audio_buffer.append` + `video_frames`, ...) landed in the package that claims to be the OpenAI surface. Now each Tier 2 class subclasses its Tier 1 twin and declares only what it adds, so `protocol/realtime` is honestly pure: * protocol/realtime -- 22 Tier-1 events, 8 Tier-1 commands, the base classes and the pure to_realtime() rendering, with OpenAI's fields and no others. * protocol/duplex -- 8 Tier-2 event subclasses, 2 Tier-2 command subclasses, 12 Tier-3 events, 7 Tier-3 commands, and our error-code vocabulary (the doc files that under Tier 3 too; `protocol/realtime` keeps only the envelope shape and OpenAI's three error classes). So the 42 events and 17 commands are split along that line: * protocol/realtime/events.py -- the 30 events OpenAI defines, the base class and the pure to_realtime() rendering. * protocol/duplex/events.py -- the complete 42-event duplex vocabulary: the 22 Tier-1 classes re-exported, the 8 Tier-2 subclasses, and the 12 vLLM-Omni adds (listen/speak, playback.acknowledged, overlap.decision, turn.event, the session resume/replace/resync trio, session.closed/expired/heartbeat_ack, duplex.raw). * protocol/realtime/commands.py -- the 10 GA client commands, carrying the WIRE type and the decoded fields. * protocol/duplex/commands.py -- the complete 17-command duplex vocabulary: the 8 Tier-1 classes re-exported, the 2 Tier-2 subclasses, and the 7 extensions (barge_in, playback.ack, input.cancel, input.text.append, session.heartbeat, session.close, turn.signal). * engine/duplex/commands.py -- the mailbox half stays: payload() renders the session runner's dictionary and its `type` is the runner's channel, which is genuinely not the client event. session.update and the three conversation.item.* commands all travel on `turn.signal`; that four-way collision is the concrete proof the halves needed separating rather than moving. Each concrete class pairs the engine half with its protocol twin, so payload() stays a method and isinstance holds against both. engine/duplex/events.py becomes a re-export shim that keeps `DuplexEvent` as the name 18 files already use (it is the same object as RealtimeEvent) and keeps DOMAIN_TERMINAL_EVENTS / MODEL_OUTPUT_EVENTS, which are the runner's epoch-filter policy over *internal* event types, not wire contract. protocol/duplex is also the only door a duplex consumer uses, which is the rest of chickeyton's proposal: the dependency chain is protocol/realtime <- protocol/duplex <- engine/entrypoints/clients, and the middle link only earns its keep if the last arrow is the only one. Ten modules had been reaching past it -- engine/duplex/{audio,commands,config,events,realtime_commands, realtime_events}.py, session/runner.py, entrypoints/duplex/{serving, realtime_input}.py and InlineDuplexClient. protocol/duplex now re-exports the 43 Tier-1 helper names alongside its own, and those ten import from it; a new AST test over the consumer packages keeps it that way, and asserts a re-exported Tier-1 name IS the Tier-1 object rather than a second declaration. The payoff is the change NickCao asked for: convert_input_audio_with_rate hardcodes MiniCPM-o's 16 kHz, and when that becomes client-rate -> model-native the override lands in one file instead of ten call sites. chickeyton proposed spelling the base as `type DuplexCommand = RealtimeCommand`. The PEP 695 form is Python 3.12+, and this repo is >=3.10 with a mypy-3.10 commit hook, so it would be a SyntaxError rather than a degraded import; it also produces a TypeAliasType rather than a class, and DuplexSessionRunner dispatches commands entirely on isinstance. Plain assignment is used instead. It applies to DuplexEvent and not to DuplexCommand: an event has no engine-internal half, so DuplexEvent really is RealtimeEvent, while a command carries payload() and its mailbox channel. A test states that asymmetry so it is not later "fixed" in the wrong direction. Two defects review found are fixed: the RealtimeProtocolCapabilities docstring claimed the defaults only accept turn_detection: null when they actually skip turn-detection validation entirely, and the package now states plainly which duplex-shaped defaults it still carries (16 kHz resample target, the MiniCPM-o camera-frame contract) and that parameterizing them is P0b/P1. Two further review findings from @amy-why-3459 are fixed: `DuplexSessionConfig.from_realtime` called the two session validators directly while `translate_realtime_command` went through `validate_session_payload` + `DUPLEX_REALTIME_CAPABILITIES` -- two session checks, which ENTRY-INV-002 (this PR's own invariant) forbids; it now uses the same seam, and `RealtimeSessionRejection` carries `param` so the config path keeps reporting `param="turn_detection"`. The `tests/protocol/...` path in `vllm_omni/protocol/__init__.py` was also wrong. One deliberate, measured diff: splitting `response_id` out of `ResponseCreated` and `ResponseDone` as a Tier 2 extension moves it after `response` in the field order, so those two events render the same keys and values in a different order. The probe records JSON key order precisely so this is visible rather than hidden; 14 of its 15 sections are byte-identical and `key_order` differs only in those two entries. RFC vllm-project#6592 open question 8 asks who owns the promoted module. A new top-level package with no CODEOWNERS rule falls to the repository-wide fallback, which would make the wire codec's reviewers an accident rather than a choice, so this proposes an answer: entrypoints.md declares vllm_omni/protocol/** and its existing owners review it. Deciding what a client may put on the wire is that document's subject, `vllm_omni/errors.py` and `inputs/` / `outputs/` are already owned by contract documents rather than by a consumer, and two of those owners have history on the Qwen3 Realtime connection that P0b will target -- so the future second consumer has review rights on day one. Ownership is review scope, not a dependency: ENTRY-INV-002 states the import rule the boundary test enforces. It is deliberately not fullduplex.md, so a non-duplex consumer never needs duplex review to change the codec. Maintainers should reassign this if they would rather it graduate to its own module document now; the expectation recorded in the doc is that it graduates when P0b gives it a second consumer. No P0b or P1 behaviour: no Qwen3 adapter, prompt builder, profile selection, route dispatch, GA audio rate change, image/video input or VAD change. Signed-off-by: psv666 <2693925048@qq.com>
2ae54ec to
393b21c
Compare
|
Took 2 and 4, kept 3 as it is, and 1 needed a different spelling. Details and evidence below. 2 + 4 — done:
|
| differential probe vs the pre-facade commit | 15/15 sections identical, 1943 observable points |
differential probe vs parent 99ff4f307 |
14/15 identical; key_order delta unchanged (the two response_id entries) |
| duplex + protocol + fullduplex suites | 779 passed |
| new facade layering test | included above |
| mypy 3.10, isolated cache, every touched file | identical error set to 99ff4f307 |
| L3 merge gate MiniCPM-o 4.5 Duplex | 11 passed, 1 deselected, 7m48s |
|
@codex review |
|
@Gaohan123 PTAL |
can the now is too confusing, we got 3 sets of Command/Event definitions: |
linyueqian
left a comment
linyueqian
left a comment
There was a problem hiding this comment.
Reviewed at 393b21c2 (static read against main; nothing executed). I checked the move mechanically before reading: every public top-level name the six engine/duplex modules defined on main still resolves at this head (defined or re-exported), each __all__ is a superset of main's, and protocol/realtime/audio.py is byte-identical to the old engine/duplex/audio.py. The bodies that did change are the intended seams: decode_audio_append is the old build_append_audio raising RealtimeProtocolError, which the duplex wrapper re-raises as DuplexCommandError with the same codes, and both translate_realtime_command and DuplexSessionConfig.from_realtime now go through one validate_session_payload with DUPLEX_REALTIME_CAPABILITIES, so MiniCPM-o's session.update acceptance and error codes are unchanged. The only wire delta I could find is key order on response.created / response.done (response_id now follows response), same keys and values. Nothing under vllm_omni/protocol/ imports the engine, entrypoints, torch or the VAD backend, and the runner reaches the audio codec through protocol.duplex, not entrypoints.
Three inline suggestions, none blocking: the parent lookup by type(self).__name__ in the Tier 2 events breaks external subclasses, and the two new guard tests cover less than their names promise. amy's thread on the realtime_commands.py module docstring still applies at this head (it says parsing, validating and decoding a client event live in protocol.realtime, but the event-type dispatch is still translate_realtime_command here); worth trimming with the next push. General lane green at 393b21c2; RTD still pending. Approving.
Merge note for the maintainers: this and #7644 (approved, held on a policy question) conflict in engine/duplex/events.py and docs/design/fullduplex.md in either order, so whichever lands second needs a rebase. #7585 applies cleanly on top of this branch.
| # Explicit parent call, not ``super()``: ``@dataclass(slots=True)`` | ||
| # rebuilds the class, so the zero-argument form's ``__class__`` cell | ||
| # points at the pre-slots class and is no longer in the MRO. | ||
| base = realtime_events.__dict__[type(self).__name__]._wire_fields(self) |
There was a problem hiding this comment.
[suggestion] Looking the parent up by type(self).__name__ works for these three classes but breaks any subclass: class CustomCommitted(InputCommitted) raises KeyError('CustomCommitted') from to_realtime(), where main serialised it fine. The slots rebuild only breaks zero-argument super(), not an explicit parent reference, so realtime_events.InputCommitted._wire_fields(self) (and likewise at lines 144 and 158) keeps the intent and stays subclass-safe.
| # `docs/serving/realtime_duplex_api.md` sorts every message into three tiers. | ||
| # Tier 1 is pure OpenAI and lives in protocol/realtime; Tier 2 (OpenAI names | ||
| # carrying our extensions) and Tier 3 (ours alone) live in protocol/duplex. | ||
| _TIER1_EVENTS = ("SessionUpdated", "ItemAdded", "ItemCreated", "TextDelta", "OutputItemAdded") |
There was a problem hiding this comment.
[suggestion] The identity checks cover a sample of the event classes (about 18 of 42), so a copied class AudioDone in the engine shim would pass both this test and the facade test, which only compares export names. Iterating protocol.realtime.events.__all__ and asserting getattr(engine_events, name) is getattr(protocol_events, name) for every name pins single-source across the whole vocabulary at no extra maintenance cost.
| for node in ast.walk(ast.parse(source.read_text())): | ||
| if isinstance(node, ast.Import): | ||
| names.update(alias.name for alias in node.names) | ||
| elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: |
There was a problem hiding this comment.
[suggestion] node.level == 0 skips relative imports and only the parent module of a from import is recorded, so a lazy from vllm_omni import engine inside a helper passes the source scan, and the subprocess check never calls that helper. Recording node.module + '.' + alias.name and resolving relative imports against the file's package closes both holes; the pre-commit change in this PR only moves base64 allowlist entries and does not enforce this boundary.
|
@chickeyton, the engine-side module names make this look like three separate Command/Event implementations. I checked the call path again and need to correct my earlier explanation.
These are visible in The naming concern with The current declarations are:
There is no need to preserve the old engine import paths just for in-tree callers. The event-category sets are used only by Since #7640 has merged, the proposed follow-up is:
The validation target is unchanged client-visible events and errors, session behavior, and internal payloads consumed by the runner. Tests tied to the old imports or inheritance layout should migrate with the code while retaining their behavioral assertions. These are follow-up changes, not changes already included in #7640. |
…engine copies Follow-up to vllm-project#7640. Three overlapping sets of command/event definitions (protocol/realtime, protocol/duplex, and commands.py / events.py / realtime_commands.py / realtime_events.py / audio.py in engine/duplex) collapse into two layers: * protocol/realtime: the OpenAI v1/realtime command and event classes and their helpers, unchanged. * protocol/duplex: DuplexCommand / DuplexEvent as aliases of RealtimeCommand / RealtimeEvent, the Tier 2 extensions, the Tier 3 duplex-only classes, and the wire decoder (decode_duplex_command, build_append_audio, DUPLEX_COMMAND_TYPES) moved in from the engine. Consumer capabilities are injected instead of imported, and the decoder raises RealtimeProtocolError; DuplexCommandError is removed. The engine keeps only its own representation under non-protocol names: engine/duplex/mailbox.py (DUPLEX_REALTIME_CAPABILITIES, command_from_realtime, mailbox_payload / mailbox_channel replacing the payload() methods and mailbox type on the removed parallel command classes) and engine/duplex/projection.py (renamed realtime_events.py). The epoch-filter sets move into session/emitter.py and duplex_response_format into config.py, their only consumers. The duplex engine, entrypoints and clients now import protocol.duplex only; tests and docs are repointed and the facade / single-source / import-boundary tests assert the new layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…engine copies Follow-up to vllm-project#7640. Three overlapping sets of command/event definitions (protocol/realtime, protocol/duplex, and commands.py / events.py / realtime_commands.py / realtime_events.py / audio.py in engine/duplex) collapse into two layers: * protocol/realtime: the OpenAI v1/realtime command and event classes and their helpers, unchanged. * protocol/duplex: DuplexCommand / DuplexEvent as aliases of RealtimeCommand / RealtimeEvent, the Tier 2 extensions, the Tier 3 duplex-only classes, and the wire decoder (decode_duplex_command, build_append_audio, DUPLEX_COMMAND_TYPES) moved in from the engine. Consumer capabilities are injected instead of imported, and the decoder raises RealtimeProtocolError; DuplexCommandError is removed. The engine keeps only its own representation under non-protocol names: engine/duplex/mailbox.py (DUPLEX_REALTIME_CAPABILITIES, command_from_realtime, mailbox_payload / mailbox_channel replacing the payload() methods and mailbox type on the removed parallel command classes) and engine/duplex/projection.py (renamed realtime_events.py). The epoch-filter sets move into session/emitter.py and duplex_response_format into config.py, their only consumers. The duplex engine, entrypoints and clients now import protocol.duplex only; tests and docs are repointed and the facade / single-source / import-boundary tests assert the new layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…gine copies Follow-up to vllm-project#7640. Three overlapping sets of command/event definitions (protocol/realtime, protocol/duplex, and commands.py / events.py / realtime_commands.py / realtime_events.py / audio.py in engine/duplex) collapse into two layers: * protocol/realtime: the OpenAI v1/realtime command and event classes and their helpers, unchanged. * protocol/duplex: DuplexCommand / DuplexEvent as aliases of RealtimeCommand / RealtimeEvent, the Tier 2 extensions, the Tier 3 duplex-only classes, and the wire decoder (decode_duplex_command, build_append_audio, DUPLEX_COMMAND_TYPES) moved in from the engine. Consumer capabilities are injected instead of imported, and the decoder raises RealtimeProtocolError; DuplexCommandError is removed. The engine keeps only its own representation under non-protocol names: engine/duplex/mailbox.py (DUPLEX_REALTIME_CAPABILITIES, command_from_realtime, mailbox_payload / mailbox_channel replacing the payload() methods and mailbox type on the removed parallel command classes) and engine/duplex/projection.py (renamed realtime_events.py). The epoch-filter sets move into session/emitter.py and duplex_response_format into config.py, their only consumers. The duplex engine, entrypoints and clients now import protocol.duplex only; tests and docs are repointed and the facade / single-source / import-boundary tests assert the new layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: chickeyton <ngton2014@gmail.com>
…project#6592 P0a) (vllm-project#7640) Signed-off-by: psv666 <2693925048@qq.com> Signed-off-by: Matthieu Laneuville <matthieu.laneuville@surf.nl>
…project#6592 P0a) (vllm-project#7640) Signed-off-by: psv666 <2693925048@qq.com>
8 participants
Purpose
RFC #6592 P0a: give the model-agnostic OpenAI Realtime protocol layer a reusable home, so P0b (Qwen3-Omni GA) can reuse it instead of growing a second parser/session/event codec. Behaviour-preserving; no Qwen3 behaviour is added.
Depends on #7413, which is merged — this branch is one commit on top of it.
What #7413 already fixed, and what was left
#7413 moved the codec out of MiniCPM and out of serving, and gave the duplex session one owner in the engine. Three of P0a's five goals were already met by that. What was left is that the codec landed inside
vllm_omni/engine/duplex/, a package whose job is engine-resident duplex sessions:translate_realtime_commandreached intoengine/duplex/turn_detection(and so the Silero VAD backend) to validatesession.update;Change
Move the stateless, model- and runtime-agnostic half into a layer-neutral
vllm_omni/protocol/realtime/:formats.pysession.pyRealtimeInputDefaults(the defaults an append inherits)items.pyaudio_input.pydecode_audio_append()→RealtimeAudioAppend; client speech hintsaudio.pygit mv, unmodified)errors.pyRealtimeProtocolError,REALTIME_ERROR_TYPES_BY_CODEcapabilities.pyRealtimeProtocolCapabilities+validate_session_payload()RFC #6592 proposes
entrypoints/openai/realtime/for this. #7413 rules that out: it would put a protocol layer back in serving, and makeengine/duplex/session/runner.pyimportentrypoints. A shared layer that both the engine and a future serving-side surface depend on has to belong to neither, so it sits besideconfig/,inputs/andoutputs/.What deliberately did not move
DuplexCommand/DuplexEventkeep [Core][Frontend] Unified Full-duplex Framework #7413's typed boundary. They also carry duplex extensions (Listen,Speak,OverlapDecision,TurnEvent) that do not belong in a shared GA vocabulary.RealtimeProjectionStateandDuplexSessionRunnerstay engine-side. The projection is session state, and the engine owns session state.DuplexModelPluginstays the MiniCPM integration seam, untouched.engine/duplex/realtime_commands.py(887 → 389 lines) becomes the duplex binding of the shared codec: whichDuplexCommanda decoded event becomes, plusduplex_response_formatandDUPLEX_REALTIME_CAPABILITIES. The names it used to define are re-exported from it so existing importers and tests keep working, andengine/duplex/audio.pyis a re-export shim for the same reason. There is one implementation of each behaviour, asserted by identity.RealtimeModelAdapteris three quarters already builtMapping the RFC's proposed interface onto the post-#7413 tree:
build_promptDuplexModelPlugin.plan_append+ the model'sstage0start_responseDuplexSessionRunner+DuplexStagePort.submitcancelDuplexDataPlane.close_streamsupports(turn_detection, formats)Adding an interface over the first three would be a second model-integration framework. Only the fourth is missing and is genuinely protocol-layer work, so it lands as
RealtimeProtocolCapabilities: two format sets and one validator hook that the session check consults. Duplex suppliesDUPLEX_REALTIME_CAPABILITIES, which defers to the existing turn-detection validator through a lazy import, so the VAD backend is still not loaded until a session object arrives.Ownership (RFC #6592 open question 8)
A new top-level package with no CODEOWNERS rule falls to the repository-wide fallback, which would make the wire codec's reviewers an accident rather than a choice. This proposes an answer:
docs/design/module/entrypoints.mddeclares the path and its existing owners review it, withENTRY-INV-002stating the import rule the boundary test enforces.Reasoning: deciding what a client may put on the wire is that document's subject;
vllm_omni/errors.pyandinputs//outputs/are already owned by contract documents rather than by a consumer; and two of those owners have history on the Qwen3 Realtime connection that P0b will target, so the future second consumer has review rights on day one. It is deliberately notfullduplex.md, so a non-duplex consumer never needs duplex review to change the codec. Ownership here is review scope, not a dependency.Maintainers: please reassign if you would rather this graduate to its own module document now. The expectation recorded in the doc is that it graduates when P0b gives it a second consumer, with owners drawn from both.
Update: the class split (review round 1)
@chickeyton and @natureofnature both asked for the other half of the codec, and they were right — extracting only the stateless helpers would have left P0b re-declaring ~40 event classes and their rendering, which is the larger half. My original acceptance criterion 12 claimed otherwise; it did not hold. Fixed here.
The classes are split, not relocated, per @natureofnature's constraint. The proof is in the command table: 4 of 17 commands have a mailbox channel that differs from the client event (
session.updateand all threeconversation.item.*travel onturn.signal), so moving the classes wholesale would have carried engine-internal semantics into the shared layer.to_realtime()protocol/realtime/**(GA),protocol/duplex/**(our extension)payload()mailbox rendering + the mailboxtypeengine/duplex/commands.pyEach concrete engine command pairs the two —
class AppendAudio(DuplexCommand, _realtime_wire.AppendAudio)— sopayload()stays a method andisinstanceholds against both bases. That is why the 24 existingcommand.payload()assertions intest_duplex_commands.pypass unchanged.engine/duplex/events.pyis now a re-export shim keepingDuplexEvent(the same object asRealtimeEvent) for the 18 files using that name, plusDOMAIN_TERMINAL_EVENTS/MODEL_OUTPUT_EVENTS— the runner's epoch-filter policy over internal event types, not wire contract.Review defects fixed
RealtimeProtocolCapabilitiesdocstring claimed the defaults only acceptturn_detection: null;validate_turn_detection=Noneactually skips validation. Reworded, and it now tells consumers they must supply their own validator.Taxonomy (three tiers)
Per
docs/serving/realtime_duplex_api.md, which @chickeyton pointed at. A first pass used atwo-bucket GA/omni split and mis-filed seven Tier 2 classes into
protocol/realtime; corrected.INFO 09-17 11:52:06 [patch.py:252] NVFP4 W4A4 weight_scale NaN-clamp: installed.
INFO 09-17 11:52:10 [patch.py:503] inductor factorable-divisibility patch: installed.
INFO 09-17 11:52:10 [patch.py:568] [cumem-cuda] CuMemAllocator._python_free_callback patched: asleep guard extended to all platforms.
Per the three tiers in
docs/serving/realtime_duplex_api.md. A Tier 2 class subclasses its Tier 1 twin and declares only the added fields.Events
conversation.item.addedItemAddedprotocol/realtimeconversation.item.createdItemCreatedprotocol/realtimeconversation.item.doneItemDoneprotocol/realtimeconversation.item.input_audio_transcription.completedInputTranscriptionCompletedprotocol/realtimeconversation.item.retrievedItemRetrievedprotocol/realtimeinput_audio_buffer.clearedInputClearedprotocol/realtimeinput_audio_buffer.speech_startedSpeechStartedprotocol/realtimeinput_audio_buffer.speech_stoppedSpeechStoppedprotocol/realtimeoutput_audio_buffer.clearedOutputAudioClearedprotocol/realtimerate_limits.updatedRateLimitsUpdatedprotocol/realtimeresponse.content_part.addedContentPartAddedprotocol/realtimeresponse.content_part.doneContentPartDoneprotocol/realtimeresponse.function_call_arguments.deltaFunctionCallArgumentsDeltaprotocol/realtimeresponse.function_call_arguments.doneFunctionCallArgumentsDoneprotocol/realtimeresponse.output_audio.doneAudioDoneprotocol/realtimeresponse.output_audio_transcript.deltaTranscriptDeltaprotocol/realtimeresponse.output_audio_transcript.doneTranscriptDoneprotocol/realtimeresponse.output_item.addedOutputItemAddedprotocol/realtimeresponse.output_item.doneOutputItemDoneprotocol/realtimeresponse.output_text.deltaTextDeltaprotocol/realtimeresponse.output_text.doneTextDoneprotocol/realtimesession.updatedSessionUpdatedprotocol/realtimeconversation.item.deletedItemDeletedprotocol/duplexdetailsconversation.item.truncatedItemTruncatedprotocol/duplexdetailserrorErrorEventprotocol/duplexextrainput_audio_buffer.committedInputCommittedprotocol/duplexdetailsresponse.createdResponseCreatedprotocol/duplexresponse_idresponse.doneResponseDoneprotocol/duplexresponse_idresponse.output_audio.deltaAudioDeltaprotocol/duplexformat,metadata,sample_rate_hzsession.createdSessionCreatedprotocol/duplexattachment_generation,resume_tokenduplex.rawDuplexRawEventprotocol/duplexoverlap.decisionOverlapDecisionprotocol/duplexplayback.acknowledgedPlaybackAcknowledgedprotocol/duplexresponse.listenListenprotocol/duplexresponse.speakSpeakprotocol/duplexsession.closedSessionClosedprotocol/duplexsession.expiredSessionExpiredprotocol/duplexsession.heartbeat_ackSessionHeartbeatAckprotocol/duplexsession.replacedSessionReplacedprotocol/duplexsession.resumedSessionResumedprotocol/duplexsession.resync_requiredSessionResyncRequiredprotocol/duplexturn.eventTurnEventprotocol/duplexTier 1: 22 · Tier 2: 8 · Tier 3: 12 = 42
Commands
conversation.item.createCreateItemprotocol/realtimeturn.signalconversation.item.deleteDeleteItemprotocol/realtimeturn.signalconversation.item.truncateTruncateItemprotocol/realtimeturn.signalinput_audio_buffer.clearClearInputprotocol/realtimeinput_audio_buffer.clearoutput_audio_buffer.clearClearOutputAudioprotocol/realtimeoutput_audio_buffer.clearresponse.cancelCancelResponseprotocol/realtimeresponse.cancelresponse.createCreateResponseprotocol/realtimeresponse.createsession.updateUpdateSessionprotocol/realtimeturn.signalinput_audio_buffer.appendAppendAudioprotocol/duplexaudio_end_ms,duration_ms,format,hints,is_speech,sample_rate_hz,video_framesinput_audio_buffer.appendinput_audio_buffer.commitCommitprotocol/duplexcreate_response,is_speech,realtime_item_idinput_audio_buffer.commitbarge_inBargeInprotocol/duplexbarge_ininput.cancelCancelInputprotocol/duplexinput.cancelinput.text.appendAppendTextprotocol/duplexinput.text.appendplayback.ackAckPlaybackprotocol/duplexplayback.acksession.closeCloseSessionprotocol/duplexsession.closesession.heartbeatHeartbeatprotocol/duplexsession.heartbeatturn.signalSignalTurnprotocol/duplexturn.signalTier 1: 8 · Tier 2: 2 · Tier 3: 7 = 17
Bold mailbox = the session runner's channel differs from the client event (4 of 17) — the reason the mailbox half stays engine-side.
Update: one door (review round 2)
@chickeyton's diagram is a chain —
protocol/realtime<-protocol/duplex<- duplex engine — and the middle link only earns its keep if the last arrow is the only one. Ten modules were reaching straight past it:engine/duplex/{audio,commands,config,events,realtime_commands,realtime_events}.py,session/runner.py,entrypoints/duplex/{serving,realtime_input}.pyandInlineDuplexClient.Now
protocol/duplexis the whole duplex vocabulary and the only door:protocol/duplex/commands.pyprotocol/duplex/events.pyDuplexEvent = RealtimeEventprotocol/duplex/errors.pyRealtimeProtocolErrorprotocol/duplex/__init__.pygrep -r "protocol\.realtime" vllm_omni/engine vllm_omni/entrypoints vllm_omni/clientsnow returns nothing, andtests/protocol/duplex/test_duplex_protocol_facade.pykeeps it that way by AST scan over the consumer packages, plus asserts a re-exported Tier 1 name is the Tier 1 object rather than a second declaration.The payoff is @NickCao's issue:
convert_input_audio_with_ratehardcodes MiniCPM-o's 16 kHz. When that becomes client-rate → model-native, the override lands inprotocol/duplexand no call site moves. Before this, ten files.type DuplexCommand = RealtimeCommand— same idea, different spellingThe PEP 695
typestatement is Python 3.12+; this repo isrequires-python = ">=3.10,<3.14"with amypy-3.10commit hook, so on 3.10/3.11 it is aSyntaxError, not a degraded import. It also yields aTypeAliasTyperather than a class, andDuplexSessionRunner._on_commanddispatches entirely onisinstance. Plain assignment is used instead.It applies to
DuplexEventand not toDuplexCommand, and that asymmetry is the same reasonengine/duplex/commands.pystays. An event has no engine-internal half — what the session emits is exactly what the socket sends. A command has one:That dictionary never reaches a socket — it goes to the session runner's mailbox, via ten call sites all inside
engine/duplex/. Four of the seventeen render ontoturn.signal, which is not their client event type. Moving it intoprotocol/duplexwould hand a Qwen3 GA consumer a mailbox vocabulary it has no runner for, which is what @amy-why-3459 and @natureofnature both objected to. A test states the asymmetry so it is not later "fixed" in the wrong direction.Test Plan
Behaviour preservation is the acceptance criterion for P0a, so it is measured rather than asserted. Every moved function body is unchanged; the new code is the constructor lift out of
build_append_audioand a session check that runs the same two validations in the same order with the same codes.A differential probe dumps everything a duplex client can observe and is run against a worktree pinned at this commit's parent (#7413's squash merge) and against this branch, each run printing which
vllm_omni/__init__.pyit loaded:event_id; 460 of them error paths)RealtimeInputDefaults/DuplexSessionConfig.from_realtime/apply_realtime_update×2 /ResponseCreateOptions×2project_internal_eventreplayresolve_*callsTest Result
1943 probe points. 14 of the probe's 15 sections are byte-identical to the parent commit. The 15th,
key_order, differs in exactly two entries:ResponseCreatedandResponseDonerender the same keys and values in a different order, becauseresponse_idis a Tier 2 extension and now followsresponsein field order. The probe records key order specifically so this is visible.Coverage was extended for the class split: every one of the 42 event classes and 17 command
classes is constructed with populated fields and its wire JSON plus mailbox
payload()dumped, closing an 8-class gap the earlier probe had (
OverlapDecision,PlaybackAcknowledged,SessionExpired,SessionHeartbeatAck,TurnEvent,InputTranscriptionCompleted,OutputAudioCleared,TextDone).(Correction: the description previously said 1879 points. The real pre-extension count was
1821 — an arithmetic error of mine. The md5 identity was never affected.)
pytest tests/protocol/ tests/engine/duplex/ tests/engine/test_duplex_{orchestrator,omni_engine,import_boundary}.py tests/entrypoints/duplex/ tests/entrypoints/test_duplex_omni.py tests/clients/ tests/e2e/features/fullduplex/ tests/model_executor/models/minicpmo_4_5/duplex/ tests/entrypoints/openai/test_duplex_session_attachment.py tests/entrypoints/openai_api/test_duplex_{api_server,capability}.py tests/examples/test_minicpmo_realtime_duplex_simple_demo.pypytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'advanced_model and cuda' --run-level 'advanced_model'11/12 (1 deselected)on both — nothing silently skippedmypy --python-version 3.10on every touched library fileFound 46 errors— byte-identical error set to99ff4f307, none introduced by the new packagespre-commit runon the changed filesmypy-3.10aboveExisting duplex tests pass unchanged; no wire expectation was edited. The four new test files are structural only: the import direction (AST scan of every protocol source, so a lazy import inside a function body is caught, plus a subprocess import-delta check), the capability seam (check order, error codes, per-consumer format narrowing), identity assertions that a copy-and-fork cannot satisfy, and the one-door layering rule.
Not run here: Qwen3-Omni and the non-duplex MiniCPM-o 4.5 merge steps — neither reaches the Realtime codec, and
ci_source_file_dependencies.ymlroutesvllm_omni/protocol/to the duplex step.Out of scope
No P0b or P1 behaviour: no Qwen3 adapter or prompt builder, no
?profile=openai-realtime/?profile=qwen3-legacy/ profile inference, no--realtime-profile, no new Qwen3 route dispatch, no 24 kHz GA audio behaviour, no Qwen3response.create/response.cancel, no image/video input, no/v1/video/chat/streammigration, no server-VAD change, no tool-calling change, no change to MiniCPM turn-taking, and no change to any duplex event name or payload shape.This is the extraction only — please do not read it as Qwen3 GA support or as OpenAI Realtime compatibility.
Unrelated observation, not fixed here
docs/design/fullduplex.mdlistsentrypoints/duplex/chat_completions.py→DuplexChatCompletionsAdapterin its package layout, but neither the file nor the class exists onmain(git grep DuplexChatCompletionsAdapterfinds nothing). Looks like a doc line that outlived the feature; left alone as out of scope.