fix(minicpmo45): deadline-align native duplex silence continuation

[Tiagosf00]

Summary

Fixes #7063.

Native duplex currently waits for a full chunk_period after each output chunk has already been processed. With a 1-second period and around 400 ms of pipeline work, audio chunks arrive roughly every 1.4 seconds.

This PR counts processing time as part of the period:

Before: processing time + chunk period
After:  previous submission time + chunk period

Solution

The scheduler now keeps a monotonic deadline for each session:

• The first continuation is scheduled from the previous native input submission.
• Later deadlines advance from the previous deadline, preventing accumulated drift.
• The scheduler sleeps only for the time remaining before the deadline.
• After a long stall, one continuation is submitted immediately and the schedule restarts from that submission.
• Real user input starts a new deadline chain.
• Turn completion and session cleanup clear the timing state.

The submission time is captured before the runtime call, but the timing state is updated only after the append succeeds and the session epoch is still valid. This update happens before returned output events are processed, allowing a terminal event to clear the state without it being restored afterward.

The input chunk duration and silence payload are unchanged.

Validation

Current-main NPU comparison

Tested with vLLM 0.28.0 using the same 3.73-second input, reference audio and client settings.

Metric	Main only (b78c31ed)	PR (122861e)
Post-commit gaps	1006 / 704 / 1700 ms	557 / 258 / 752 / 990 ms
Mean gap	1137 ms	639 ms
Audio chunks	5	5
Transcript	Same	Same

Only gaps after input_audio_buffer.committed were included. The number of measured gaps differs because some output arrived before the commit marker.

The PR reduced the observed post-commit mean from 1137 ms to 639 ms without changing the number of chunks or the transcript.

CUDA comparison

An independent test on an RTX 6000 Ada found the same direction:

Metric	Main	Main + PR
Post-commit mean gap	1359 ms	1004 ms

That test used a different input sample and response length, so its absolute numbers should not be directly compared with the NPU results. It produced the same 20 chunks and transcript in both CUDA runs, with byte-identical output WAV files.
Thanks to @twu3202.

Tests

Added CPU coverage for:

• Deadline calculation and drift prevention
• Long-stall recovery
• Timing-state cleanup
• MiniCPM-o and PersonaPlex session-state compatibility
• Runtime acceptance before terminal-output handling
• Failed appends not updating timing state

Experiments after #7413

All tests were done using NPUs.

Comparison before and after PR/#7413.

config	stream_rtf
New main + re-based fix (current head)	0.99 (0.96–1.06)
New main (After #7413)	1.66 (1.64–1.68)
Old main + fix	0.985 (0.925–1.082)
Old main baseline	1.80 (1.53–1.90)

Before rebase (c5a97a51):

run	stream_rtf	stream_ttfp_ms	audio turns
ported-1	1.00735	4140.8	4
ported-2	0.94452	2489.4	4
ported-3	0.92549	2400.2	4
mean	0.959	~3010	4

Current PR (67101c9c):

run	stream_rtf	stream_ttfp_ms	audio turns
head-1	1.05604	4322.5	4
head-2	0.96438	2538.7	4
head-3	0.96369	2512.5	3
mean	0.995	~3124	3–4

---

Tiago Fernandes - Huawei - AI Innovation Lab

[Sy0307]

RFC #6494 tracks this as a P2 cadence follow-up for #7063. The reported fixed silence continuation adds roughly 400 ms per output interval, but the current PR is still draft, DCO-blocked, and its evidence is not yet current-main GPU/NPU evidence. Please preserve the distinction between cadence improvement, audio fidelity, and steady-state RTF.

[twu3202]

Current-main CUDA data point: RTX 6000 Ada 48 GB, vLLM 0.28.0, vLLM-Omni main @ 3204a0b, MiniCPM-o 4.5 from the Hub, head a8f3d61 applied as a diff (applies cleanly), single-GPU three-stage deployment, realtime_duplex_demo.py --input-wav <12 s speech clip> --ref-audio HT_ref_audio.wav with the default 200 ms client chunks and real-time pacing. Numbers are client-side arrival intervals between consecutive response.audio.delta (1.0 s chunks) of the same response: the first 6 gaps fall while the input is still streaming, the last 9 after input_audio_buffer.committed, i.e. once silence continuation drives the units.

	input still streaming (6 gaps)	after commit, silence continuation (9 gaps)
main	933–1046 ms, mean 992	1277–1438 ms, mean 1359
main + this PR	944–1043 ms, mean 994	906–1047 ms, mean 1004

So on CUDA the ~1.35 s cadence shows up only in the silence-continuation regime (consistent with ≈ 0.35 s of pipeline work plus the full 1 s sleep), and the PR brings it back to the chunk period. Output is unchanged: same 20 chunks and transcript in both runs, output.wav byte-identical, response done 3.2 s earlier. tests/entrypoints/duplex/ on main + this PR: 58 passed (38 existing + 20 new; needs pytest-asyncio from [dev]).

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

@Tiagosf00, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[Tiagosf00]

Self-Review

The implementation is very simple, we have two time variables:

1. last_native_submit_monotonic: Time right before the last accepted input append was submitted.
2. silence_deadline_monotonic: Expected deadline for the next continuation.

And two functions:

1. _on_append_accepted(): updates those two variables above.
2. compute_silence_continuation_deadline(): computes how much time we should sleep in order to meet the deadline, and calculates the next deadline by adding the chunk period (1s for MiniCPM-o) to the previous deadline, which doesn't accumulate drift. Also in this function you find the logic to readjust the deadline after a long delay.

I also added compatibility tests for PersonaPlex since this pipeline is shared by different models.

The last commit I made was just cleaning up some confusing variable names and unnecessary conditions made by AI. I also left some AI-generated comments in the code; let me know if you think they’re unnecessary.

I've been manually testing this model in the full-duplex mode using the UI, but the results are still not good. I've already tested this model in other frameworks and the quality and responsiveness of the output were much better, so there are still bugs to find out in this implementation. This sleep issue was just a simple one I found. Also, in an earlier version of the main branch, when vllm 0.25 was still used, it used to work better, so I suspect there was some regression. This is a good open-source model and I hope to see it running smoothly on vllm-omni soon.

@Sy0307 Ready for review.

[linyueqian]

#7413 merged as 99ff4f30 and deleted the three files this fix lives in (entrypoints/duplex/session_runner.py, runtime_bridge.py, runtime_adapter.py), so this branch is now a modify/delete conflict rather than a rebase. The bug it fixes is still present on main, though: the new engine-owned scheduler in vllm_omni/engine/duplex/session/runner.py (_schedule_silence_continuation, around line 1035) still does await asyncio.sleep(chunk_period_ms / 1000) after the previous chunk has been processed, which is exactly the processing-time-plus-period cadence #7063 describes, and maybe_continue_response in session/model_channel.py is the caller that decides when a continuation is due. If you re-target the deadline chain there (previous submission plus period, advancing from the previous deadline, one immediate catch-up after a stall, reset on real user input) I am happy to review it; the design in your description carries over unchanged, only the home moved. If you would rather not carry it, say so and I will pick it up, since #7063 remains open.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 67101c9c9ac9 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[linyueqian]

Reviewed at 7d2d2f5f (the re-target onto the engine-owned scheduler, plus a merge of current main that leaves the PR's own hunks unchanged). Thank you for carrying this across #7413; the design survived intact and it lands in the right place. compute_silence_continuation_deadline is a clean pure function with the four cases the description promises: first continuation anchored to the last accepted submission, later ones advancing from the stored deadline so processing time does not accumulate, a schedule more than one period overdue collapsing to one immediate submission plus a restart from now, and a non-negative sleep in every branch. The timing state lives on the model session state, the acceptance callback commits it only when the append actually submitted, and a real (non-silence) append re-anchors the chain through _reanchor_chain. Reset points check out: barge-in and cancel are caught by the epoch check after the sleep, session close cancels the sleeping task and run_in_wire_order refuses non-open sessions, and MiniCPM-o clears both timestamps in clear_continuation. PersonaPlex has no plugin on this runner and AURA is not on it, so only MiniCPM-o changes behaviour.

Two timing interleavings survive, inline. The [important] one is the case this PR exists to get right: a continuation whose delay was computed while a real append was already in flight is queued behind that append, the real append re-anchors the chain when it is accepted, and then the queued continuation submits immediately anyway and overwrites the anchor with its pre-computed deadline, so one silence unit goes out up to a period early right after user speech. The [suggestion] is a smaller one: a stall during the sleep stores the pre-stall deadline, so the catch-up is followed by a second immediate continuation instead of one. A second [suggestion] covers the test gap both reviewers noted: the new tests exercise the arithmetic and clear_continuation() but never run _schedule_silence_continuation under a fake clock, so a revert of the sleep to a flat period would still pass.

Validation: static read of the seven-file diff against origin/main and of the scheduler path around the sleep, plus a two-model panel (grok found no production defect and the test gap; codex found the two interleavings, which I traced in the code); no PR code executed. I added ready and re-fired the general lane at this head after the merge push.

[linyueqian]

[important] The delay here is computed from the chain as it stands before the sleep, and the only guards after the sleep are append_tail is not append_tail, _real_input_waiting(), and the stale check. If a real (non-silence) append is already in flight when this runs (so append_tail is that task and _real_input_waiting() is false because the input is submitting, not waiting), the continuation is queued behind it in wire order; the real append is accepted first and _reanchor_chain sets last_native_submit_monotonic to its submit time and clears the deadline, and then the queued silence submits at once (its delay was relative to the old anchor) and _on_append_accepted overwrites the anchor with its pre-computed next_silence_deadline. Concretely with a 1 s period: last silence at 0, user audio starts submitting at 0.8, model output arrives at 1.0, the scheduler computes zero delay and queues silence behind the speech; speech is accepted at 1.1 and re-anchors, silence submits at 1.1 and stores deadline 2.0, so the next unit is due at 2.0 instead of 2.1 and the one just sent was 0.7 s early relative to the speech. Neither before_append nor the epoch check sees a timing change. Simplest fix: have _still_valid (or the acceptance callback) compare model_state.last_native_submit_monotonic against the value captured when the delay was computed and drop the continuation if it changed, letting the next output reschedule from the fresh anchor.

[linyueqian]

[suggestion] This stores the deadline that was computed before the sleep. If the event loop stalls during the sleep so the append submits well after that deadline (say the plan was deadline 2.0 and next 3.0 but submit_time is 5.0), the chain now says the next unit is due at 3.0; the following scheduling pass sees now - deadline > period, collapses to deadline = now, and submits again immediately, so a stall yields two back-to-back continuations rather than the single catch-up the docstring promises. Deriving the stored value from the actual submission when it landed late, for example next = next_silence_deadline if submit_time <= next_silence_deadline - chunk_period_s else submit_time + chunk_period_s, keeps the one-catch-up contract.

[linyueqian]

[suggestion] These cases pin the arithmetic well, but nothing here runs _schedule_silence_continuation itself: with a fake monotonic clock and a patched asyncio.sleep, a test that drives two continuations then a real append and asserts the recorded sleeps and the re-anchored deadline would catch both interleavings above and would fail if the scheduler ever went back to a flat sleep(chunk_period). Right now that revert passes this file, and removing _reanchor_chain fails nothing.

[linyueqian]

Re-reviewed at 67101c9c. All three items from the first pass are addressed. The interleaving gap is closed the right way: _schedule_silence_continuation snapshots last_native_submit_monotonic after the wait for pending silence and _still_valid now refuses the unit when that anchor has moved, so a real append accepted between planning and submission re-anchors the chain instead of letting an early silence unit land on top of it. The acceptance callback clamps a late submission (submit_time > next_silence_deadline stores submit_time + chunk_period), so a stalled event loop no longer leaves the chain pointing at a deadline already in the past. The now-unused silence_continuation and next_silence_deadline fields on AppendAttempt are gone. And test_deadline_pacing_scheduler.py drives the scheduler itself with a fake monotonic clock and a simulated asyncio.sleep: normal cadence, an in-flight real append skipping the outdated unit and re-anchoring, a long stall, and a small wake-up delay preserving the planned deadline, which is exactly the coverage I asked for. Approving once the general lane is green on this head.

[Tiagosf00]

@linyueqian Thanks for the attention and review on this PR.