Repository navigation
[Model] Prefetch the MiniCPM-o 4.5 Code2Wav reference during Stage-2 … - #8198
Conversation
…idle steps On a reference-cache miss, Code2Wav prepares the request's reference audio inside the chunk-0 forward, on the first-audio-packet path (~228 ms on H100). The async-chunk prewarm placeholder reaches Stage 2 about 300 ms earlier, and Stage 2 spends that window in zero-token steps. Attach the reference to the Stage-2 prewarm placeholder on the initial add, deliver it to the worker through OmniSchedulerOutput.pending_request_prewarms, and let Code2Wav run materialize + prepare_prompt for it in the generation runner's idle steps (new optional model hooks on_requests_added and run_idle_prefetch). Chunk 0 then hits vllm-project#7968's warm caches; whenever the prefetch did not run, it prepares the reference inline as before. Knobs in the Stage-2 connector extra block: token2wav_ref_prefetch (default true) and token2wav_ref_prefetch_setup (default false; also warms setup_batch). H100, C1: audio TTFP -8.3%, text TTFT +2.8%; C8 unchanged; WER unchanged. Fixes vllm-project#8193 Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
This PR appears to belong to: docs/design/module/ar_runtime.md. Module owners: @tzhouam @fake0fan @Gaohan123 Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files @MinhaoLi0318, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
b6131f3 to
051f264
Compare
Omni ReviewBot triage noteAutomated triage of commit
These are automated triage suggestions only — the final decision belongs to the maintainers. |
|
I ran the comparison on a different GPU, in case numbers from a second setup help. This was one RTX 6000 Ada (48 GB) with vLLM 0.30.0, comparing 6715c57 with 051f264, on MiniCPM-o-4_5 @
Each ran as 3 blocks, with the arm order shuffled in each block and a fresh server for every run. The default yaml doesn't fit on 48 GB, so every arm used the same variant: stage memory 0.62 / 0.13 / 0.15 and a 6 GiB stage-0 KV cache. Mean [min–max] over the 3 blocks, in ms:
At C8 the median improved. The mean was 32 ms higher and p99 was about 400 ms higher, and the ranges don't overlap for either. I wonder if this is the limitation you mention, with another request's chunk 0 waiting behind a phase at C > 1. I didn't measure how often phases ran at C8. Seed-TTS WER (block 1) was 0.0115 on main and 0.0081 with the patch; given the run-to-run sampling differences below, I'd read that as noise. To check the mechanism, I timed the Stage-2 calls in separate, untimed C1 runs:
I also logged the codec deltas that the Talker hands to Code2Wav, to see whether the output changes.
|
A prefetch phase runs inline on the Stage-2 thread. The busy gate keeps it away from Stage 2's own live streams. With several placeholders queued, for example the initial burst at C > 1, those requests' thinker and talker work is starting on the stages that share the GPU, and a phase would compete with it. Start a phase only when exactly one reference is queued; the other records are dropped by their own chunk 0 or finish as before. H100, 5 interleaved blocks: C1 audio TTFP -11.0% vs main (-11.5% with the busy gate only); C8 means within 1% of main and of prefetch-off. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
@twu3202 thank you, this is really useful. You added a second GPU class, the chunk-0 timing that confirms the mechanism, and the WAV identity check, none of which I could produce myself. Your data confirms the C1 story:
On the C8 p99. I looked at it from three angles.
So I pushed a follow-up commit: a phase now starts only when exactly one reference is queued, on top of the existing busy gate. C1 is unaffected. At C > 1, phases no longer run during a burst; the only exception is the first arrival, if it is alone on an idle step. New H100 run: 5 interleaved blocks, shuffled order, fresh server per run. "old" is the busy-gate-only version and "new" is this branch; values are mean [min–max].
I couldn't reproduce a C8 tail regression on H100 with either version, so I can't say whether the gate fixes what you saw on the Ada card. If it's easy, could you rerun C8 there with the new commit (main vs this branch, ideally 5 or more blocks)? Since p99 at P128 is set by the 2nd and 3rd slowest requests, it would also help to know whether your slowest requests are burst requests (idx 0–7). The Stage-2 |
Omni ReviewBot routing recordAssigned Strict under experiment |
vllm-omni-review-bot
left a comment
vllm-omni-review-bot
left a comment
There was a problem hiding this comment.
Omni ReviewBot review
PR description
This change uses Stage-2 idle time to warm MiniCPM-o 4.5 Code2Wav reference-audio caches before chunk 0, so a cache miss no longer pays cold prepare_prompt on the first-audio path. On the initial async-chunk prewarm only (not streaming re-prewarm, duplex, or resumable sessions), the orchestrator attaches a flat _async_chunk_prewarm.* payload; the scheduler delivers it once, and the GPU/NPU generation runners invoke optional on_requests_added / run_idle_prefetch hooks. Default prefetch runs only materialize plus prepare_prompt for a lone waiter with no live Stage-2 stream; setup_batch prefetch stays off, and chunk 0 still prepares inline whenever prefetch did not run.
Change flow
flowchart TD
A["[EXISTING] Stage-0 add_request / async-chunk placeholder"]:::existing --> B["[CHANGED] Orchestrator initial prewarm attach"]:::changed
B --> C["[NEW] code2wav_prewarm_payload flat _async_chunk_prewarm keys"]:::new
C --> D["[CHANGED] Scheduler take/drain pending_request_prewarms"]:::changed
D --> E["[NEW] OmniRequestPrewarm + generation-runner idle hooks"]:::new
E --> F["[CHANGED] Code2Wav busy/lone-waiter prefetch and pin"]:::changed
F --> G["[EXISTING] Chunk-0 prepare_prompt / setup_batch cache hit"]:::existing
classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px
CI at
1412da42b9e5(2026-09-27T20:30:14.794974+00:00): required check(s) blocking:buildkite/vllm-omni(missing), and -5 more.
See inline comments below.
| self._setup_cache_size = int(extra.get("token2wav_setup_cache_size", 1)) | ||
| if self._setup_cache_size < 0: | ||
| raise ValueError("MiniCPM-o Code2Wav setup cache capacity must be >= 0") | ||
| # Reference prefetch: the async-chunk placeholder reaches this stage |
There was a problem hiding this comment.
[P1] Ready/merge MiniCPM-o e2e guards that this diff selects were not run
Changing vllm_omni/model_executor/models/minicpmo_4_5/ selects omni_minicpmo_4_5_function (and the same tree is in omni_minicpmo_4_5_perf / duplex). Frozen-head CI at 2026-09-27T20:30:14Z has required buildkite/vllm-omni missing, so those jobs are unverified. The PR Test Result ran local unit files plus a custom vllm bench serve Seed-TTS command, not the CI selectors. Ready step "Omni · MiniCPM-o 4.5 Offline Test" (.buildkite/cuda/test-ready.yml:165) runs pytest -s -v tests/e2e/offline_inference/test_minicpmo_4_5.py -m 'core_model' --run-level 'core_model'; the paired Online Test uses tests/e2e/online_serving/test_minicpmo_4_5.py with the same marker/run-level. Merge step "Omni · MiniCPM-o 4.5 Test" (.buildkite/cuda/test-merge.yml:127) runs both files with -m 'advanced_model and cuda' --run-level 'advanced_model'. The body does not say those guards were skipped. CPU Simple Test alignment is fine: the new files are marked core_model and cpu. Until Buildkite reports and those e2e selectors run or the skip is stated, the enabled serving path is not CI-closed.
There was a problem hiding this comment.
Thanks. I ran the four L2 selectors locally on one H100 at 1412da4, using the commands from test-ready.yml, and all passed: Offline 2, Online 3, Duplex 2, and Perf with 10/10 and 40/40 completed. Details and same-host perf numbers are in the self-review comment and the PR description. The L3 advanced_model selectors need 2x H100 for GPU Whisper, so I've left them to CI and said so in the description.
Omni ReviewBot: finding feedback[p1] Ready/merge MiniCPM-o e2e guards that this diff selects were not run — Changing This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR. |
0z5a
left a comment
0z5a
left a comment
There was a problem hiding this comment.
I reproduced a cache-ownership collision on this HEAD with two live requests sharing a reference. The control case with noncolliding IDs passes, while the colliding case evicts a still-live request's cached reference and deletes its WAV. Please separate prefetch-owner and request-owner identities and cover this case in the tests. The immediate next chunk still produced audio with the fake backend, so the demonstrated failure is premature eviction of live reference state.
| # and releases ownership only through ``_request_prompt_keys``, so a chunk-0 | ||
| # reference that differs from the prefetched one must not leave the | ||
| # prefetched entry owned by the request forever. | ||
| return f"prefetch:{state_id}" |
There was a problem hiding this comment.
owners is a set[str], so prefetch owner prefetch:foo collides with the real request ID prefetch:foo. I reproduced this at normal runtime-prompt cache capacity 1: prefetch foo with reference A, run chunk 0 for prefetch:foo with the same reference, finish foo, then run a third request with reference B. The normal cache trim evicts prefetch:foo's entry and unlinks its WAV even though that request remains in _states and _request_prompt_keys. The identical control using request ID bar retains its WAV (1 failed / 1 passed on this HEAD). Please give prefetch pins and request pins distinct owner identities, such as tagged tuples or separate sets, and add this shared-reference/ID-collision case to the tests.
There was a problem hiding this comment.
Thanks @0z5a, good catch, and thanks for the repro with the bar control. You're right: owners mixed request ids with prefetch pins spelled prefetch:<id>, so a request whose id is literally prefetch:foo aliased foo's pin. It also went the other way. When prefetch:foo finished, it released foo's pin, and at capacity 0 foo's prefetched entry could be evicted before its chunk 0.
Fixed in the new commit "Keep Code2Wav prefetch pins apart from request owners":
_RuntimePromptkeeps pins in a separatepins: set[str], keyed by the plain request id.ownersnow holds only request ids committed at chunk 0._trim_runtime_promptsevicts only entries with no owners and no pins._prefetch_owner()is gone.- Two new tests:
- Your sequence at capacity 1, parametrized over
prefetch:fooandbar. It fails on the previous head forprefetch:fooand passes now. - The reverse direction at capacity 0.
- Your sequence at capacity 1, parametrized over
I also checked the other state keyed by request id for the same kind of aliasing: the prefetch queue, the pin index, the setup-slot holder and _request_prompt_keys. That included ids reused after finish and several requests sharing one reference. A randomized check of this bookkeeping found no violations on the new code. It used mixed ids such as foo and prefetch:foo, capacities 0–3, and injected phase failures. The same harness reproduces your failure on the previous head.
On an RTX 4090 with torch 2.13.0+cu130:
- The 174 new tests pass on both CPU and GPU.
- [Model] Reduce MiniCPM-o 4.5 reference setup and Talker overhead #7968's regression list and the touched test directories match main, with the same 7 failures as on main.
- Each of the 17 bugs I injected into the production code, one at a time, makes the new tests fail. One of them is a trim that ignores pins.
|
I reran C8 with 1412da4 on the same RTX 6000 Ada: main (6715c57) vs this branch, 5 blocks, shuffled order, a fresh server per run, and the same Seed-TTS en command (128 prompts, 8 warmups) and 48 GB stage split as before. One change from my first run: both arms used The C8 tail regression doesn't show up with the new commit. Values are mean [min–max] over 5 blocks:
The p99 ranges overlap almost completely. The branch was lower in 3 blocks and higher in 2, and main's own p99 varied by about 640 ms across blocks, so I'd read this as no difference. I didn't rerun 051f264 this time, so I can't say how much of the earlier +11.8% came from the burst phases and how much was noise. On the slowest requests, I used the Stage-2 |
`_RuntimePrompt.owners` held both request ids and prefetch pins spelled `prefetch:<id>`, so a request whose id is literally `prefetch:foo` aliased request foo's pin. Releasing one released the other, and a later trim could evict a still-streaming request's reference and unlink its WAV, or evict foo's prefetched entry before its chunk 0. Keep pins in their own set keyed by the plain request id. A trim now evicts only entries with neither owners nor pins. Add tests for both directions of the aliasing. Reported in review of vllm-project#8198. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
@twu3202 thanks for the rerun, and for the tail breakdown. That settles it for me: with the lone-waiter gate, 5 blocks on your Ada card show no C8 difference (p99 −3.4% with overlapping ranges, mean −0.9%), matching the 5-block H100 run in the description. I'll keep the gate either way. It costs nothing measurable at C1 (−11.0% vs −11.5% TTFP on H100), and it removes the one path by which a phase could compete with other requests' work during a burst, even though neither of us can now tell how much of the earlier +11.8% it accounted for. Your breakdown matches what I saw on H100. Burst requests are over-represented among the slowest ones on both arms, but most of the tail comes from steady-state requests, again on both arms. So it's the workload, not the prefetch. For completeness, the latest commit (7dfe0ae) only fixes the pin/owner aliasing from @0z5a's review. It doesn't change when a phase runs, so these numbers still apply. |
|
Self-review. What I checked before asking for review:
The first offline/online run failed only in
Could a maintainer add the |
hsliuustc0106
added
core
Add a "Placeholder prewarm" subsection to docs/design/feature/async_chunk.md. It describes how downstream placeholders are submitted and the four touchpoints of async_chunk_prewarm_payload_func: pipeline config, orchestrator attach, scheduler pop into pending_request_prewarms, and the runner hooks. It also states the flat-key constraint on tensor values. State in run_idle_prefetch's docstring that the runner ignores its return value; the unit tests use it to check which phase ran. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
Thanks @hsliuustc0106! Both comments are addressed in the new commit: the prewarm contract is now documented in @natureofnature and @0z5a have reviewed it, but 0z5a's approval doesn't count toward the merge requirement. If this looks good, could you approve and add |
…ch-pr1 Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com> # Conflicts: # vllm_omni/core/sched/omni_scheduler_mixin.py # vllm_omni/engine/orchestrator.py
…32 policy vllm-project#8222 made Code2Wav's forward apply token2wav_allow_tf32 and restore the previous matmul policy afterwards. The idle prefetch runs chunk 0's preparation calls (prepare_prompt, setup_batch) outside forward, so with the knob on it would warm the caches under a different policy than the inline path. Move the policy into _matmul_policy(), apply it to both forward and the prefetch phase, and test both settings. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
Merged main again: #8184 and #8222 landed and conflicted with this PR in imports only. One interaction needed a code change. #8222 made Code2Wav's The MRv2 turn profile from #8222 runs Code2Wav on the v2 runner, which has no prefetch hooks. On that profile the prefetch simply stays off, and chunk 0 prepares the reference inline as before. The Code2Wav prefetch tests pass after the merge (62, including the new one), and ruff passes. |
vllm-omni-review-bot
left a comment
vllm-omni-review-bot
left a comment
There was a problem hiding this comment.
Omni ReviewBot review
PR description
This PR shortens MiniCPM-o 4.5 first-audio latency when Stage-2 reference audio misses the content-addressed caches by preparing that reference during async-chunk idle steps before chunk 0. The orchestrator optionally attaches a flat prewarm payload to the initial Code2Wav placeholder; the scheduler delivers it once; generation runners call optional on_requests_added / run_idle_prefetch hooks; Code2Wav runs materialize/prepare_prompt (and optional setup_batch) under busy and lone-waiter gates with separate prefetch pins. Chunk 0 still prepares inline on miss/failure, so the feature is a best-effort C1 TTFP optimization with documented concurrency limits.
Change flow
flowchart TD
A["[EXISTING] Async-chunk Stage-2 placeholder"]:::existing --> B["[CHANGED] Orchestrator attach flat prewarm payload"]:::changed
B --> C["[NEW] OmniRequestPrewarm via scheduler pop"]:::new
C --> D["[CHANGED] GPU/NPU generation idle hooks"]:::changed
D --> E["[CHANGED] Code2Wav prefetch phases + pin set"]:::changed
E --> F["[EXISTING] Chunk-0 warm cache or inline prepare"]:::existing
classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px
CI at
d57a14baa434(2026-09-30T22:13:26.428042+00:00): required check(s) blocking:buildkite/vllm-omni(missing).
No actionable findings.
🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Full review, layer by layer — approving. This is well-engineered: every optimization path has an inline fallback, every piece of new state has a release path, and the failure modes are gated rather than trusted.
Wire/orchestrator. Payload attaches only on the initial add of a non-session, non-resumable request; re-prewarm and streaming updates submit the plain placeholder (parametrized tests cover every skip path including func-raises and non-dict payloads). Passing req_state.prompt rather than the mm-stripped base_input to the payload func is the right call. Flat _async_chunk_prewarm.* keys with the tensor-stays-top-level rationale documented in data_entry_keys.py.
Scheduler. _take_async_chunk_prewarm pops only on a fresh id, strips a stale undrained payload on id reuse, and reduces an emptied payload to None so the wire shape matches a plain placeholder. Malformed payloads drop with a debug log and the request is still admitted. _wrap_omni_scheduler_output drains fully each step and delivers only ids still live; finish_requests pops the rest. The parked-session (#6670) branch is preserved exactly as an elif.
Runners (GPU + NPU mirrored). _prev_step_idle is recorded before any return path; the double-idle gate correctly avoids the first idle step after a busy one under async scheduling. on_requests_added\] runs after on_requests_finished] so a freed-and-re-added id keeps its new prewarm. Exceptions are contained with warn-once-per-hook — an EngineCore kill from a cache-warming hook would be unacceptable, and this makes it impossible.
Code2Wav. The busy and lone-waiter gates confine the prefetch to exactly the regime where the C1 gap lives (lone request, idle Stage 2), and everything else falls back to the existing inline path. Pin lifecycle is correct: released only after chunk 0 commits its own reference or finish, with pins kept separate from owners so a differing chunk-0 reference can't strand ownership; pin count capped at the LRU capacity so _trim_runtime_prompts\] can always find a victim. The TF32 refactor into _matmul_policy()] preserves forward's behavior and puts the idle prefetch under the same policy #8222 added — the interaction called out upthread is handled. batched_token2wav\] changes are a pure cache-key extraction plus a read-only has_cached_setup`.
Payload parity. code2wav_prewarm_payload\] reuses the llm2tts] extractor, so the prefetched waveform hashes to the same content-addressed keys chunk 0 will use — that parity is what makes chunk-0 a warm-cache hit rather than a second cold pass. 60s cap keeps the placeholder small.
Residual risks I accept as disclosed: prefetch defers indefinitely under any live duplex stream (records still dropped at finish), and token2wav_ref_prefetch defaults on — given the gates, the inline fallback, and the −11%/−7.2% C1 TTFP evidence on two GPUs with C8 unchanged, default-on is justified.
On merge bookkeeping: the review-gate concern from #7865 is cleared from my side. One process note — the ready label appears to be applied by a single gatekeeper (Sy0307 across recent merges), so I'm leaving it to that flow rather than adding it myself.
Resolve the conflict in minicpmo_4_5_code2wav.py with vllm-project#8443: - Keep both sets of imports, including both batched_token2wav names (PromptFeatures and row_offset_signature). - Keep forward() on _matmul_policy(), and make _matmul_policy() use vllm-project#8443's _tf32_mode(). The idle prefetch and forward still share one TF32 policy, which now follows main: on by default, with the MINICPMO_CODE2WAV_TF32 override. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
- Move the stage TF32 policy into a module-level _stage_matmul_policy(extra). forward() and the idle prefetch call it with self._extra_config(). test_vocoder_restores_tf32_policy calls forward() on a stub with only _extra_config and _forward_impl, so the _matmul_policy method failed it. It has failed since d57a14b. - In the prewarm scheduler test, stub kv_holding_waiting and deferred_waiting, as vllm-project#8459 does for the other hand-built schedulers. The vLLM 0.31 finish path reads both. Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com>
|
Merged main again (9bfd7b5). #8443 and #8459 landed, and the merge conflicted only in
Tests (CPU only, vLLM 0.31.0, torch 2.13): I ran the PR's tests plus @Sy0307 could you add the |
hsliuustc0106
added
ready
amy-why-3459
added
ready
vllm-project#8198) Signed-off-by: MinhaoLi0318 <lminhao039@gmail.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
7 participants
Purpose
Improves the MiniCPM-o 4.5 C1 audio TTFP tracked in #8193. After #6538 refreshed the baseline the nightly row passes, so this is a performance improvement, not a CI fix.
When a request's reference audio misses #7968's content-addressed caches, Stage 2 (Code2Wav) prepares it synchronously inside the chunk-0 forward, on the first-audio-packet path. On H100 this costs ~228 ms:
_materialize_runtime_prompt~8 ms,prepare_prompt~120 ms,setup_batch~100 ms. The async-chunk prewarm placeholder reaches Stage 2 about 300 ms before chunk 0, and Stage 2 spends that window in zero-token steps. This PR uses those idle steps to make the same preparation calls chunk 0 would make, so that chunk 0 hits the warm caches.Changes
engine/orchestrator.py): on the initial add only (not streaming re-prewarm, duplex or resumable sessions),_prewarm_async_chunk_stages(..., attach_prewarm_payload=True)asks the downstream stage client for a payload. It attaches the payload to the placeholder as flatadditional_informationkeys under_async_chunk_prewarm.*; the keys are flat so the tensor survives serialization. If the payload function fails or returns nothing, the placeholder is submitted exactly as today.config/stage_config.py,config/omni_config.py,engine/stage_init_utils.py,engine/stage_engine_core_client.py): a stage opts in withStagePipelineConfig.async_chunk_prewarm_payload_func(a dotted path), which is resolved on both the legacy and the typed metadata paths. MiniCPM-o sets it on Code2Wav only (code2wav_prewarm_payload, which reuses thellm2ttsreference extractor).core/sched/omni_scheduler_mixin.py,core/sched/output.py):add_requestpops the keys for new request ids and delivers them once in the newOmniSchedulerOutput.pending_request_prewarmsfield. Finish and abort drop payloads that were never delivered. MiniCPM-o's Stage 2 uses the scheduler-side chunk adapter, sopending_input_registrationsis not involved.on_requests_added(prewarms)runs right afteron_requests_finishedand only enqueues.run_idle_prefetch()runs only on a zero-token step that follows another zero-token step. It never runs on a step that has a chunk scheduled, or on the first idle step after a busy one, whose output may still be in flight under async scheduling.minicpmo_4_5_code2wav.py,batched_token2wav.py):run_idle_prefetchruns one phase for the queued request, on the same thread asforward. There are no threads, locks or side streams, because the CFM/HiFT CUDA graphs share static buffers andprepare_promptsets the process-global default dtype.prepare_prompt.batched_token2wav.pyonly movessetup_batch's cache key into_setup_cache_keyand adds a read-onlyhas_cached_setup. Its behavior is unchanged.extrablock:token2wav_ref_prefetch: defaulttrue.token2wav_ref_prefetch_setup: defaultfalse. It also warms the batch-1setup_batchfor one waiting request at a time. That lowers TTFP further but slows the Stage-0 thinker on a shared GPU (see the results), the contention studied in RFC [RFC]: Recover MiniCPM-o 4.5 C8 TTFT without losing audio performance #8136.recipes/OpenBMB/MiniCPM-o-4_5.md: a "Reference prefetch" note on both switches, their defaults and when they apply. It describes the prefetch as a low-concurrency first-audio optimization, not a batching or duplex throughput feature.docs/design/feature/async_chunk.md: a "Placeholder prewarm" subsection covering the placeholder submission and the four touchpoints ofasync_chunk_prewarm_payload_func(config, orchestrator attach, scheduler pop, runner hooks), with the flat-key constraint.Known limitation. A phase runs inline, and the default phase takes ~128 ms on a cold reference. Because of the two gates, it can only start when Stage 2 has no live stream and exactly one reference is waiting. At C > 1 the prefetch therefore mostly stays idle; the one exception is the first request of a burst, if it arrives alone on an idle step. A chunk that becomes ready mid-phase (for example a request that sent no reference) waits for that phase.
Follow-up (tracked in #8193): make the cold path itself cheaper.
prepare_promptis dominated by host work: campplus on onnxruntime CPU withintra_op_num_threads=1(~76 ms), plus a WAV write/read round trip.Test Plan
vLLM Version: 0.30.0
vLLM-Omni Commit: the unit tests and the L2 selectors below were run on
8ebef4458, which is merged with main5b3abe2f4. Later commits merge main up to9097fa654and add documentation. The merge with #8184 conflicted only in imports. One later commit changes behavior: the idle prefetch now runs under the TF32 policy that #8222 added to Code2Wav's forward (token2wav_allow_tf32, default off, so the default path is unchanged). The 5-block performance run (the first table below) was measured on this branch before the merge, based on6715c57b2. The earlier 3-block run further down was measured on mainb308e196cwith the busy gate only; its production code otherwise differs from this branch only in one merged import line, one removed typing-onlyProtocolline and comments.Unit tests, on CPU (
CUDA_VISIBLE_DEVICES=) and on GPU (H100, torch 2.13.0+cu130), for8ebef4458and for main5b3abe2f4:A mutation check injects 17 bugs into the production code, one at a time, and reruns the new tests. Examples: the idle flag uses
== 0, prefetch runs on the first idle step, no busy gate, no lone-waiter gate (or> 2), a pin that is never taken or never released, a trim that ignores pins, prewarms delivered twice, a payload attached on streaming updates, and a float64 payload.Performance setup:
503e7542.vllm_omni/deploy/minicpmo_4_5.yaml, with all three stages on one GPU.token2wav_ref_prefetch: false;token2wav_ref_prefetch_setup: true.84977d95, running the same commands as [Model] Reduce MiniCPM-o 4.5 reference setup and Talker overhead #7968. C8 uses--num-prompts 128 --max-concurrency 8 --num-warmups 8, and Seed-TTS WER was measured in block 1.Test Result
Unit tests (
8ebef4458vs main5b3abe2f4, torch 2.13.0+cu130):The 7 failures are the same set on main:
test_async_omni_engine_abort::test_abort, 2 intest_cuda_graph_wrapper, and 4 intest_batched_omni_output. A mutation check injects 17 bugs into the production code one at a time and reruns the new tests; all 17 make them fail. The 17 include removing the lone-waiter gate, loosening it to> 2, and a trim that ignores pins.ruff checkandruff format --check(v0.14.10) pass on all changed files.L2 e2e selectors. These ran on one local H100 at
8ebef4458(merged with main5b3abe2f4), using the exact commands from.buildkite/cuda/test-ready.yml. They also passed earlier at1412da42b:1412da42b). The Perf Test only asserts completion (10/10 and 40/40). Same-host interleaved runs, 3 of main and 4 of this PR, give:advanced_modelmerge selectors need 2x H100 so Whisper can run on GPU; they are left to CI.Performance, lone-waiter gate (after review). H100 SXM, vLLM 0.30.0, 5 interleaved blocks with the arm order shuffled in each block and a fresh server per run. Arms:
6715c57b2;token2wav_ref_prefetch: false, which is the same build as "new";051f264f2), labeled "old";Values are client means [min–max of block means]; percentages are relative to main.
Earlier run (busy gate only, 3 blocks, main
b308e196c). Values are the mean [min–max] over the 3 blocks; percentages are relative to main.setup_batchas well saves more TTFP but costs +16% text TTFT, so it stays off by default.Raw JSON results, server logs and scripts are attached to #8193.
AI assistance: Claude Code helped with the implementation, tests and write-up; I reviewed every changed line and ran the tests and benchmarks listed above.