[natureofnature]

Contributor intent

1. What outcome do you want to achieve?

Improve MiniCPM-o 4.5 serving throughput and latency on a single GPU while checking speech accuracy.

2. What is the highest-leverage variable for that outcome?

The cost of Code2Wav/CFM inference, measured through audio throughput, first-audio latency and RTF under concurrent serving.

3. How will this work move that variable?

Batch compatible cold-reference CFM initialization while preserving each reference's conditioning and cache state, then measure it with the official serving benchmark.

4. Whose customer or user needs does this address, and what are those needs?

MiniCPM-o 4.5 users serving concurrent speech requests on one GPU need higher throughput, predictable latency and checked audio quality.

Purpose

#9159 batches cross-reference decoding. Cold reference initialization still performs one initial CFM solve per reference. This patch groups compatible setup-cache misses and runs their initial solve together, bounded by the existing microbatch and graph batch limits.

Reference extraction, encoder initialization and speaker projection remain M=1. Cached references retain independent storage and conditioning; setup-cache hits survive eviction while a group is being prepared, and publication preserves the caller's reference order.

The optimization uses the existing CUDA fused FP32/TF32 ragged Whole-Euler path with zero frame padding. Other execution policies retain scalar setup. If a batched graph replay is refused, setup reuses the already-computed M=1 encoder/speaker outputs and retries each reference's CFM solve, allowing an existing M=1 graph to remain usable. This avoids duplicate encoding and silently switching a fused scalar solve to an unfused batched fallback when the graph budget is full.

For eligible non-slot execution, the existing reference-driven precapture now warms offset-zero setup graphs after decode/tail shapes, M=1 first, within the unchanged shared graph budget. Warmup still needs to run before measurement. Independent per-reference cache copies are retained; the documentation explains their transient-memory tradeoff.

No deployment defaults or new configuration knobs are introduced. The execution contract is updated in docs/design/minicpm_o45_mrv2_performance.md. This is separate from #9213's FP16 fused-body work.

Performance limitations: the six-point sweep at 41cb8686 improves throughput at C4–C32, while C2 regresses slightly and C32 first-audio latency increases. C1 is approximately flat in that sweep; the earlier three-point experiment's approximately 5% C1 regression remains unexplained. The review follow-up addresses repeated fallback encoding and lazy setup capture; its serving regression does not replace the earlier ABBA comparison or establish that all latency regressions are resolved.

Test Plan

vLLM Version: 0.32.0; PyTorch 2.13.0+cu130; Transformers 5.19.0; datasets 3.6.0.

vLLM-Omni Commit: baseline bea9b9e9ecb980d3cbdbf0ee061a42f325b83e75; current review fix and regression evidence at 532e9ca1145cef8b7cdd3f5334cff20c6ad9c0ac. The complete six-point serving sweep and unfused fallback check were measured at 41cb868615e8480e5d75f6b9dfc9732ccac3eddc. Historical three-point timing was collected at 03fe9c65f462a9dd3cbe87b685fa3137921124ff; separate WER and C1-only control at da8df8a98bebb1f589e211f49aa0c75c1a5219f1.

The 41cb8686 sweep includes the precision/body eligibility gate and scalar fallback on refused graph replay. Timing uses fused FP32/TF32. The earlier quality evidence remains tied to its measured revision; WER evaluation is disabled in the timing sweep and follow-up serving regression.

One H800 80 GB; MPS enabled in both arms; MRV2 turn serving; Thinker/Talker BF16; Code2Wav FP32/TF32; stage sequence limits 8/8/8; CFM microbatch 8. The same compatibility adjustment was applied outside this PR to both serving checkouts for vLLM 0.32: self.llm.get_input_embeddings(input_ids) → self.llm.embed_input_ids(input_ids) in minicpmo_4_5_omni_llm.py.

The unchanged baseline vllm bench serve client drives both servers. Input is a fixed ordered 64-request English Seed-TTS subset. Each server receives 16 warmup requests at C16, reusing the first 16 evaluation requests. Then C1, C2, C4, C8, C16 and C32 each have one discarded 64-request run and two measured 64-request runs. Server order is A1 → B1 → B2 → A2, giving four measurements per arm/concurrency. Client concurrency increases to 32 while stage sequence limits remain 8/8/8. The historical sweep used C1/C8/C16; its results are retained separately below.

Serving configuration and official benchmark commands

Set SERVER_CHECKOUT to the baseline or candidate checkout, BASE_CHECKOUT to the baseline, and MODEL_PATH to the local openbmb/MiniCPM-o-4_5 snapshot. Use the same model and Seed-TTS rows for both arms. DATASET is the prepared Seed-TTS root containing en/meta.lst and its reference WAVs; use the 64 IDs listed below in that order. WARMUP_DATASET contains its first 16 rows with the same reference paths. RUNTIME_ENV is the serving/timing Python environment and EVAL_ENV is the separate environment with the optional Seed-TTS WER dependencies. The software versions above describe the serving environment.

# bench-deploy.yaml; replace SERVER_CHECKOUT with the actual absolute path.
base_config: SERVER_CHECKOUT/vllm_omni/deploy/minicpmo_4_5_turn_mrv2.yaml
connectors:
  connector_of_shared_memory:
    extra:
      micro_batch_size: 8
      cfm_graph_precapture: true
      token2wav_float16: false
      token2wav_allow_tf32: true
stages:
- stage_id: 0
  max_num_seqs: 8
  max_model_len: 4096
  max_num_batched_tokens: 4096
  limit_mm_per_prompt: {image: 0, audio: 0, video: 0}
- stage_id: 1
  max_num_seqs: 8
- stage_id: 2
  max_num_seqs: 8
platforms:
  cuda:
    cuda_mps: true
    stages:
    - stage_id: 0
      kv_cache_memory_bytes: 4294967296
    - stage_id: 1
      kv_cache_memory_bytes: 2147483648

# Use the same available physical GPU in both server and client terminals.
export CUDA_VISIBLE_DEVICES=2
export VLLM_CLI="$(command -v vllm)"
export OMP_NUM_THREADS=8
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn TOKENIZERS_PARALLELISM=false
unset CUDA_MPS_PIPE_DIRECTORY CUDA_MPS_LOG_DIRECTORY CUDA_MPS_ACTIVE_THREAD_PERCENTAGE
PYTHONPATH="$SERVER_CHECKOUT" "$RUNTIME_ENV/bin/python" "$VLLM_CLI" serve "$MODEL_PATH" --omni \
  --deploy-config bench-deploy.yaml --trust-remote-code \
  --host 127.0.0.1 --port 18770

Save the following as extra-body.json:

{
  "modalities": ["text", "audio"],
  "chat_template_kwargs": {"enable_thinking": false, "use_tts_template": true},
  "sampling_params_list": [
    {"temperature": 0.0, "top_p": 1.0, "top_k": -1, "max_tokens": 256, "seed": 42, "detokenize": true, "repetition_penalty": 1.0},
    {"temperature": 0.8, "top_p": 0.85, "top_k": 25, "max_tokens": 2048, "min_tokens": 50, "seed": 42, "detokenize": false, "repetition_penalty": 1.05},
    {"temperature": 0.0, "top_p": 1.0, "top_k": -1, "max_tokens": 65536, "detokenize": true}
  ]
}

In a second terminal after server readiness, use a new RESULT_DIR for each server start:

export CUDA_VISIBLE_DEVICES=2
export VLLM_CLI="$(command -v vllm)"
export OMP_NUM_THREADS=8
export BENCH_PYTHON="$RUNTIME_ENV/bin/python"
export SEED_TTS_WER_EVAL=0 SEED_TTS_SIM_EVAL=0 SEED_TTS_UTMOS_EVAL=0
bench() {
  PYTHONPATH="$BASE_CHECKOUT" "$BENCH_PYTHON" "$VLLM_CLI" bench serve --omni \
    --backend openai-chat-omni --model "$MODEL_PATH" \
    --base-url http://127.0.0.1:18770 --endpoint /v1/chat/completions \
    --dataset-name seed-tts --dataset-path "$1" --seed-tts-locale en \
    --seed-tts-system-prompt 'Please read the following text out loud in English.' \
    --num-prompts "$2" --max-concurrency "$3" --no-oversample --disable-shuffle \
    --trust-remote-code --seed 42 --output-len 256 --num-warmups 0 \
    --ready-check-timeout-sec 0 --disable-tqdm \
    --extra-body "$(<extra-body.json)" \
    --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,audio_duration \
    --metric-percentiles 50,95,99 --save-result --save-detailed \
    --result-dir "$RESULT_DIR" --result-filename "$4.json"
}
bench "$WARMUP_DATASET" 16 16 warmup
for c in 1 2 4 8 16 32; do
  bench "$DATASET" 64 "$c" "c$c-discard"
  bench "$DATASET" 64 "$c" "c$c-r1"
  bench "$DATASET" 64 "$c" "c$c-r2"
done

For the separate quality run, start a fresh server for each arm, repeat the same 16-request C16 warmup, then enable the existing Seed-TTS WER evaluator and save generated audio:

export BENCH_PYTHON="$EVAL_ENV/bin/python"
export SEED_TTS_WER_EVAL=1 SEED_TTS_WER_SAVE_ITEMS=1
export SEED_TTS_EVAL_DEVICE=cuda:0
export SEED_TTS_HF_WHISPER_MODEL="$WHISPER_PATH"
export SEED_TTS_WER_SAVE_AUDIO_DIR="$RESULT_DIR/audio"
bench "$DATASET" 64 8 c8-quality

WHISPER_PATH is the local openai/whisper-large-v3 snapshot. Quality-run latency is excluded from the performance table. The C1-only diagnostic instead warms 16 requests at C1, then runs only C1 with one discarded and two measured runs per fresh server, A → B.

Fixed ordered Seed-TTS evaluation IDs (warmup reuses the first 16)

common_voice_en_19724242-common_voice_en_19724243
common_voice_en_691043-common_voice_en_691044
common_voice_en_568078-common_voice_en_568079
common_voice_en_37360566-common_voice_en_37360564
common_voice_en_20266775-common_voice_en_20266781
common_voice_en_19176854-common_voice_en_19176853
common_voice_en_25694807-common_voice_en_25694818
common_voice_en_19867710-common_voice_en_19867713
common_voice_en_21802531-common_voice_en_21802534
common_voice_en_25257658-common_voice_en_25257662
common_voice_en_17255102-common_voice_en_17255101
common_voice_en_93674-common_voice_en_93672
common_voice_en_20791751-common_voice_en_20791750
common_voice_en_64065-common_voice_en_64062
common_voice_en_21293493-common_voice_en_21293494
common_voice_en_18944281-common_voice_en_18944280
common_voice_en_20461046-common_voice_en_20461051
common_voice_en_19629167-common_voice_en_19629165
common_voice_en_19294198-common_voice_en_19294199
common_voice_en_19972684-common_voice_en_19972725
common_voice_en_203357-common_voice_en_203358
common_voice_en_590785-common_voice_en_590794
common_voice_en_25346256-common_voice_en_25346259
common_voice_en_581024-common_voice_en_581028
common_voice_en_17861489-common_voice_en_17861487
common_voice_en_23640041-common_voice_en_23640040
common_voice_en_27794266-common_voice_en_27794264
common_voice_en_20303932-common_voice_en_20303930
common_voice_en_599060-common_voice_en_599064
common_voice_en_23982302-common_voice_en_23982305
common_voice_en_25271639-common_voice_en_25271686
common_voice_en_175348-common_voice_en_175349
common_voice_en_28064064-common_voice_en_28064058
common_voice_en_19117097-common_voice_en_19117098
common_voice_en_21345624-common_voice_en_21345626
common_voice_en_19724030-common_voice_en_19724028
common_voice_en_17889905-common_voice_en_17889907
common_voice_en_24985801-common_voice_en_24985802
common_voice_en_36705534-common_voice_en_36705545
common_voice_en_23729047-common_voice_en_23729050
common_voice_en_153872-common_voice_en_153873
common_voice_en_18105426-common_voice_en_18105430
common_voice_en_20095636-common_voice_en_20095638
common_voice_en_19345360-common_voice_en_19345359
common_voice_en_24837486-common_voice_en_24837489
common_voice_en_23809069-common_voice_en_23809074
common_voice_en_21444162-common_voice_en_21444163
common_voice_en_31959919-common_voice_en_31959922
common_voice_en_19641789-common_voice_en_19641784
common_voice_en_20066900-common_voice_en_20066899
common_voice_en_205647-common_voice_en_205648
common_voice_en_20791578-common_voice_en_20791581
common_voice_en_28827679-common_voice_en_28827680
common_voice_en_36705534-common_voice_en_36705530
common_voice_en_17429166-common_voice_en_17429171
common_voice_en_71427-common_voice_en_71423
common_voice_en_22763534-common_voice_en_22763537
common_voice_en_30471104-common_voice_en_30471105
common_voice_en_24988318-common_voice_en_24988314
common_voice_en_23963776-common_voice_en_23963795
common_voice_en_20661858-common_voice_en_20661861
common_voice_en_17875915-common_voice_en_17875925
common_voice_en_19632634-common_voice_en_19632644
common_voice_en_17263752-common_voice_en_17263755

Test Result

Review follow-up at 532e9ca1

• The old refusal path fails all three encoding-count regressions (enabled refusal, wrapper disablement, and full budget). With this change, each reference is encoded once, including later groups that hit a memoized refusal.
• The six-point sweep logs show 25/32 Whole-Euler graphs for each baseline visit and 28/32 for each candidate visit. In candidate B1, M=2 setup capture occurred during the second measured C2 run: host warmup plus capture took 1.051 seconds. That is a measured source of latency, but does not explain all reported regressions. The logs show no budget saturation; their last-capture counters are not whole-run fallback totals.
• Native H800 checks warm 24 decode graphs followed by four setup graphs (M=1/2/4/8), using 28/32 slots. Equal-length setup and continuation checks add no captures, eager fallbacks, or flushes after precapture. Mixed-length B=2/4/8 checks and full-budget B=3/B=8 fallback checks also pass. Cache and PCM comparisons remain within atol=1e-5, rtol=1.3e-6; the largest PCM absolute difference in these checks is 6.44e-6. These are real-weight component checks, not a new WER evaluation.
• Official vllm bench serve serving regression: C2 and C32, one discarded run plus two measured 64-request runs at each concurrency. All 384/384 requests passed, plus 16/16 disjoint warmup requests. All 28 Whole-Euler captures completed during warmup; none occurred in the six subsequent runs, and no service errors were logged.

Candidate-only means from the two measured runs (same command/configuration below; this is not a new A/B comparison):

Concurrency	Successful measured requests	Audio throughput (audio-s/s)	Mean TTFA (ms)	Mean completion RTF
2	128/128	13.305	364.27	0.1531
32	128/128	33.990	1413.56	0.9146

Concurrency curves at 41cb8686: 1, 2, 4, 8, 16, 32

Fresh complete ABBA sweep at 41cb8686, using the protocol above: 48 measured runs, 3,072/3,072 successful requests. Each point is the mean of four runs per arm; shading shows the observed min/max, not a confidence interval. All six concurrency levels were measured in this sweep.

Run-level measurements (CSV) · Exact summaries and ranges · Measurement protocol

TTFA is the official client's audio TTFP; RTF is its completion-based audio RTF. The horizontal axis uses log2 spacing for the six measured concurrency levels. The result artifacts live on a separate branch, keeping them out of the implementation diff.

C	Arm	Audio throughput (audio-s/s)	Mean TTFA (ms)	Mean RTF	Requests/s
1	Baseline	8.400	309.61	0.1213	2.333
1	PR #9221	8.414	309.06	0.1210	2.337
2	Baseline	13.461	361.84	0.1511	3.739
2	PR #9221	13.155	370.31	0.1542	3.654
4	Baseline	19.704	456.15	0.2072	5.469
4	PR #9221	20.469	452.42	0.2002	5.673
8	Baseline	26.738	685.14	0.3031	7.339
8	PR #9221	28.851	663.09	0.2861	7.944
16	Baseline	30.817	870.49	0.5186	8.446
16	PR #9221	32.862	792.01	0.4801	9.093
32	Baseline	32.672	1334.49	0.9723	9.031
32	PR #9221	35.449	1399.41	0.8952	9.816

• C8/C16/C32 audio throughput improves 7.90% / 6.64% / 8.50%; mean RTF decreases 5.61% / 7.42% / 7.93%.
• C16 mean TTFA decreases 9.02%. C32 mean TTFA increases 4.86%, despite better throughput and completion-based RTF.
• C1 is approximately flat. C2 audio throughput decreases 2.27%, mean TTFA increases 2.34%, and mean RTF increases 2.05%. These measurements do not support a uniform latency or throughput improvement at every concurrency.

This timing sweep does not replace the separate WER run or establish the cause of the earlier C1 regression. The historical study is retained below with its original revisions and limitations.

Historical three-point study and C1 limitation (timing at 03fe9c6)

All 24 measured performance runs completed 64/64 requests. Values are the mean across four runs; brackets show the observed min/max for audio throughput and first-audio latency. TTFA below is the official client's audio TTFP; RTF is its completion-based audio RTF.

Concurrency	Arm	Audio throughput (audio-s/s)	Mean TTFA (ms)	Mean RTF	Requests/s	Mean E2E (ms)
1	Baseline	8.511 [8.477, 8.554]	303.41 [302.87, 304.62]	0.1196	2.364	422.76
1	Candidate	8.075 [7.957, 8.186]	320.27 [317.05, 323.73]	0.1262	2.243	445.64
8	Baseline	27.144 [26.821, 27.497]	697.22 [691.81, 699.60]	0.3009	7.432	1061.46
8	Candidate	27.511 [26.509, 28.701]	650.78 [617.51, 669.63]	0.2976	7.454	1060.44
16	Baseline	31.246 [31.010, 31.411]	910.84 [818.52, 1009.83]	0.5065	8.555	1806.89
16	Candidate	32.786 [31.673, 33.762]	808.03 [779.48, 869.07]	0.4829	8.929	1732.09

C16: audio throughput +4.93%, mean TTFA −11.29%, mean RTF −4.65%. Mean per-run p95 TTFA changes from 1502.44 to 1494.93 ms, so the mean-latency improvement should not be read as a comparable p95 improvement. C8 throughput improvement is small and its run ranges overlap.

C1 limitation: the main sweep shows a 5.13% throughput regression. A separate C1-only warmup/control gives throughput 8.340→8.350 audio-s/s, mean TTFA 314.03→312.01 ms and RTF 0.12227→0.12178 (two measured runs per arm). Scalar setup microbenchmarks are also approximately equal. These controls do not establish the cause or resolve the original mixed-load regression; warmup/concurrency history and experiment order differ.

Memory: sampled device usage every 2 seconds, not allocator peak. At C16, mean per-run observed peak is 48,725→51,338 MiB; the highest candidate sample is 52,902 MiB. This is a memory tradeoff, not a memory reduction. Two initial baseline C1 intervals lack samples; the other 22 measured intervals are covered. Generated audio duration is identical at C1 (230.4 s/run) and varies slightly with concurrent sampling at C8/C16; request throughput is included alongside audio throughput for that reason.

Separate C8 quality run (candidate da8df8a):

Arm	Completed	Saved/judged audio	Mean per-request WER
Baseline	64/64	64/64	0.053008
Candidate	64/64	64/64	0.044707

WER did not worsen in this sample. The baseline is above 0.05; the candidate is below it. This is one 64-request comparison, not a general quality guarantee. SIM and UTMOS were not measured.

Correctness regression coverage at 532e9ca1: 279 passed, 14 warnings in 64.90s. It covers mixed shapes, graph batch limits, cache hits/LRU eviction, duplicate references and continuing streams, independent cache ownership, failed setup publication, disabled/unsupported execution policies, graph replay refusal without re-encoding, and decode-first precapture under tight budgets.

python -m pytest -q \
  tests/model_executor/models/minicpmo_4_5/test_code2wav_batching.py \
  tests/model_executor/models/minicpmo_4_5/test_code2wav_batch_buckets.py \
  tests/model_executor/models/minicpmo_4_5/test_cfm_graph_bucketing.py \
  tests/model_executor/models/minicpmo_4_5/test_cfm_graph_capture_gating.py

Real-H800 component checks use native setup APIs and fixed reference features, including unequal prompt lengths, cache ownership, and six continuation chunks extending beyond the 100-frame history suffix. Normal fused execution and full-graph-budget fallback pass numerical comparison at atol=1e-5, rtol=1.3e-6; earlier TF32 opt-out and explicit unfused checks retain scalar setup. These component checks are separate from WER. Changed-file pre-commit passes except for two inherited mypy missing-annotation errors in the untouched key-mask tests; both errors were reproduced on baseline bea9b9e9. NPU E2E, perceptual quality, and current-head WER remeasurement have not been run.

Codex assisted implementation, tests, review and PR drafting.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@natureofnature, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[amy-why-3459]

Thanks for splitting this out from #9159 and for the clear tradeoff write-up. I went through setup_references / _create_initial_states in batched_token2wav.py, the _prepare_decode_batch change, and the Whole-Euler replay/_entry paths they now hit. Cache ordering, hit retention and the per-row isolation look right to me: the initial solve has no att_cache/prompt_lengths, so replay returns a stacked out_att and _split_flow_cache copies each row into its own exact-size buffer. That keeps the cached prompt state un-growable in place, which setup_batch relies on.

Two things on the graph side that might also bear on the C2 / C32 TTFA numbers:

1. The batched cold solve runs at offset 0 with graph_batch from _plan_groups(B) (2/4/8) and a query width taken from the reference's mel length. precapture_whole_euler only covers decode offsets range(prompt_len, steady + 1), so these setup graphs are captured lazily on the serving path. They also draw from the same max_graphs budget (default 32) as the decode graphs. Each distinct (group size, aligned prompt width) is a new entry, so with varied reference lengths this can take budget from decode shapes, or find it already full. Could you check the capture/hit stats from the sweep? It might be worth either precapturing the setup widths for the precaptured prompt or keeping them out of the decode budget.

2. When the batched replay is refused (budget full is memoized in _unsupported, so it stays refused for that key), the fallback discards the B per-reference _encode_initial_prompt outputs and re-runs the encoder and speaker projection per reference (inline below). Once a key is unsupported, every later cold group of that shape pays the encoder twice. Checking eligibility first, or passing the already-encoded rows into the scalar path, would avoid that.

Minor: the transient peak during a cold group is now the stacked out_att for 2B CFG rows plus the per-row copies made in _split_flow_cache (and the extra per-row clone() of the other three caches on top of the batch clone that _split_flow_cache already does). That fits the +2.6 GiB observed at C16. It may be worth a line in the design doc next to the microbatch bound, since it scales with micro_batch_size.

[amy-why-3459]

On refusal, the per-reference encoder/speaker outputs computed above for the whole group are dropped and recomputed by each _create_initial_states(feature, 1). Since an over-budget key is added to _unsupported and stays refused, this double encode repeats for every later cold group of the same shape. Could the scalar fallback reuse encoded[i] (or could eligibility be checked before encoding)?

[natureofnature]

Fixed in 532e9ca. The scalar retry now receives each reference's existing encoded_prompt, so it reuses the M=1 encoder and speaker-projection outputs rather than recomputing them. This also covers later groups hitting a memoized refusal.

The encoding-count regressions fail on the parent and pass with this change, covering replay refusal, wrapper disablement, and a full graph budget. Real H800 full-budget checks at B=3 and B=8 also pass cache/PCM comparison against independent scalar streams at atol=1e-5, rtol=1.3e-6.

[amy-why-3459]

This is the first place offset-0 Whole-Euler graphs with graph_batch > 1 get captured, lazily during serving, since precapture_whole_euler only warms decode offsets. Each (group size, prompt query width) uses a slot of the shared max_graphs budget. Do the sweep logs show how many setup captures happened, and whether decode shapes were pushed out to eager as a result?

[natureofnature]

Checked the original sweep logs at 41cb8686: both baseline visits reached 25/32 graphs and both candidate visits reached 28/32. In candidate B1, an M=2 offset-zero setup graph was captured during the second measured C2 run; host warmup plus capture took 1.051 seconds. No graph-budget saturation was observed. Those log snapshots are not whole-run fallback totals, so they do not establish a zero-fallback claim for the entire sweep.

Addressed in 532e9ca: eligible non-slot precapture warms decode/tail shapes first, followed by setup shapes in ascending batch order, within the unchanged shared budget. The real-weight H800 check warms 24 decode graphs plus four setup graphs (M=1/2/4/8). Tight-budget tests retain decode priority, and refused batches reuse their existing encodings for scalar retry. Unseen reference widths still use the existing lazy-capture/fallback path.

On this revision, the official serving regression completed 384/384 requests across C2/C32, plus 16 warmup requests. All 28 Whole-Euler captures occurred during warmup, with none logged in the six subsequent runs. This addresses the observed capture mechanism; it does not establish that every earlier latency regression is resolved.

Also added the transient-memory explanation to the design document: the stacked CFG output and independent per-reference cache copies coexist during setup, with the group size bounded by the existing microbatch limit.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 532e9ca1145c produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot: three questions on the performance claim

@natureofnature this PR reads as a performance or value claim:

• claim: - C8/C16/C32 audio throughput improves 7.90% / 6.64% / 8.50%; mean RTF decreases 5.61% / 7.42% / 7.93%.
• claim: A separate C1-only warmup/control gives throughput 8.340→8.350 audio-s/s, mean TTFA 314.03→312.01 ms and RTF 0.12227→0.12178 (two measured runs per arm).

Before the full evidence checklist, three short questions:

1. Bottleneck — what is the current bottleneck, and which profile, trace or per-stage measurement shows it?
2. Value — what does the change buy the user or the system (latency, throughput, memory, cost), and at which workload?
3. A/B or ablation — is there a same-workload, same-head/config comparison that isolates each main claim on its own? For stacked optimizations, one number per item rather than a blended delta.

When you answer, the evidence that settles it is: base and head SHA, hardware, model, workload, warm-up and repeat count, mean or percentiles with their spread, and a correctness/quality-equivalence signal; an end-to-end claim also needs stage attribution.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-21T15:38:27.462295Z We couldn’t complete this review. Comment @codex review to try again.	41cb868	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[natureofnature]

Following up on the performance-evidence questions:

Scope and attribution. This change targets the separate initial CFM solve performed for each cold reference. It groups compatible misses while keeping reference extraction, encoder initialization, and speaker projection at M=1. It does not add FP16 execution or change steady-state decode batching. The call-count/cache tests verify that mechanism, and the serving comparison measures its end-to-end effect. A complete per-stage timing profile was not collected, so these results do not establish that cold setup is the dominant service bottleneck.

Controlled comparison. Base bea9b9e9ecb980d3cbdbf0ee061a42f325b83e75 versus candidate 41cb868615e8480e5d75f6b9dfc9732ccac3eddc, on one H800 80 GB with MiniCPM-o 4.5, vLLM 0.32.0, MPS enabled in both arms, stage limits 8/8/8, and CFM microbatch 8. Both use the baseline's official vllm bench serve client, the same ordered 64-request English Seed-TTS subset, and the same configuration. Server order was A1/B1/B2/A2; each visit had 16 warmup requests, then one discarded and two measured runs at each of C1/2/4/8/16/32. This gives four measurements per arm/point and 3,072/3,072 successful measured requests. Run-level data and means/ranges are published alongside the chart.

Value and limits. At C8/C16/C32, audio throughput improves 7.90%/6.64%/8.50%, and mean completion RTF decreases 5.61%/7.42%/7.93%. C1 is approximately flat; C2 throughput falls 2.27%, and C32 mean TTFA rises 4.86%. The earlier C16 memory study also shows an approximately 2.6 GiB increase in sampled peak device usage. This is a concurrent-throughput tradeoff, not a uniform latency or memory improvement.

The current head, 532e9ca1, fixes the two graph-related review findings. Its 279-test regression suite, real-weight H800 cache/PCM comparisons, and C2/C32 serving checks pass. The complete A/B sweep has not been rerun on this head. The separate 64-request C8 WER comparison remains tied to da8df8a9 (baseline 0.053008, candidate 0.044707); it is not current-head WER qualification. The PR description keeps these evidence sets and their limitations separate.

[natureofnature]

@amy-why-3459 PTAL

[natureofnature]

@natureofnature

[amy-why-3459]

Thanks for the follow-up. Checked the delta in 532e9ca:

• Refused batched replay now hands each reference's existing encoded_prompt to the scalar retry, so the M=1 encoder/speaker outputs are reused instead of recomputed, including for later groups that hit the memoized _unsupported key. The encoding-count tests cover the refusal, disabled-wrapper and full-budget paths.
• Offset-0 setup graphs are now precaptured after the decode/tail keys, M=1 first, within the same max_graphs budget, and the setup width goes through the same _capture_query_width as replay, so the warmed keys match what serving hits. The doc note on the transient per-reference cache copies covers the memory point.

Both earlier points look addressed; no further comments on this delta. As you note, reference widths not seen during precapture still go through the lazy capture/fallback path, and the full A/B sweep plus WER are from earlier heads.

[natureofnature]

Reviewed the six-file change through fresh-reference grouping, CFG/cache row splitting, scalar fallback and decode-first precapture. The existing encodings now survive repeated graph refusals, and cached references retain independent storage and caller order. Both earlier graph findings are addressed.

The reported current-head 279-test suite, H800 mixed-length/continuation and full-budget cache/PCM comparisons, and C2/C32 serving regression cover the changed mechanisms. The earlier A/B sweep and WER remain evidence for their stated revisions; I have not rerun these tests. Repository CI still has pending checks and an AMD failure whose cause I could not inspect.

【Conclusion: Found no blocking issues in commit 532e9ca.】
【Action recommended: None.】