[ZhengWG]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Improve latency and throughput for shared-prefix requests in Qwen3-Omni and MiniCPM-o turn serving.

2. What is the highest-leverage variable for that outcome?

The reuse of cached KV states and Thinker outputs across requests.

3. How will this work move that variable?

Enable Thinker prefix caching by default for supported V1 turn profiles, while keeping MRv2 and duplex profiles disabled.

4. Whose customer or user needs does this address, and what are those needs?

Users serving repeated or shared-prefix requests need better performance without manual cache configuration.

Purpose

Part [3/N] of #7902: [1/N] #9162 fixes the collapse under concurrent hits, [2/N] #9193 removes the per-step cost at 0% hit, and this PR turns the Thinker prefix cache on by default for Qwen3-Omni and MiniCPM-o 4.5 (CUDA turn serving). Merge after #9162 and #9193: on main alone the cache collapses on long shared prefixes at C64 (Qwen3-Omni −68% to −77%) and costs MiniCPM-o 19% on decode (column "main, cache on" below).

With the stack, the cache costs at most 3.0% when nothing hits and speeds up any traffic that shares a prompt prefix (system prompts, multi-turn history, shared media).

• qwen3_omni_moe.yaml: stage 0 on; qwen3_omni_moe_mrv2.yaml and qwen3_omni_duplex.yaml set it back to false (not validated with the cache).
• minicpmo_4_5_turn.yaml: stage 0 on for CUDA; the NPU platform block, the base and duplex profiles stay off; minicpmo_4_5_turn_mrv2.yaml sets it back to false (MRv2 does not merge cached Thinker rows yet).

Follow-up: #9039 replaces the mirror/COW core with refcounted payload references. It can land after this series and take over the code from #9162 / #9193 under the same defaults, with this test matrix as its acceptance check.

Prefix-cache scope by model

model / profile (stage)	default	evidence
Qwen3-Omni qwen3_omni_moe.yaml (Thinker)	on (this PR)	Test Result below. Daily-Omni 0.7157 vs 0.7157; batch-invariant outputs 46/46 equal
MiniCPM-o 4.5 minicpmo_4_5_turn.yaml (LLM, CUDA)	on (this PR)	Test Result below; Seed-TTS WER 0.0207 vs 0.0215
Qwen3-Omni MRv2 / duplex; MiniCPM-o MRv2 / duplex / NPU	off	not validated; MiniCPM-o MRv2 fails 79/80 requests on stage-0 hits until it merges cached Thinker rows
Higgs Audio v3 (stage 0)	on (unchanged)	−1.2% to −1.8% vs off at 0–95% hit (main −2.8% to −4.5%); WER/SIM unchanged; hits only on exact repeats
Audex-2B thinker_only / s2s	on (unchanged)	thinker_only +0.4% / −0.1% at 0% hit, ~9× on 98% hit replays, WER equal
Audex-2B tts / tta	on (unchanged)	never hits (CFG requests skip cache reads); −0.3% / −2.3%; owners may turn it off
JoyAI-VL-Interaction, AURA	on (unchanged)	no visible cost; AURA 60 sessions, 0 errors, 53–91% hit, batch-invariant replies 10/10 equal
Qwen2.5-Omni (Thinker)	off	−2.0% / −2.5% at 0% hit, no gain on short prompts, +12% to +66% on long prompts; opt in for long-prompt traffic
Step-Audio 2 mini (ASR)	off	±2.7%, replay +1.4%; 12.9 GB host mirror with no downstream consumer
Qwen3-TTS V1 / MRv2	off	V1 hits only exact repeats and lacks an offline salt; MRv2 TTFP +18% to +131%
GLM-Image, MOSS-TTS local	on (unchanged)	not validated
talker / TTS stages; Fish, Voxtral, VoxCPM2, CosyVoice3, IndexTTS-2	off	static review: hit offsets, missing salt or wrong embedding table

Test Plan

vLLM Version: 0.32.0

vLLM-Omni Commit: #9162 + #9193 + these commits

Test bases: GPU runs on main da572cddb38383deae9caef33b8d490e292ef071.

Hardware: H20 (96 GB).

1. Unit: pytest tests/core tests/worker/test_gpu_ar_model_runner.py tests/worker/test_gpu_ar_model_runner_cuda.py tests/config/test_config_factory.py on the stack; pytest tests/config/test_config_factory.py for this PR alone, also on the latest main 15bce345863c6e2db882534876b8977f5b1ae1e3.
2. Throughput: Qwen3-Omni 4 rounds, MiniCPM-o 3 rounds. Each round runs cache off and the stack on the same GPUs, one fresh server each (order alternating across rounds), with the warmup and benches in the order below. Main cache on comes from a separate run with its own cache-off arm (3 rounds; DECV_* 4 / 3). Metric: request_throughput from vllm bench serve --save-result; ratio = mean over rounds / mean over rounds of the same run's cache off − 1; range = min..max of the per-round ratios. Hit rate = delta of stage 0's vllm:prefix_cache_{hits,queries}_total on /metrics around each bench.
3. Memory: stage-0 VmRSS (/proc/<pid>/status) after each throughput run.
4. Accuracy: MiniCPM-o Seed-TTS EN, all 1088 utterances, C10, cache off vs the stack.

reproduction commands

Servers (overlay.yaml sets stage 0 enable_prefix_caching: false for cache off; the stack runs the shipped yaml):

# Qwen3-Omni: the stack serves vllm_omni/deploy/qwen3_omni_moe.yaml; cache off serves overlay.yaml
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --deploy-config vllm_omni/deploy/qwen3_omni_moe.yaml \
  --stage-init-timeout 600 --init-timeout 900 --log-stats
# MiniCPM-o: both arms serve overlay.yaml
vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code --deploy-config overlay.yaml \
  --stage-init-timeout 600 --init-timeout 900 --log-stats

# Qwen3-Omni cache off
base_config: <checkout>/vllm_omni/deploy/qwen3_omni_moe.yaml
stages:
  - stage_id: 0
    enable_prefix_caching: false
---
# MiniCPM-o (both arms; add enable_prefix_caching: false for cache off)
base_config: <checkout>/vllm_omni/deploy/minicpmo_4_5_turn.yaml
stages:
  - stage_id: 0
    max_model_len: 17408

Benches, in this order on each server:

COMMON="--backend openai-chat-omni --endpoint /v1/chat/completions --request-rate inf --num-warmups 0 \
  --ready-check-timeout-sec 0 --trust-remote-code --save-result"
# rand <model> <extra-body> <percentile-metrics> < table; columns: name shared_prefix random_len output range_ratio concurrency prompts seed
rand() {
  while read -r name prefix input output ratio c n seed; do
    vllm bench serve --omni --model "$1" $COMMON --dataset-name random --random-prefix-len $prefix \
      --random-input-len $input --random-output-len $output --random-range-ratio $ratio --ignore-eos \
      --extra-body "$2" --percentile-metrics "$3" --max-concurrency $c --num-prompts $n --seed $seed \
      --result-filename $name.json
  done
}
# mp32 <model> <extra-body> <seed>: 32 prefixes of 3840 tokens, 256 own tokens each, C64, 512 prompts
mp32() {
  vllm bench serve --omni --model "$1" $COMMON --dataset-name prefix_repetition --prefix-repetition-prefix-len 3840 \
    --prefix-repetition-suffix-len 256 --prefix-repetition-num-prefixes 32 --prefix-repetition-output-len 8 \
    --ignore-eos --extra-body "$2" --percentile-metrics ttft,tpot,itl,e2el --max-concurrency 64 --num-prompts 512 \
    --seed $3 --result-filename MP32_c64.json
}
TXT=ttft,tpot,itl,e2el AUD=ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,audio_duration

Q=Qwen/Qwen3-Omni-30B-A3B-Instruct
rand $Q '{"modalities": ["text"]}' $TXT <<'B'
warm               512   512   16  0.0 4  16  50001
P1024_h0_c1        0     1024  8   0.0 1  256 50002
P1024_h0_c8        0     1024  8   0.0 8  256 50003
P1024_h0_c64       0     1024  8   0.0 64 256 50005
P4096_h0_c1        0     4096  8   0.0 1  64  50014
P4096_h0_c64       0     4096  8   0.0 64 256 50017
P16384_h0_c64      0     16384 8   0.0 64 128 50029
P1024_h50_c32      512   512   8   0.0 32 256 50008
P1024_h50_c64      512   512   8   0.0 64 256 50009
P4096_h50_c32      2048  2048  8   0.0 32 128 50020
P4096_h50_c64      2048  2048  8   0.0 64 256 50021
P1024_h94_c64      960   64    8   0.0 64 256 50013
P4096_h94_c32      3856  240   8   0.0 32 128 50024
P4096_h94_c64      3856  240   8   0.0 64 256 50025
P16384_h94_c32     15408 976   8   0.0 32 64  50036
P16384_h94_c64     15408 976   8   0.0 64 128 50037
O256_L4096_h94_c64 3856  240   256 0.0 64 256 50043
B
rand $Q '{"modalities": ["text", "audio"]}' $AUD <<'B'
AUD_L4096_h94_c32  3856  240   128 0.0 32 64  50054
B
mp32 $Q '{"modalities": ["text"]}' 50057
rand $Q '{"modalities": ["text"]}' $TXT <<'B'
DEC100_c32         0     100   100 0.0 32 256 50059
DEC100_c64         0     100   100 0.0 64 512 50060
DECV_c32           0     100   100 0.5 32 256 50063
DECV_c64           0     100   100 0.5 64 512 50064
B

M=openbmb/MiniCPM-o-4_5
MT='{"modalities": ["text"], "chat_template_kwargs": {"enable_thinking": false}}'
MA='{"modalities": ["text", "audio"], "chat_template_kwargs": {"enable_thinking": false, "use_tts_template": true}}'
rand $M "$MT" $TXT <<'B'
warm               512   512   16  0.0 4  16  50061
P256_h0_c32        0     256   8   0.0 32 512 50063
P1024_h0_c32       0     1024  8   0.0 32 256 50070
P1024_h0_c64       0     1024  8   0.0 64 256 50071
P4096_h0_c32       0     4096  8   0.0 32 128 50088
P4096_h0_c64       0     4096  8   0.0 64 256 50089
P8192_h0_c32       0     8192  8   0.0 32 128 50099
P16384_h0_c32      0     16384 8   0.0 32 64  50106
P16384_h0_c64      0     16384 8   0.0 64 128 50107
DEC100_c32         0     100   100 0.0 32 256 50129
DEC100_c64         0     100   100 0.0 64 512 50130
P1024_h50_c64      512   512   8   0.0 64 256 50075
P4096_h50_c64      2048  2048  8   0.0 64 256 50093
P16384_h50_c64     8192  8192  8   0.0 64 128 50111
P4096_h94_c32      3856  240   8   0.0 32 128 50096
P4096_h94_c64      3856  240   8   0.0 64 256 50097
P16384_h94_c64     15408 976   8   0.0 64 128 50115
B
mp32 $M "$MT" 50136
# Seed-TTS EN lines 113-240 (128 utterances) of seed-tts-eval en/meta.lst
mkdir -p s128/en && sed -n '113,240p' seed-tts-eval/en/meta.lst > s128/en/meta.lst \
  && ln -sfn "$PWD/seed-tts-eval/en/prompt-wavs" s128/en/prompt-wavs
vllm bench serve --omni --model $M $COMMON --dataset-name seed-tts --dataset-path s128 --no-oversample \
  --disable-shuffle --seed-tts-locale en --extra-body "$MA" --percentile-metrics ttft,e2el,audio_ttfp,audio_rtf,audio_duration \
  --max-concurrency 8 --num-prompts 128 --result-filename SEEDTTS_c8.json
rand $M "$MA" $AUD <<'B'
AUD_L2048_h94_c32  1920  128   128 0.0 32 128 52043
B
rand $M "$MT" $TXT <<'B'
DECV_c32           0     100   100 0.5 32 256 50063
DECV_c64           0     100   100 0.5 64 512 50064
B

# accuracy, MiniCPM-o (all 1088 Seed-TTS EN utterances)
python tests/e2e/accuracy/qwen3_omni/run_qwen_omni_acc_benchmark.py --host 127.0.0.1 --port $PORT --model $M \
  --result-dir $OUT --num-prompts 2000 --max-concurrency 10 --skip-daily-omni --seed-tts-dataset-path seed-tts-eval \
  --seed-tts-wer-save-items --max-seed-tts-mean-wer 0.05 --seed-extra-body-json "$MA" --trust-remote-code \
  --seed-tts-eval-device cuda:0

Test Result

1. Unit: the stack, full command, 1153 passed, 2 skipped (CPU); this PR alone on 15bce3458, tests/config/test_config_factory.py 500 passed (main 492).
2. Throughput vs cache off ("stack" = [1/N][Perf][Core] Stop the omni prefix cache from collapsing under concurrent hits #9162 + [2/N][Perf][Core] Cut the per-step cost of the omni prefix cache #9193 + this PR):

Qwen3-Omni (request_throughput vs cache off; TTFP = median audio time to first packet)

bench	stage-0 hit	cache off (req/s)	main, cache on	stack	per-round range
P1024_h0_c1	0.0%	7.916	−2.3%	−1.4%	−2.6% .. +0.0%
P1024_h0_c8	0.0%	15.70	−6.4%	−0.9%	−6.2% .. +3.2%
P1024_h0_c64	0.8%	18.26	−1.5%	+0.2%	−0.1% .. +0.8%
P4096_h0_c1	0.0%	3.243	−5.2%	−0.3%	−1.6% .. +0.7%
P4096_h0_c64	0.8%	4.343	+0.6%	+0.6%	+0.5% .. +0.7%
P16384_h0_c64	0.2%	0.679	+0.3%	+0.1%	+0.1% .. +0.2%
P1024_h50_c32	49.4%	18.76	+64.6%	+70.1%	+69.0% .. +71.7%
P1024_h50_c64	49.4%	19.44	+68.8%	+76.5%	+75.7% .. +77.2%
P4096_h50_c32	49.5%	4.307	+69.9%	+74.4%	+73.9% .. +74.8%
P4096_h50_c64	49.7%	4.345	+75.6%	+76.9%	+76.7% .. +77.2%
P1024_h94_c64	92.6%	19.51	+113.2%	+355.8%	+302.7% .. +392.7%
P4096_h94_c32	93.2%	4.311	+105.9%	+658.4%	+611.2% .. +703.6%
P4096_h94_c64	93.6%	4.349	−68.1%	+834.4%	+827.9% .. +838.6%
P16384_h94_c32	92.5%	0.679	+108.6%	+775.7%	+773.8% .. +777.5%
P16384_h94_c64	93.3%	0.679	−26.9%	+858.6%	+854.9% .. +862.0%
O256_L4096_h94_c64	93.6%	2.771	−51.1%	+133.7%	+131.5% .. +135.1%
AUD_L4096_h94_c32	92.5%	1.255	+16.2%; TTFP −66.5%	+19.5%; TTFP −74.9%	+18.9% .. +20.8%
MP32_c64	87.7%	4.352	−77.3%	+511.6%	+509.2% .. +513.8%
DEC100_c32	0.3%–64.7%	19.25	+4.0%	+5.8%	+4.5% .. +8.0%
DEC100_c64	0.0%	26.77	−6.5%	−0.8%	−2.0% .. −0.3%
DECV_c32	0.0%	16.66	−11.6%	−3.0%	−4.1% .. −1.8%
DECV_c64	0.5%	22.41	−21.6%	−2.1%	−3.2% .. −0.8%

MiniCPM-o (request_throughput vs cache off; TTFP = median audio time to first packet)

bench	stage-0 hit	cache off (req/s)	main, cache on	stack	per-round range
P256_h0_c32	0.4%	32.12	−6.6%	−0.7%	−1.0% .. −0.6%
P1024_h0_c32	0.0%	8.930	−3.5%	−1.4%	−1.5% .. −1.3%
P1024_h0_c64	0.0%	8.928	−3.3%	−0.6%	−1.2% .. −0.2%
P4096_h0_c32	0.0%	2.171	−0.7%	−0.4%	−0.5% .. −0.4%
P4096_h0_c64	0.0%	2.175	−0.6%	−0.5%	−0.5% .. −0.5%
P8192_h0_c32	0.0%	1.009	−0.1%	−0.1%	−0.2% .. −0.1%
P16384_h0_c32	0.0%	0.441	−0.3%	−0.3%	−0.3% .. −0.2%
P16384_h0_c64	0.0%	0.442	−0.2%	−0.3%	−0.4% .. −0.2%
DEC100_c32	0.0%	23.39	−18.8%	−2.5%	−2.7% .. −2.3%
DEC100_c64	0.2%	23.87	−19.3%	−1.8%	−3.1% .. −0.8%
P1024_h50_c64	49.2%	8.938	+78.4%	+85.6%	+85.3% .. +85.8%
P4096_h50_c64	49.7%	2.175	+84.1%	+86.5%	+86.5% .. +86.6%
P16384_h50_c64	49.6%	0.442	+75.5%	+75.5%	+75.3% .. +75.6%
P4096_h94_c32	93.1%	2.172	+28.3%	+1027.3%	+1026.4% .. +1029.0%
P4096_h94_c64	93.5%	2.174	−35.1%	+1076.5%	+1074.2% .. +1078.6%
P16384_h94_c64	93.2%	0.441	+89.7%	+1028.4%	+1026.4% .. +1029.5%
MP32_c64	85.9%	2.176	+511.5%	+513.1%	+511.5% .. +514.7%
SEEDTTS_c8	78.5%	2.956	+2.4%; TTFP −3.4%	+3.7%; TTFP −5.7%	−0.2% .. +7.3%
AUD_L2048_h94_c32	92.4%	0.970	+7.6%; TTFP −65.6%	+7.3%; TTFP −64.6%	+6.0% .. +9.5%
DECV_c32	0.0%	24.11	−19.3%	−2.5%	−2.8% .. −2.2%
DECV_c64	0.4%	23.82	−19.1%	−2.5%	−4.3% .. −0.0%

3. Stage-0 VmRSS: Qwen3-Omni 15.2–16.1 GiB (main cache on 21.7–32.5, off 5.2–5.3); MiniCPM-o 7.3–7.6 GiB (main 9.0–9.1, off 4.4–4.7).
4. Accuracy, MiniCPM-o Seed-TTS EN (82.4% stage-0 hit): WER 0.0207 vs 0.0215 cache off, SIM 0.8645 vs 0.8644, 1088/1088 completed in both.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.

Note

To avoid agent spam, vLLM-Omni limits contributors without write access to 5 open PRs. If this blocks well-intentioned critical work, please contact a committer to request bypass list access.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-22T11:52:49.682193Z We couldn’t complete this review. Comment @codex review to try again.	e091c85	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/vllm_omni_config.md, docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @lishunyang12 @NickCao

Routing: @alex-jw-brooks via module of the changed files, CODEOWNERS; @lishunyang12 via module of the changed files, CODEOWNERS; @NickCao via module of the changed files, CODEOWNERS

@ZhengWG, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 7e9e23bcfdf1 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[ZhengWG]

@amy-why-3459 PTLA

[amy-why-3459]

@amy-why-3459

[amy-why-3459]

Thanks for the thorough matrix. Two things on the scope of the new default:

1. The PR describes the new default as CUDA turn serving, but in qwen3_omni_moe.yaml the flag is flipped in the platform-shared stage-0 block. None of the platforms.npu / rocm / xpu / musa blocks override it, so Qwen3-Omni stage 0 also gets the Thinker prefix cache on Ascend (TP=2, PIECEWISE graphs), ROCm, XPU and MUSA, none of which are in the validation runs. MiniCPM-o handles this with an explicit NPU-off override; Qwen3-Omni probably wants the same (either set it in platforms.cuda only, or turn it off in the non-CUDA blocks), plus matching rows in test_thinker_prefix_cache_defaults.

2. The PR says it must land after #9162 and #9193, and that on main alone the cache collapses at C64 (Qwen3-Omni −68% to −77%) and costs MiniCPM-o ~19% on decode. Since this is a default flip, it would be safer to keep it blocked or in draft until both are merged, so it can't go in first by accident.

[amy-why-3459]

Looked at the new commit e091c85a. Turning stage 0 prefix caching off in the npu / rocm / xpu / musa blocks of qwen3_omni_moe.yaml, and adding the rocm / xpu / musa blocks to minicpmo_4_5_turn.yaml, resolves correctly: _merge_platforms deep-merges per platform and per stage_id, so the new stage-0 entries sit next to the existing stage-1/2 overrides and the base's NPU block is kept. Those are all the platforms under vllm_omni/platforms, and the new test_thinker_prefix_cache_defaults rows cover each one. That addresses my first point.

On the second point: the branch still only has the three config commits, and #9162 and #9193 are both still open. So if this merged first, main would get the default flip without the fixes the description depends on. Keeping it in draft or blocked until both land would still be the safer option.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[ZhengWG]

@amy-why-3459 PTLA

[amy-why-3459]

Why does minicpmo_4_5.yaml still leave Thinker prefix caching off?

This file is the model's default deploy config (default_deploy_config_name="minicpmo_4_5.yaml") and the base the other MiniCPM-o profiles inherit. Root enable_prefix_caching stays false (line 13), and this shared stage-0 block still does not set the stage key, so on CUDA stage 0 resolves to off (test_thinker_prefix_cache_defaults). The new default is only on minicpmo_4_5_turn.yaml, and only for CUDA.

That is the inverse of Qwen3-Omni, where the primary turn yaml turns stage 0 on and the duplex / MRv2 profiles opt back out. With this layout, vllm serve of MiniCPM-o 4.5 on the default config does not get the cache this PR says it enables by default; the caller has to pass the turn deploy config.

The new comment attributes that to the duplex KV window plus the turn overlay. validate_duplex_window_install does refuse enable_prefix_caching=True, but that only covers the duplex session — it does not say why the shared stage-0 block stays unset on CUDA instead of stating false the way the NPU block does. The previous text in this hunk gave a different reason: setting stage-level prefix caching in the platform-shared block killed the stage-1 replica on CUDA (empty TTS handoff, then an indexSelectSmallIndex device assert). This revision drops that rationale and still does not enable the flag.

Please spell out which constraint still keeps this file off on CUDA. If it is only the duplex window, stage 0 here should set enable_prefix_caching: false explicitly, same as platforms.npu, rather than inheriting the root default through a missing key. If that CUDA stage-1 failure no longer applies, why shouldn't stage 0 of this default profile follow minicpmo_4_5_turn.yaml?

[amy-why-3459]

Prerequisites are met: #9162 (798fef99a3) and #9193 (949907817e) are both on main, so the earlier "must land after both" blocker is cleared. On e091c85a I ran tests/config/test_config_factory.py (507 passed) and walked _apply_platform_overrides → merge_pipeline_deploy by hand to check the final engine args:

• qwen3_omni_moe.yaml: cuda [True, False, False], npu/rocm/xpu/musa [False, False, False]
• minicpmo_4_5_turn.yaml: cuda True, npu/rocm/xpu/musa False (the NPU off comes from the base's platforms.npu stage-0 entry; _merge_platforms deep-merges per platform and per stage_id, so an overlay without an npu block does not drop it)
• minicpmo_4_5.yaml / _duplex_h200 / both mrv2 profiles: all False

The platform overrides, the MRv2/duplex fallbacks and the parametrization look right. The four inline comments below are the specific ones; three broader points first:

1. The branch is 52 commits behind main. The merge base is baa2f5adcb, which does not contain 798fef99a3 / 949907817e. The throughput matrix in the description was measured on the stack, so please rebase onto a main that has both PRs and re-run at least the config unit tests plus one high-hit scenario, so the evidence behind "on by default" matches the target revision.
2. Host memory is a real cost of the new default. Stage-0 VmRSS goes 5.2 → 15.2–16.1 GiB for Qwen3-Omni and 4.4 → 7.3–7.6 GiB for MiniCPM-o. The pinned pool is allocated as num_blocks × block_size × feat with no YAML/CLI cap (vllm_omni/core/prefix_cache/block_pool.py, plus the staging note in interface.py), and with N replicas it multiplies by N. Worth recording the measured numbers in the yaml comment, along with an overlay snippet for users who want it off.
3. The new minicpmo_4_5.yaml comment says the shared block "leaves stage-level prefix caching unset", but the pipeline-wide enable_prefix_caching: false on line 13 of the same file is still what applies (the platforms.npu stage-0 false is extra belt-and-braces). As written it reads as if nothing is configured there. Suggest spelling out that the pipeline-wide value covers duplex and the turn profile overrides it at stage level.

Overall: the direction and the evidence are stronger than most config PRs in this repo. Fix the e2e assertion and move the test assertion down to the final engine args, and I think this can land.

[amy-why-3459]

This line breaks test_structured_multistage_config_reaches_runtime in tests/e2e/offline_inference/test_qwen3_omni.py, which runs in the merge gate at .buildkite/cuda/test-merge.yml:149:

# tests/e2e/offline_inference/test_qwen3_omni.py:109
assert all(config.cache_config.enable_prefix_caching is False for config in engine.stage_vllm_configs)

That case runs production_test_params, i.e. this file, so with stage 0 now True the assertion must fail. The Test Plan only lists tests/core, tests/worker and tests/config, so the e2e run was not covered and this did not surface. Either flip the assertion to stage 0 True and stages 1/2 False, or move the case onto an overlay that explicitly disables the cache. If the intent is to keep checking that deploy values really reach the VllmConfig objects, the former is better.

Separately, tests/e2e/online_serving/test_qwen3_omni.py:31-36 still says "Prefix cache is opt-in on the deploy YAML", which is now stale; please update it in the same pass.

[amy-why-3459]

This assertion stops at the deploy layer and does not cover the mechanism it actually depends on. enable_prefix_caching is not a StageDeployConfig field, so it lands in engine_extras and is applied last by engine_args.update(ds.engine_extras) in _build_engine_args (around vllm_omni/config/stage_config.py:1324), overriding the pipeline-wide value. That precedence is exactly why this PR works, but the test never verifies it: if the merge order of engine_extras changes, this assertion stays green while production silently falls back to false.

Suggest adding at least one assertion per enabled profile that goes through merge_pipeline_deploy(...) and checks yaml_engine_args["enable_prefix_caching"], for qwen3_omni_moe.yaml/cuda and minicpmo_4_5_turn.yaml/cuda. The existing test_step_audio2_* cases nearby already use that pattern.

Also worth adding the two derived profiles that stay off only through inheritance: minicpmo_4_5_turn_mrv2_h200.yaml and qwen3_omni_duplex_mrv2.yaml. They do not change when the base does, but that is precisely the kind of inheritance that gets silently flipped the next time someone edits a base file.

[amy-why-3459]

This removes a CUDA regression note that had a clear provenance (added in b60a7e9e86): enabling stage-0 prefix caching in the platform-shared block once killed the stage-1 replica on CUDA — empty tts handoff, then an indexSelectSmallIndex device assert. This PR effectively lifts that constraint (the CUDA turn profile turns the same switch on) without saying why it no longer applies.

The Seed-TTS evidence (1088/1088 completed, WER 0.0207 vs 0.0215, 82.4% stage-0 hit) is strong, but please tie it to the conclusion in the description or this comment: was that failure specific to the duplex path (duplex now disables the cache explicitly, and window_kv.py:153 raises anyway), or was it fixed by a later change? Otherwise the next reader will assume the original note was overlooked.

[amy-why-3459]

Turning this on by default also has a consequence that is not in the PR description: the video streaming API silently switches to incremental prefill.

# vllm_omni/entrypoints/openai/video_stream_base.py:296-298
return self._incremental_prefill_supported and "audio" not in config.modalities

where _incremental_prefill_supported = _stage0_prefix_caching_enabled(engine_client) (line 321 of the same file) reads exactly this stage-0 flag. So under this profile, text-only /v1/video/chat/stream sessions move from the legacy path to incremental prefill: every buffered frame goes into the prompt instead of stride-sampling num_frames, a full buffer drops the oldest half at once, and each query waits for the whole window to finish decoding.

That is a default-value change and deserves a line in the PR description. docs/serving/video_stream_api.md:19-21 currently still says "to enable incremental prefill, set enable_prefix_caching: true"; it should say it is on by default and that disabling the cache falls back to legacy.

[twu3202]

Ran main 798fef9 (with #9162 and #9193) + e091c85 on one RTX 6000 Ada (48 GB) with MiniCPM-o 4.5: minicpmo_4_5_turn.yaml as shipped vs stage 0 enable_prefix_caching: false, with the deploy trimmed to fit the card (8 seqs, MPS off). I started two fresh servers per setting and sent each 75 text+audio requests (greedy Thinker), first one at a time and then the same 75 again, shuffled, 8 at a time:

• 40 questions against a ~2.3k-token fact list in the system prompt
• the same request 4 times
• 6 questions each about two audio clips in the first user turn, plus the last one without the clip
• 18 turns of scripted multi-turn history

Results:

• Stage-0 hit rate is 96.1% one at a time and 99.6% at c=8 (the c=8 pass repeats prompts already in the cache). Mean request time goes from 0.65–0.66 to 0.51–0.52 s one at a time, and from 2.97–3.02 to 1.46–1.52 s at c=8. All 600 responses returned text and audio, with no server errors.
• Fact answers are 42/44 in every run, and the two misses are the same with the cache on and off.
• One at a time, each setting reproduces its own text across restarts (75/75). Cache on vs off give the same text on 64/75. Most of the other 11 are rewordings ("712" vs "712 crates of glass jars."), but one audio question answers in Chinese with the cache off and in English with it on, and one multi-turn summary comes out different.
• At c=8, cache on vs off agree on 61–64/75, against 66/75 for off vs off. With the cache off at c=8, that audio question also comes back in English on one of the two servers.
• The audio clip is in the shared prefix: 209 prompt tokens with it vs 74 without.
• Running Whisper on the returned audio and comparing with the returned text shows no on/off pattern overall. Medians are 0.11–0.25 per run, and the large per-request values come from short number answers. One answer, "About 10 seconds or so.", has the same text in every run, but with the cache on its audio is 4.5–5.6 s instead of 2.1 s and Whisper hears garbled speech in it (all 4 cache-on runs, none of the cache-off runs).

[amy-why-3459]

Daily-Omni, Qwen3-Omni, concurrency 10. Same code for both arms: main 798fef99a (includes #9162 and #9193). The only difference is stage 0 enable_prefix_caching, which is what this PR turns on for qwen3_omni_moe.yaml. Not a checkout of the PR branch itself (e091c85, based before #9162/#9193).

Setup. Qwen/Qwen3-Omni-30B-A3B-Instruct, text only ({"modalities":["text"]}), local Daily-Omni, all 1197 questions, --no-oversample, --max-concurrency 10, one fresh server per arm. Two shared L20X (140 GB): Thinker on GPU 4, Talker + Code2Wav on GPU 5. Both arms use the same reduced memory caps so the cards stay shared (gpu_memory_utilization 0.58 / 0.42 / 0.08 instead of the shipped 0.9 / 0.6 / 0.1). Engine logs confirm enable_prefix_caching=False vs True on stage 0.

Accuracy. 857/1197 (0.7160) cache off, 856/1197 (0.7151) cache on. 1196/1197 predicted letters match. The one flip is 2zO-nlygkFo (30s, AV Event Alignment): gold D, cache off D, cache on C. All 1197 requests completed, no parse failures. Per-type and per-duration counts are otherwise identical.

Performance (request_throughput, median TTFT / e2e):

	cache off	cache on
throughput	1.309 req/s	1.544 req/s (+18.0%)
wall time	914.2 s	775.0 s
median TTFT	7.18 s	5.92 s
median e2e	7.27 s	6.01 s
p99 TTFT	21.83 s	14.29 s

Stage-0 prefix-cache hit rate with the cache on is 1.97% (235,504 / 11,965,481 tokens). Daily-Omni reuses videos across questions, but the question text differs and the bench shuffles requests, so the video tokens are not a stable long prefix.

Caveat. Single run. GPU 5 (also holding another process) spiked to 100% utilization during the cache-off run and stayed at or below 16% during cache-on, so the +18% is not a clean cache-only delta. The accuracy comparison is the firmer result: one answer changed out of 1197.

[amy-why-3459]

Seed-TTS EN, MiniCPM-o 4.5, concurrency 16. Same code for both arms: main 798fef99a (includes #9162 and #9193). The only difference is stage 0 enable_prefix_caching on minicpmo_4_5_turn.yaml, which is what this PR turns on for MiniCPM-o turn serving. Not a checkout of the PR branch itself (e091c85, based before #9162/#9193).

Setup. Local MiniCPM-o-4_5, one fresh server per arm, full English set (1088 utterances, --no-oversample, --max-concurrency 16). Request body: {"modalities":["text","audio"],"chat_template_kwargs":{"enable_thinking":false,"use_tts_template":true}}. One free L20X for the server, a second L20X for Whisper-large-v3 and WavLM (SEED_TTS_SIM_EVAL=1). Engine logs confirm enable_prefix_caching=False vs True on stage 0.

Accuracy. 1088/1088 completed in both arms, no missing PCM and no ASR failures.

	cache off	cache on
mean WER	0.0218	0.0198
mean SIM	0.8655	0.8652
median SIM	0.8689	0.8691

Performance:

	cache off	cache on
request throughput	3.98 req/s	6.34 req/s (+60%)
audio throughput	18.5 s/s	29.4 s/s
wall time	273.7 s	171.6 s
median audio TTFP	1.24 s	0.89 s
median e2e	3.46 s	2.37 s
median RTF	0.74	0.51
p99 e2e	32.7 s	10.5 s

Stage-0 prefix-cache hit rate with the cache on is 82.4% (134,688 / 163,442 tokens). That matches the 82.4% hit this PR reports for the full 1088 utterances.

Caveat. During startup the cache-off server failed two Code2Wav Whole-Euler graph captures with CUDA OOM and continued; the cache-on server captured those graphs. Both then served all 1088 requests. The accuracy numbers are unaffected. Part of the tail-latency and RTF gap can come from that eager fallback, not only from the prefix cache.

[amy-why-3459]

Looked at tip 40a38318 ("[Tests] Expect the Thinker prefix cache on in the Qwen3-Omni e2e checks") after the merges onto main.

Addressed since e091c85a.

• Offline e2e test_structured_multistage_config_reaches_runtime now asserts Thinker True and Talker/Code2Wav False — that unblocks the merge-gate case that would have failed with the default flip.
• Online-serving comment no longer says the deploy YAML is opt-in; it correctly notes that production turns stage 0 on and that _PREFIX_CACHE_SERVER_ARGS still force-enables Talker for that fixture.
• Branch is no longer stuck on the pre-#9162/#9193 merge base (merges through ea08ab61 are present).

Still open from the earlier review on e091c85a (not changed by this tip).

1. test_thinker_prefix_cache_defaults still stops at stage0.engine_extras.get(...). Please push the assertion through merge_pipeline_deploy / _build_engine_args and check the final yaml_engine_args["enable_prefix_caching"] for at least the two CUDA-on profiles (and keep the derived off profiles that only inherit, e.g. minicpmo_4_5_turn_mrv2_h200.yaml / qwen3_omni_duplex_mrv2.yaml).
2. minicpmo_4_5.yaml comment / layout: please answer why the default MiniCPM-o profile stays off on CUDA while the turn overlay is on, and either set stage 0 to an explicit false or document that callers must pass the turn deploy config. The dropped CUDA stage-1 regression rationale is still unexplained relative to the turn profile flipping the same switch.
3. Default side effect: text-only /v1/video/chat/stream under Qwen3-Omni now silently takes the incremental-prefill path via _stage0_prefix_caching_enabled. Worth a line in the PR description and a touch of docs/serving/video_stream_api.md.
4. Host VmRSS cost of the new default (Qwen ~5→15 GiB, MiniCPM-o ~4→7 GiB) is still not recorded next to the yaml flip / overlay-off snippet.

No new blocking findings in the tip itself beyond those carry-overs. Once (1)–(2) land, this looks ready.

[amy-why-3459]

Still open from the earlier review: this only checks the deploy/engine_extras layer. enable_prefix_caching is applied last via engine_args.update(ds.engine_extras) in _build_engine_args, so a merge-order regression would leave this green while production silently falls back to false.

Please add at least one assertion per CUDA-on profile that goes through merge_pipeline_deploy(...) (same pattern as the nearby test_step_audio2_* cases) and checks yaml_engine_args["enable_prefix_caching"]. Also worth covering the derived off profiles that only inherit (minicpmo_4_5_turn_mrv2_h200.yaml, qwen3_omni_duplex_mrv2.yaml).

[amy-why-3459]

This is the right fix for the merge-gate breakage called out earlier: stage 0 True, stages 1/2 False, and the assertion still exercises that deploy values reach VllmConfig. Thanks.

[amy-why-3459]

Looked at tip 0ee846a0 ("[Config] Check prefix-cache defaults on the engine args and document the cost").

Addressed since 40a38318.

• test_thinker_prefix_cache_defaults now asserts through merge_pipeline_deploy on yaml_engine_args["enable_prefix_caching"], and covers the derived-off profiles.
• minicpmo_4_5.yaml sets stage-0 enable_prefix_caching: false explicitly and explains duplex KV-window vs the turn overlay.
• docs/serving/video_stream_api.md now matches the new Qwen3-Omni default for incremental prefill.
• Yaml comments document the host-mirror cost and include an overlay-off snippet.

Still open on this tip.

1. The new test monkeypatches current_omni_platform.device_name but never calls _apply_platform_overrides. load_deploy_config only parses YAML; platform stage overrides are applied by _apply_platform_overrides (every sibling test in this file does that). As written, the npu/rocm/xpu/musa rows still see the shared-stage value and do not exercise the platform blocks the matrix is meant to lock. Please apply overrides for platform before the merge, same as test_minicpmo_talker_multi_frame_is_npu_scoped.
2. MiniCPM rows resolve with Q3_OMNI_ALL_STAGES_HF_CONFIG. Sibling MiniCPM tests use resolve_pipeline_config("minicpmo_4_5") (no Qwen HF config). Prefer that for the minicpmo_* cases so the pipeline under test matches the deploy.

Soft leftover: the host RSS note is still qualitative ("several GiB" / "a few GiB"); the measured ~5→15 GiB / ~4→7 GiB numbers from earlier would make the cost concrete next to the overlay snippet.

Once (1) lands, the config matrix looks ready to me.

[amy-why-3459]

load_deploy_config does not apply platforms.* overrides — that happens in _apply_platform_overrides (optionally reading current_omni_platform.device_name). Monkeypatching device_name alone leaves the shared-stage value in place, so the npu/rocm/xpu/musa parametrized rows do not actually check the platform blocks.

Please do something like:

deploy = _apply_platform_overrides(
    load_deploy_config(Path(get_deploy_config_path(yaml_name))),
    platform=platform,
)

before merge_pipeline_deploy, matching the other platform-scoped tests in this file.

Also, for the minicpmo_* cases, resolve with resolve_pipeline_config("minicpmo_4_5") (or deploy.pipeline) without Q3_OMNI_ALL_STAGES_HF_CONFIG — that HF stub is Qwen3-Omni-specific.

[amy-why-3459]

Looked at tip def52c83 (two commits since 0ee846a0: platform-override test fix, then drop host-RSS yaml notes).

Addressed since 0ee846a0.

• test_thinker_prefix_cache_defaults now calls _apply_platform_overrides(..., platform=platform) and pins current_omni_platform.device_name before merge_pipeline_deploy. That is what the npu/rocm/xpu/musa rows needed — they no longer stop at the shared-stage value.
• MiniCPM rows resolve via deploy.pipeline when present (falling back to the Qwen HF path only when the deploy has no pipeline key), which matches how the turn/base MiniCPM profiles are wired.

On def52c83. Dropping the qualitative host-RSS lines from the yaml comments is fine; the overlay-off snippet stays. The measured ~5→15 / ~4→7 GiB figures remain in the PR discussion if anyone wants them later.

No new blocking findings on this tip. The config matrix looks ready from my side.