[ZhengWG]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Reduce prefix-cache overhead during decode and first-request startup.

2. What is the highest-leverage variable for that outcome?

The main variable is CPU time spent on engine-thread cache bookkeeping and host-mirror initialization.

3. How will this work move that variable?

Buffer pure-decode rows, batch writes, vectorize slot mapping, and eliminate unnecessary host-mirror pinning and zero-filling.

4. Whose customer or user needs does this address, and what are those needs?

Users serving multimodal models with prefix caching need higher throughput and lower startup latency, even when cache hit rates are low.

Purpose

Part [2/N] of the series that turns the omni prefix cache on for Qwen3-Omni and MiniCPM-o (#7902): [1/N] #9162 fixes the collapse under concurrent hits, this PR removes the per-step cost at 0% hit. It applies to main on its own; whichever of #9162 and this PR lands second needs a small rebase in _register_step_writes.

With the cache on, every step is saved whether or not anything hits. At 0% hit, main loses up to 6.5% on Qwen3-Omni, 19–20% on MiniCPM-o decode, and 21.6% on Qwen3-Omni decode with varying output lengths (C64). Under that last load py-spy puts 27–32% of the engine thread in save_outputs: each save opened one write per request, so every request paid a slot-table update, a commit and a pool scatter. The host mirror was also pinned and zero-filled on the first request (+6.8 s for Step-Audio 2 mini).

This PR:

• One write per step. A step's immediate (JOIN_NEXT_STEP) rows share one WriteTask; each request still tracks it, so finish/abort and hit reads are unchanged. Deferred writes stay per request.
• Buffers pure decode steps. When every request decodes one token and nothing arrives, finishes or is preempted, the GPU AR runner skips the save and writes up to block_size steps at once (bounded by staging_capacity_tokens and gpu_staging_bytes). Any other step first writes the buffer, partial blocks included.
• Vectorizes step_slots_cpu: 302 → 21 µs for a 16-request decode step.
• Plain torch.empty host mirror: only CPU copies touch it; reads zero the slots that were absent when planned.

Test Plan

vLLM Version: 0.32.0

vLLM-Omni Commit: #9162 + these commits

Test bases GPU runs on main da572cddb38383deae9caef33b8d490e292ef071.

• Unit: pytest tests/core tests/worker/test_gpu_ar_model_runner.py tests/worker/test_gpu_ar_model_runner_cuda.py tests/config/test_config_factory.py.
• GPU: Qwen3-Omni-30B-A3B (qwen3_omni_moe.yaml, 2× H20-3e, 4 rounds) and MiniCPM-o 4.5 V1 (minicpmo_4_5_turn.yaml, 1× H20, 3 rounds), cache off vs on, fresh server per run, rounds alternating off / on on the same GPUs. Main cache on was measured the same way. py-spy (20 s) on the stage-0 engine during DECV_c64.
• Reproduction: serve with a deploy overlay that only sets stage 0's enable_prefix_caching (true / false); for MiniCPM-o the overlay also sets stage 0 max_model_len: 17408 and the server runs with --trust-remote-code. Run a warmup bench, then the benches below. Hit rates are the delta of stage 0's vllm:prefix_cache_{hits,queries}_total on /metrics before and after each bench.

server, overlay, bench and accuracy commands

vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --deploy-config overlay.yaml \
  --stage-init-timeout 600 --init-timeout 900 --log-stats
vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code --deploy-config overlay.yaml \
  --stage-init-timeout 600 --init-timeout 900 --log-stats

# overlay.yaml, Qwen3-Omni
base_config: <checkout>/vllm_omni/deploy/qwen3_omni_moe.yaml
stages:
  - stage_id: 0
    enable_prefix_caching: true  # false for cache off
---
# overlay.yaml, MiniCPM-o
base_config: <checkout>/vllm_omni/deploy/minicpmo_4_5_turn.yaml
stages:
  - stage_id: 0
    max_model_len: 17408
    enable_prefix_caching: true  # false for cache off

# bench <model> <extra-body> < table; columns: name shared_prefix random_len output range_ratio concurrency prompts seed
bench() {
  while read -r name prefix input output ratio c n seed; do
    vllm bench serve --omni --model "$1" --backend openai-chat-omni --endpoint /v1/chat/completions \
      --dataset-name random --random-prefix-len $prefix --random-input-len $input \
      --random-output-len $output --random-range-ratio $ratio --ignore-eos --extra-body "$2" \
      --max-concurrency $c --num-prompts $n --request-rate inf --seed $seed --num-warmups 0 \
      --ready-check-timeout-sec 0 --trust-remote-code --save-result --result-filename $name.json
  done
}

bench Qwen/Qwen3-Omni-30B-A3B-Instruct '{"modalities": ["text"]}' <<'BENCHES'
warm           512 512  16  0.0 4  16  50001
P1024_h0_c8    0   1024 8   0.0 8  256 50003
P4096_h0_c1    0   4096 8   0.0 1  64  50014
P1024_h50_c64  512 512  8   0.0 64 256 50009
DEC100_c32     0   100  100 0.0 32 256 50059
DEC100_c64     0   100  100 0.0 64 512 50060
DECV_c32       0   100  100 0.5 32 256 50063
DECV_c64       0   100  100 0.5 64 512 50064
BENCHES

bench openbmb/MiniCPM-o-4_5 '{"modalities": ["text"], "chat_template_kwargs": {"enable_thinking": false}}' <<'BENCHES'
warm           512 512   16  0.0 4  16  50061
P256_h0_c32    0   256   8   0.0 32 512 50063
P1024_h0_c32   0   1024  8   0.0 32 256 50070
P1024_h0_c64   0   1024  8   0.0 64 256 50071
P4096_h0_c32   0   4096  8   0.0 32 128 50088
P4096_h0_c64   0   4096  8   0.0 64 256 50089
P8192_h0_c32   0   8192  8   0.0 32 128 50099
P16384_h0_c32  0   16384 8   0.0 32 64  50106
P16384_h0_c64  0   16384 8   0.0 64 128 50107
DEC100_c32     0   100   100 0.0 32 256 50129
DEC100_c64     0   100   100 0.0 64 512 50130
DECV_c32       0   100   100 0.5 32 256 50063
DECV_c64       0   100   100 0.5 64 512 50064
BENCHES

# accuracy, MiniCPM-o
python tests/e2e/accuracy/qwen3_omni/run_qwen_omni_acc_benchmark.py --host 127.0.0.1 --port $PORT \
  --model openbmb/MiniCPM-o-4_5 --result-dir $OUT --num-prompts 2000 --max-concurrency 10 --skip-daily-omni \
  --seed-tts-dataset-path $SEED_TTS_EVAL --seed-tts-wer-save-items --max-seed-tts-mean-wer 0.05 \
  --seed-extra-body-json '{"modalities": ["text", "audio"], "chat_template_kwargs": {"enable_thinking": false, "use_tts_template": true}}' \
  --trust-remote-code --seed-tts-eval-device cuda:0

Test Result

• Unit at the PR head: 1145 passed, 2 skipped (CPU).
• Throughput vs cache off (0% hit unless noted):

bench	engine hit	cache off (req/s)	main, cache on	#9162 + this PR	range
Qwen3-Omni P1024_h0_c8	0.0%	15.70	−6.4%	−0.9%	−6.2% .. +3.2%
Qwen3-Omni P4096_h0_c1	0.0%	3.24	−5.2%	−0.3%	−1.6% .. +0.7%
Qwen3-Omni P1024_h50_c64	49.4%	19.44	+68.8%	+76.5%	+75.7% .. +77.2%
Qwen3-Omni DEC100_c32	0.3–64.7%	19.25	+4.0%	+5.8%	+4.5% .. +8.0%
Qwen3-Omni DEC100_c64	0.0%	26.77	−6.5%	−0.8%	−2.0% .. −0.3%
Qwen3-Omni DECV_c32	0.0%	16.66	−11.6%	−3.0%	−4.1% .. −1.8%
Qwen3-Omni DECV_c64	0.5%	22.41	−21.6%	−2.1%	−3.2% .. −0.8%
MiniCPM-o P256_h0_c32	0.4%	32.12	−6.6%	−0.7%	−1.0% .. −0.6%
MiniCPM-o P1024_h0_c32	0.0%	8.93	−3.5%	−1.4%	−1.5% .. −1.3%
MiniCPM-o P1024_h0_c64	0.0%	8.93	−3.3%	−0.6%	−1.2% .. −0.2%
MiniCPM-o P4096_h0_c32	0.0%	2.17	−0.7%	−0.4%	−0.5% .. −0.4%
MiniCPM-o P4096_h0_c64	0.0%	2.17	−0.6%	−0.5%	−0.5% .. −0.5%
MiniCPM-o P8192_h0_c32	0.0%	1.01	−0.1%	−0.1%	−0.2% .. −0.1%
MiniCPM-o P16384_h0_c32	0.0%	0.44	−0.3%	−0.3%	−0.3% .. −0.2%
MiniCPM-o P16384_h0_c64	0.0%	0.44	−0.2%	−0.3%	−0.4% .. −0.2%
MiniCPM-o DEC100_c32	0.0%	23.39	−18.8%	−2.5%	−2.7% .. −2.3%
MiniCPM-o DEC100_c64	0.2%	23.87	−19.3%	−1.8%	−3.1% .. −0.8%
MiniCPM-o DECV_c32	0.0%	24.11	−19.3%	−2.5%	−2.8% .. −2.2%
MiniCPM-o DECV_c64	0.4%	23.82	−19.1%	−2.5%	−4.3% .. −0.0%

• Stage-0 host RSS: Qwen3-Omni 15.2–16.1 GiB (main cache on 21.7–32.5, off 5.2–5.3); MiniCPM-o 7.3–7.6 GiB (main 9.0–9.1, off 4.4–4.7). First request after start: Step-Audio 2 mini 1.84–1.89 s vs 8.3–8.9 s main cache on (1.83–1.84 s off).
• Accuracy, MiniCPM-o Seed-TTS EN (1088 utterances, 82.4% hit): WER 0.0207 vs 0.0215 cache off, SIM 0.8645 vs 0.8644.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.

Note

To avoid agent spam, vLLM-Omni limits contributors without write access to 5 open PRs. If this blocks well-intentioned critical work, please contact a committer to request bypass list access.
(anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-21T03:41:43.066149Z We couldn’t complete this review. Comment @codex review to try again.	c05f9be	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md.

Module owners: @Gaohan123 @fake0fan @Sy0307

Routing: @Gaohan123 via module of the changed files, CODEOWNERS; @fake0fan via module of the changed files; @Sy0307 via module of the changed files

@ZhengWG, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit ccbedc1b3c4f produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[hsliuustc0106]

@ZhengWG For c05f9be7af8c2c33abcfc7b22a3b592feab9e8b9, the model, hardware and workload details are useful, but #<PR1> + these commits and “main” do not identify the measured revisions.

• Supply the exact baseline, PR1 and combined measured-head SHAs, with per-round result files mapped to each revision. Confirm the vLLM version and matching model/checkpoint, GPU configuration, workload, cache settings and measurement conditions for each comparison; keep the v0.31 startup result separate from the v0.32 throughput results.

Acceptance: the reported cache-overhead and startup improvements can be traced to a reproducible matched baseline/head comparison, with any PR1-dependent effect distinguished from this PR. This is an evidence request, not a demonstrated code defect; I have not independently run these benchmarks.

[amy-why-3459]

@amy-why-3459

[amy-why-3459]

Went through the manager/controller/runner-mixin changes and the decode-buffer tests. The ordering looks right to me: the flush runs before new_step_starts on any non-pure step, so finishes, preemptions, and new hits always see the buffered rows. The _hit_spans check in save_outputs is also safe, since publish clears those spans. Two smaller points, both about device memory on the buffered path (inline).

Also, the branch currently conflicts with main, so the _register_step_writes rebase mentioned in the description is still pending.

[amy-why-3459]

This buffer is capped by cfg.gpu_staging_bytes, but its clones aren't charged to the controller's GPU-byte budget, which uses the same 512 MiB limit for its own device clones. So the worst case is roughly 2× gpu_staging_bytes of device memory held for the cache, plus the torch.stack copy at flush. All of that is allocated after KV-cache profiling, and Qwen3-Omni stage 0 runs at gpu_memory_utilization: 0.9. The realistic decode footprint is small, but could the buffer use its own, much smaller cap, or take a budget ticket?

[amy-why-3459]

Each buffered row is now copied on device three times: val.clone() when it is parked, torch.stack here, and then hidden_states[:n].clone() / val[:n].clone() again in _split_step_outputs. The stacked tensors are already fresh, unaliased storage, so the last clone is redundant for this caller. It's cheap at decode sizes, but it's easy to skip, e.g. with a flag telling save_outputs that its inputs are already owned.

[ZhengWG]

Went through the manager/controller/runner-mixin changes and the decode-buffer tests. The ordering looks right to me: the flush runs before new_step_starts on any non-pure step, so finishes, preemptions, and new hits always see the buffered rows. The _hit_spans check in save_outputs is also safe, since publish clears those spans. Two smaller points, both about device memory on the buffered path (inline).

Also, the branch currently conflicts with main, so the _register_step_writes rebase mentioned in the description is still pending.

@amy-why-3459 Already fixed.

[amy-why-3459]

Re-checked the two new commits (e5de4d3, 2a36aa2); the merge from main is not covered.

Both earlier points look resolved:

• Each parked decode step is now charged through reserve_device_bytes before the clone, and the flush releases the whole hold before save_outputs charges the stacked rows through its own ticket, so the buffer and the step clones share one gpu_staging_bytes budget. The reserve runs outside the state lock, and the new test checks that.
• owned=True drops the third device copy. The unpadded guard in _split_step_outputs makes sure a live, padded buffer can't take that path by mistake.

Nothing further from me on this delta.