[Sy0307]

Supersedes #4594 after branch rename to dev/voxcpm2-unified-decode-graph.

Purpose

This PR improves VoxCPM2 TTS serving latency by reducing Python routing and kernel launch overhead in the decode path.

VoxCPM2 decode currently crosses multiple small stages per step, including scaffold decode, FSQ/fusion, residual decode, CFM solve, feature encoding, stop prediction, and audio collection. The segmented path leaves overhead between these stages, especially under high concurrency.

This PR adds CUDA graph support for the VoxCPM2 decode path while keeping the default behavior quality-safe.

Key changes:

• Add decode tail graph support for the CFM / feature encoder / stop prediction tail.
• Add an opt-in full unified decode graph path for scaffold, residual, CFM, feature encoding, and audio-side state commit.
• Add runner-assisted attention metadata support so full decode graph capture can reuse vLLM-built paged-attention metadata instead of reconstructing it inside the model layer.
• Add bounded graph bucketing for dynamic decode batch sizes, avoiding all-size graph capture and reducing fallback for non-power-of-two active batches.
• Preserve VoxCPM2 runtime configuration through the existing deploy YAML / engine overrides path.
• Add static regression coverage for the runner-assisted graph hooks and bucketed unified graph path.

Review Follow-up

This branch keeps the runner changes opt-in and fail-closed:

• Models without get_runner_assisted_full_attention_metadata_request keep the normal runner path.
• VoxCPM2 graph bucketing, capture policy, graph lifecycle, and fallback decisions stay in the VoxCPM2 talker layer.
• Lazy tail/unified graph capture now falls back instead of failing the request if compile/capture raises.
• Tail graph pre-capture uses tail-specific candidates instead of unified graph buckets.
• Deterministic CFM noise disables the tail/unified graph paths so deterministic mode keeps the existing per-request noise behavior.

Test Plan

• ruff check .
• ruff format --check .
• python3 -m py_compile vllm_omni/model_executor/models/voxcpm2/voxcpm2_talker.py vllm_omni/worker/gpu_ar_model_runner.py tests/model_executor/models/voxcpm2/test_runner_assisted_unified_graph_static.py
• python3 tests/model_executor/models/voxcpm2/test_runner_assisted_unified_graph_static.py
• Remote H20 c=64 VoxCPM2 HTTP benchmark
• Verified output bytes/request stayed in the expected range, with 100% success and no stop runaway

vLLM Version:

0.23.0

vLLM-Omni Commit:

d295fcd

Test Result

H20 benchmark, VoxCPM2 HTTP serving, c=64.

Metric	Baseline segmented	Unified graph fixed	Delta
TTFB mean	1.945s	1.06-1.10s	~44-46% lower
Wall mean	8.26s	4.56s	~45% lower
Throughput	6.4 req/s	11.7-12.1 req/s	up to ~89% higher
Bytes/request	498-517 KB	~526 KB	expected range
Success	100%	100%	no regression observed

Kernel/profile evidence:

• Baseline segmented path before this commit used separate scaffold/residual graphs plus compiled CFM/tail execution; profiling showed the split decode work at roughly 12-15 ms/request-equivalent under c=64, with Python routing between graph segments.
• Unified decode graph replay after warmup is about 14 ms/batch for batch=64 in the optimized path; the HTTP numbers below include scheduler, IPC, streaming, and audio collection overhead.

Latest review-follow-up smoke on H20, c=64:

• Warm/capture run: 64/64 success, 0 errors, Bytes mean=544.2KB.
• Steady run: TTFB mean=1.099s, Wall mean=4.478s, Throughput=11.8 req/s, Bytes mean=513.0KB.

Additional validation:

• Warm/capture run completed with 64/64 success, 0 errors, Bytes mean=525.9KB.
• Steady run 1: TTFB mean=1.101s, Wall mean=4.566s, Throughput=11.7 req/s, Bytes mean=525.5KB.
• Steady run 2: TTFB mean=1.060s, Wall mean=4.563s, Throughput=12.1 req/s, Bytes mean=527.8KB.

Runner-assisted bucket logs confirmed non-power-of-two active batches are padded into bounded graph buckets, for example:

• num_reqs=47 padded_reqs=48
• num_reqs=23 padded_reqs=24
• num_reqs=55 padded_reqs=56

This avoids the previous exact-size/power-of-two fallback pattern without pre-capturing every batch size.

PTAK @linyueqian @hsliuustc0106

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[linyueqian]

Thanks for this @Sy0307. I ran an end-to-end A/B on the current head (66b51741) with vLLM 0.23 on a Hopper-class GPU and found a reproducible audio-quality regression in the decode tail graph at higher concurrency. Flagging it before this is enabled by default.

[Blocker] Decode tail graph degrades audio at batch sizes above 8

Method. Identical texts, identical hardware and environment; the only change between the two columns is enable_unified_decode_graph + enable_decode_tail_graph. Audio is scored by a Whisper-medium ASR round-trip (word error rate vs the input text). I ran it both offline (engine.generate) and online (vllm-omni serve + concurrent /v1/audio/speech, the same path as the benchmark in the PR description).

Path (16 concurrent / batch)	graphs OFF (baseline)	graphs ON
Online HTTP /v1/audio/speech	mean WER 0.017	mean WER 0.359 (worst 0.909)
Offline batched	mean WER 0.000	mean WER ~0.28 (two runs)
Either path at 8 concurrent	clean	clean

The baseline on the exact same serving path is clean, so this is not a harness or environment artifact.

Isolated to the tail graph. Disabling only the unified graph while keeping the tail graph on still degrades at 16 concurrent (mean WER 0.29), whereas the bs=1 unified graph is clean in every run. So the regression is specifically the decode tail graph at batch sizes above 8.

The output is genuine synthesis drift, not ASR noise. Two examples (graphs ON, 16 concurrent):

• ref The quick brown fox jumps over the lazy dog near the riverbank produced The quick brown fox whelps over the lazy brown moody w-
• ref Children laughed and played in the park until the sun went down produced Children laughed and played and cried and the rest of the children just kind of went down

This is the kind of degradation a bytes-per-request check cannot catch, since the degraded audio is still the right length.

Repro. Serve the default voxcpm2.yaml, send 16 or more concurrent /v1/audio/speech requests with distinct text, and transcribe the outputs. Compare against the same server with enable_decode_tail_graph: false.

Suggestion. Gate or fix the decode tail graph for batch sizes above 8 before enabling it by default (the default ships enable_decode_tail_graph: true with max_num_seqs: 32, so any real concurrency above 8 hits this). The bs=1 unified graph looks quality-safe and could ship independently.

[Minor, unconfirmed] One worker crash during graph capture

In one early offline run the engine worker died (hard CUDA abort, no Python traceback) right after the bs=8 tail-graph capture. It did not reproduce in roughly seven later runs and coincided with unrelated GPU load on the shared box, so it is most likely transient contention rather than a defect; I could not confirm it. Noting it only for awareness. The new try/except capture-to-segmented fallback is a good safety net, though it will not catch a hard CUDA abort.

[hsliuustc0106]

Unified decode graph optimization looks correct.

[Sy0307]

The latest patch now defaults VoxCPM2 to the full unified decode graph path and removes the decode-tail graph path, so we only keep one graph optimization strategy. Based on the c=64, prompts=500 benchmark sweep, large admission batches caused ITL to grow quickly; we set the deploy default to max_num_seqs=8 with unified graph pre-capture sizes 1,2,4,8 as the safer default. In our measurements, max=12 had the best raw throughput, while max=8 was close and more conservative for default serving.

[linyueqian]

Thanks for iterating so quickly @Sy0307. Removing the tail graph and consolidating on a single strategy was the right call. I went through the latest head end to end on an H200-class card, including fixing the boot crash locally so I could exercise the new path. Notes below.

🔴 [blocking] The default config crashes at startup, with a root cause and a fix

Serving the default voxcpm2.yaml (14988475), the engine core dies during warmup graph capture and then rejects all requests (HTTP 000). Root cause: in _capture_unified_decode_graph, the capture estimator comes from _estimator_for_unified_capture, which, despite being compiled with options={"triton.cudagraph_trees": False}, still carries inductor's cudagraph-trees wrapper (it shows up as cudagraph_trees.deferred_cudagraphify in the trace). Running it inside the manual torch.cuda.graph() capture raises Cannot prepare for replay during capturing stage, then a fatal use_count > 0 allocator assert that the try/except cannot recover from.

What worked for me: compile the capture estimator and feat-encoder fresh from the eager module, via the existing helper that disables inductor cudagraphs during compile, instead of reusing _estimator_for_unified_capture:

_est_eager = getattr(original_estimator, "_orig_mod", original_estimator)
capture_estimator = self._voxcpm2_compile_without_inductor_cudagraphs(_est_eager, mode=None, fullgraph=False)
tts.feat_decoder.estimator = capture_estimator
# same for feat_encoder via _orig_mod

After that the unified graph captures bs=1..32 cleanly and serves normally. Happy to send the full diff.

🟢 Quality looks good at scale

With the boot fix in place, I checked that the unified graph stays intelligible as batch grows: ASR word error rate 0.006 at bs<=8 and bs=16, and 0.000 at bs=32. So the unified path does not show the high-batch divergence the earlier tail-graph path had.

❓ [question] On throughput I could not separate the graph from batching on this hardware

I tested on an H200-class card, which runs roughly 2x the throughput of the H20 in your table, so batched-eager here already lands where your graph numbers are, and I cannot isolate the graph's marginal value from this hardware. Sustained req/s (median of reps), unified graph vs batched-eager at the same config:

in-flight	unified graph	batched-eager
8	9.65	9.76
32	10.3	10.2

Within noise here. Separately, turning batching off (enable_batched_cfm=false) drops to 5.3 req/s, so on this card the ~2x is the batching, which the unified graph force-enables in _normalized(). To line this up with your H20 table: did the "baseline segmented" row have enable_batched_cfm: true? If it did and the graph still gave ~2x on H20, that is a genuine H20-specific win that I would not see on an H200, and a per-stage timing breakdown on H20 would confirm it.

🟡 [design] The shared model-runner change is large for a single-model gain

gpu_ar_model_runner.py grows by about +192/-45, and VoxCPM2 is the only consumer of the runner-assisted hooks (gated on num_reqs > 1). If a model-specific runner is warranted, vllm-omni already resolves worker_cls per stage, so a VoxCPM2ARModelRunner subclass could carry it without touching the shared base.

Where I land

1. The boot crash is the one blocking item, and it has a clean fix worth landing regardless.
2. The unified graph is quality-safe up to bs=32 in my runs.
3. The graph-vs-batching attribution on H20 is the open question; confirming the baseline config (or a per-stage timing breakdown on H20) would settle whether the runner-assisted machinery is buying the throughput or batching is.

[hsliuustc0106]

This runner-assisted path needs at least one behavioral test. These checks mostly grep source strings, so the metadata request, padding, and context reset logic can break while this test still passes.

[hsliuustc0106]

This enables the new graph path by default, but the PR evidence only checks success and output byte counts. Please add baseline-vs-PR audio quality validation for the same prompts/seeds before making this the default.

[Sy0307]

Benchmark update for the latest PR head (0cc02cebc2fbaf1439b3c3f4399cc4a911f306b8):

I re-ran an apples-to-apples E2E benchmark on the H20 remote box with c=64, prompts=500, and max_num_seqs=64 for both branches. Each run first used a c=64, prompts=64 warmup before the measured p=500 run. The PR run also set unified_decode_graph_max_batch_size=64 so that the full unified graph path could actually cover the max-64 batches.

Metric	origin/main baseline	PR latest	Delta
Success	500/500	500/500	same
Throughput	4.8 req/s	11.5 req/s	+139.6%
Total wall clock	104.78s	43.54s	-58.4%
TTFB mean	1.903s	2.008s	+5.5% slower
TTFB p50	1.965s	1.911s	-2.7% faster
TTFB p95	2.308s	3.151s	+36.5% slower
Wall mean	13.057s	5.376s	-58.8%
Bytes/request	511.1KB	517.8KB	close / normal

The main improvement is throughput and request completion latency: wall mean drops by ~59% and throughput is ~2.4x higher. TTFB is more mixed under max_num_seqs=64: p50 is slightly better, mean is roughly flat/slightly worse, and p95 is worse. That matches the expected tradeoff of admitting large decode batches: the unified graph makes decode much faster overall, but some requests still wait longer for their first chunk under large-batch scheduling.

I also checked the PR server logs. The full unified graph path was actually used and captured max-64 buckets during warmup, including batch_size=64,56,48,40,32,24,16,8,4,2,1. I did not see runner_full_metadata_missing, capture_failed, or exception traces in the PR run logs.

[linyueqian]

Correction to my previous comment: you are right, and I can now reproduce the win.

After the boot fix (thanks for picking that up), I re-ran on an H200-class card at your max_num_seqs=64, unified graph vs batched-eager on the same head (N=128, median of 5 reps):

in-flight 64	unified graph	batched-eager
Throughput	23.2 req/s	10.5 req/s

That is ~2.2x from the graph, in line with your 2.4x on H20. My earlier "within noise / it is mostly the batching" was a batch-size artifact: I had only measured up to bs=32, where the graph and batched-eager are tied on this card, so I missed the win that only shows up at the large decode batches the PR actually targets. It grows with batch because the segmented path still does per-request Python work each step, which the unified graph collapses into a single replay.

Quality at bs=64 is clean as well (ASR word error rate 0.000 over 64 prompts), which also covers @hsliuustc0106's request for baseline-vs-PR audio validation before defaulting. And the boot fix holds: the unified graph captured bs=1..64 with no capture_failed or exception traces.

So the throughput question is settled in the PR's favor. The remaining items are the smaller ones already raised: the shared-runner footprint (still about +196/-47 and VoxCPM2-only, so a VoxCPM2ARModelRunner subclass via per-stage worker_cls would keep it out of the shared base), and @hsliuustc0106's point that the static tests mostly grep source strings and want at least one behavioral check of the metadata-request / padding / context-reset logic.

[linyueqian]

need to fix dco

[hsliuustc0106]

Please fix pre-commit too.

[Shirley125]

can this method be applied to other models?

[Sy0307]

currently, can we monitor the real batch size?

Added.

[linyueqian]

Confirmed fixed on my side too. I re-ran the same A/B on the latest head 01cc1c9 on an H20 (141 GB) with vLLM 0.23, and both issues I flagged are gone. Only enable_unified_decode_graph differs between ON and OFF; both keep enable_batched_cfm: true.

Config	ON (unified)	OFF (batched-eager)
Default bs=8, WER	0.000 (clean)	0.000
Default bs=8, throughput	10.6 req/s	8.6 req/s
bs=64, WER	0.000 (clean)	0.000
bs=64, throughput (p=256, c=64)	16.5 to 17.1 req/s	8.1 to 8.5 req/s

WER is a whisper round-trip on the same sentences; every OFF control is 0.000, which validates the scorer. So the batched (bs>1) corruption is gone (default was 0.74 to 0.80 before, bs=64 was 0.98), and the bs=64 unified path now runs about 2x over batched-eager instead of collapsing. Your root cause matches what I saw: the pre-capture path was capturing bs>1 graphs while the forward context still held bs=1 attention metadata, and removing that (lazy capture under runner-assisted FULL metadata) fixes both.

Two small notes, neither a blocker:

1. One bs=64 run hit EngineDeadError (hard abort during the startup-warmup graph capture) when another tenant was holding ~53 GB on the same shared GPU; on a clean card the exact same config booted, captured all buckets, and ran 256/256 clean. So the large-batch capture seems to need some scratch headroom and can OOM-crash under memory pressure. The shipped default caps at bs=8, which is more forgiving, so this is mostly a caution for anyone raising unified_decode_graph_max_batch_size on a busy card.

2. Version note for anyone reproducing: this head needs vLLM 0.23 (it imports vllm.utils.torch_utils.supports_xccl, which 0.24 removed), matching the CI VLLM_VERSION: v0.23.0.

Thanks for the quick turnaround on the fix.

[linyueqian]

test on 0.24. lgtm

[hsliuustc0106]

This gate does not match the effective model-side default. _VoxCPM2RuntimeConfig.from_vllm_config() returns the dataclass defaults when no voxcpm2_runtime_config is present, and that default currently has enable_unified_decode_graph=True; it also disables the graph when deterministic_cfm_noise is true. This scheduler only checks hf_config.voxcpm2_runtime_config, so direct/non-deploy launches can have the model trying to use unified graphs without the deferral policy, while deterministic-noise launches can still defer waiting even though the model cannot use the graph. Please derive this from the same normalized runtime config/source as the model so the admission policy matches the actual graph availability.

[hsliuustc0106]

One remaining issue: the VoxCPM2 scheduler gate for unified decode graph does not match the model-side runtime config/defaults, so some launch paths can enable the graph without the admission policy, while deterministic-noise launches can defer even though the graph is disabled: #4606 (comment)

The earlier core-module concern looks addressed: the shared scheduler now only exposes a generic deferral hook, and the VoxCPM2 policy is model-local. I did not see another model-specific leak in the shared runner hooks.

Current gate risk: AMD/Intel/NPU Buildkite are still failing and the main Buildkite lane is pending.

[Sy0307]

One remaining issue: the VoxCPM2 scheduler gate for unified decode graph does not match the model-side runtime config/defaults, so some launch paths can enable the graph without the admission policy, while deterministic-noise launches can defer even though the graph is disabled: #4606 (comment)

The earlier core-module concern looks addressed: the shared scheduler now only exposes a generic deferral hook, and the VoxCPM2 policy is model-local. I did not see another model-specific leak in the shared runner hooks.

Current gate risk: AMD/Intel/NPU Buildkite are still failing and the main Buildkite lane is pending.

Thx and will handle it.