[Perf][TTS] Restore Qwen3-TTS default_voice c=64 TTFP to v021 baseline

[linyueqian]

Summary

On qwen3_tts_high_concurrency.yaml from #3662, default_voice short c=64 median TTFP regressed ~2.18× vs v0.21.0rc1 (736 → 1604 ms on 2× H20). The #3662 author (@Sy0307) measured only a ~6 % delta at the merge commit c99df1eb, so the catastrophic part of the regression is in one or more of the 60 commits between c99df1eb and current main HEAD; which specific commit is still under bisect.

This PR restores v021-or-better TTFP on the affected cell by defaulting the code_predictor_prefix_graphs knob off in the bundled yaml and tightening a few adjacent code paths. Result: 710 ms (3.5 % better than v021).

Voice_clone deployments that relied on the captured prefix graphs can re-enable them via a one-line yaml override; the keys remain in the bundled file with false so the override is documented in-place.

Measured (default_voice short c=64, p=512, w=8, 2× H20)

state	median TTFP (ms)	gate ≤ 773 ms
v021 (0.21.0rc1)	736	—
#3662 merge c99df1eb (per @Sy0307)	~757	✓
vanilla current main + bundled yaml	1604	✗ (2.18×)
this PR	710	✓ (0.97×)

Changes

• Site 1 (qwen3_tts_talker.py): scalar decode-preprocess fast-path when the decode batch is small (≤ scalar_decode_preprocess_threshold, default 8) or contains no task_type=Base request. Both knobs exposed as connector_extra fields for yaml tuning.
• Site 2 (same file): raise the trailing-text compaction floor from 64 → 256 frames so short prompts no longer pay a mid-stream slice/copy. The original 64 is the legacy default if the connector_extra knob isn't set.
• Site 3 (qwen3_tts_high_concurrency.yaml): default code_predictor_prefix_graphs: false. This single knob alone is the dominant fix. Voice_clone deployments can re-enable via a one-line override.
• Site 4 (same yaml): widen decode_cudagraph_capture_sizes to [25, 49, 73, 97, 145, 169, 325] so default_voice's 49 / 145-frame chunks no longer pay re-compile cost.
• Tests: 12-case parametrized parity test between scalar fast-path and batched path, plus 4 routing-predicate unit tests. Runs in <1 s without GPU.

Scope honestly stated

Only the worst-regression cell (default_voice short c=64) has been measured on this branch. Other concurrencies and voice_clone cells are not yet verified. Two known risks:

1. voice_clone c=64 — [TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662's headline win. This PR defaults prefix-graphs off, so a vanilla voice_clone user loses that improvement unless they re-enable the flag in a downstream yaml. A reviewer-side check at voice_clone short c=64 with the override yaml closes this gap.
2. The unexplained 700+ ms cost of prefix graphs on current main — the dispatch is a single dict.in check per AR step (~100 ns), so the empirical 700+ ms cost must come from a CUDA-graph replay / memory-layout interaction inside the captured graphs. Needs a bisect across the 60 post-[TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662 commits to pin down the actual breaking change.

@Sy0307 — would appreciate your eyes on this. Two specific questions:

• Does the voice_clone-override-yaml story match your original intent for [TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662?
• Have you observed any perf change between c99df1eb and current main HEAD on your end? Your ~757 ms baseline plus my 1604 ms on the same yaml is what flagged the post-merge regression.

Test plan

• pytest tests/model_executor/models/qwen3_tts/test_decode_preprocess_parity.py — 12 cases pass
• sanity bench: default_voice short c=64 p=512 on 2× H20 — 710 ms
• full 60-cell sweep (default_voice × voice_clone × 6 concurrencies × 2 text lens)
• voice_clone short c=64 with prefix-graph override yaml: confirm ≤ 1.05× current main
• post-[TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662 bisect to localize the prefix-graph-cost amplifier
• quality eval (WER / SIM / UTMOS) at voice_clone short c=8 within ±2 % of v021

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[linyueqian]

Benchmark: this PR vs current main

Hardware: 2× H20. Stage 0 talker on GPU 0 (S0=64), Stage 1 Code2Wav on GPU 1 (S1=10). Deploy config: vllm_omni/deploy/qwen3_tts_high_concurrency.yaml.

Workload: default_voice (task_type=CustomVoice), seed-tts-text EN short bucket, --num-prompts 512 --num-warmups 8 --max-concurrency 64 --request-rate inf. Bench cmd is vllm bench serve --omni ….

build	median TTFP (ms)	P99 TTFP (ms)	successful / failed
v0.21.0rc1 (baseline)	736	1353	512 / 0
current main HEAD 748470c1 + bundled yaml	1604	1936	512 / 0
this PR (ca6b175e)	710	2092	512 / 0

Headline: this PR's median TTFP is 0.97× v021 and 0.44× current main at the cell where #3662 most visibly regressed.

The driver of the 1604 → 710 ms improvement is yaml-level: defaulting code_predictor_prefix_graphs: false. Sites 1, 2 and 4 are smaller contributors (about 200 ms in aggregate vs the 700+ ms prefix-graph drop). The code paths for prefix graphs themselves are untouched, so a downstream yaml that re-enables code_predictor_prefix_graphs: true falls back to the pre-PR behavior for that connector.

Other concurrencies (c=1, 8, 32, 128, 256) and long-text cells are still under measurement; I'll follow up with a wider sweep, and with the post-#3662 bisect to locate which commit amplified the prefix-graph cost.

[hsliuustc0106]

do we have a test case to avoid perf regression?

[hsliuustc0106]

Approved. The fix is correct and performance claims are well-evidenced.

What I validated:

• Root cause: code_predictor_prefix_graphs: true on the high-concurrency YAML is the dominant regression source. The knob is consumed by Qwen3CodePredictor._stage_connector_extra_config at qwen3_code_predictor.py:474-477 — turning it off is safe and the YAML comment documents the voice_clone override path.
• Scalar decode fast-path: The routing logic (_should_use_scalar_decode_preprocess) is correct — scalar path for small batches (≤8) and batches with no task_type=Base requests, batched path otherwise. 16 parity tests confirm byte-identical output.
• Config parsing: _stage_connector_extra_config mirrors the existing implementation in qwen3_code_predictor.py:566-573. _parse_non_negative_int safely handles None, invalid types, and negatives.
• CUDA graph capture: Adding 49 and 145 to decode_cudagraph_capture_sizes stops default_voice chunks from paying re-compile cost.
• Benchmarks: 710ms vs 1604ms (current main) vs 736ms (v021 baseline) measured on 2× H20. The table is clear and credible.
• Gates: All pass.

Follow-ups (non-blocking):

1. Only one cell (default_voice short c=64) has been measured. The full sweep is important before declaring victory — especially voice_clone c=64 with prefix graphs re-enabled to confirm ≤ 1.05× current main, as the PR itself notes.
2. The trailing_text_compact_min_frames knob change (64→256) affects how aggressively short prompts compact their tail — no specific test covers it. A follow-up quality check (WER / SIM) at the boundaries would be reassuring but is not gating.
3. The _stage_connector_extra_config fallback (connector_cfg when "extra" is missing) would be worth extracting to a shared utility in a follow-up cleanup.

[Sy0307]

Thanks for the fix. A few notes from my side.

1. For the voice_clone override story: when [TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662 was merged, code_predictor_prefix_graphs was true by default in the bundled qwen3_tts_high_concurrency.yaml. So changing it to false in [Perf][TTS] Restore Qwen3-TTS default_voice c=64 TTFP to v021 baseline #3839 does change the default behavior introduced by [TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662. I understand it as a mitigation: if the prefix-graph path currently hurts default/custom voice c64 TTFP, disabling it by default is reasonable, while voice_clone deployments can explicitly re-enable it in a downstream yaml.

2. For perf between c99df1eb and current main: on my single-H20 CustomVoice C=64,N=500 run, I did not reproduce a 1604ms TTFP regression. The post-[TTS][Perf] Optimize Qwen3-TTS high-concurrency serving #3662 numbers were around 756/1103-1118 ms median/p90; my main snapshot retest was 733/1075 ms with ~29.3x audio throughput. This is still not the same workload as your 2x H20 default_voice C=64,N=512 bundled-yaml run.

3. I agree with the scalar fast-path for decode preprocess. For small batches or batches without task_type == "Base", the batched preprocess path may not amortize its extra Python coordination cost. Routing CustomVoice/default_voice through the scalar path makes sense, and the parity test covers Base/CustomVoice consistency.

One concern: the new Code2Wav decode_cudagraph_capture_sizes look inactive under the bundled yaml, because Stage1 still has enforce_eager: true, and qwen3_tts_code2wav.py returns before enabling the inner decoder CUDA Graph. That part should either be documented as inactive/override-only, or guarded by a separate explicit knob that decouples Stage1 engine eager mode from inner Code2Wav decoder graph.

[linyueqian]

Closing this for now — re-tested against latest main (8f45e68b) on a second hardware class (2× L20X). On L20X this PR is essentially a wash vs vanilla main (1637 → 1584 ms median TTFP at default_voice c=64 p=512); the dramatic improvement is H20-specific. That makes the PR a workaround for an H20-only regression whose root cause we haven't diagnosed, and not the right shape of fix.

Will open a separate issue with the H20 measurements and ping @Sy0307 for root-cause investigation. H20 users who need an immediate mitigation can set code_predictor_prefix_graphs: false in their downstream yaml.