[Benchmark] Universal TTS benchmark: Qwen3-TTS + VoxCPM2 with 3 task types (voice-clone/default/design)

[linyueqian]

Summary

• Adds universal TTS benchmark covering 3 task types: voice clone, default voice, voice design
• Extends seed_tts_dataset.py with SeedTTSTextDataset (default voice) and SeedTTSDesignDataset (voice design)
• Fixes seed_tts_row flag so WER/UTMOS PCM capture runs for all task types, not just voice clone
• Adds bench_tts.py — a model-agnostic CLI backed by model_configs.yaml registry; adding a new TTS model requires only a YAML entry, no code changes
• Adds plot_results.py — bar-chart visualization (TTFP / E2EL / RTF / throughput) for comparing runs or task types
• Adds bench_voxcpm_offline.py — offline VoxCPM benchmark using Omni/AsyncOmni directly (sync + streaming, voice cloning, torch profiling)
• Wires both Qwen3-TTS (3 tasks) and VoxCPM2 (voice clone) into DFX nightly perf dashboard via test_tts.json
• Includes 20-prompt voice-design dataset (benchmarks/build_dataset/seed_tts_design/en/meta.lst)
• Removes old per-model benchmark directories (benchmarks/qwen3-tts/, benchmarks/voxcpm/) — superseded by this framework

Metric coverage per task

Task	RTF	TTFP	Throughput	WER	SIM	UTMOS
voice_clone	✓	✓	✓	✓	✓	✓
default_voice	✓	✓	✓	✓	—	✓
voice_design	✓	✓	✓	✓	—	✓

SIM is skipped for tasks without a reference audio (empty seed_tts_ref_wav_path → sim_skipped_no_ref counter in seed_tts_eval.py).

Models

• Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice: voice_clone + default_voice + voice_design
• Qwen/Qwen3-TTS-12Hz-1.7B-Base: voice_clone only
• openbmb/VoxCPM2: voice_clone only

New files

File	Purpose
benchmarks/tts/bench_tts.py	Universal serving benchmark CLI
benchmarks/tts/model_configs.yaml	Model registry (add new TTS model here)
benchmarks/tts/plot_results.py	Visualization: bar charts + markdown table from JSON results
benchmarks/tts/bench_voxcpm_offline.py	Offline VoxCPM benchmark (sync/streaming, profiling)
benchmarks/tts/stage_configs/qwen3_tts.yaml	Qwen3-TTS stage config for CLI
benchmarks/tts/stage_configs/voxcpm2.yaml	VoxCPM2 stage config for CLI
benchmarks/build_dataset/seed_tts_design/en/meta.lst	20 voice-design prompts
tests/dfx/perf/stage_configs/voxcpm2.yaml	VoxCPM2 DFX nightly config
tests/dfx/perf/tests/test_tts.json	Universal TTS DFX benchmark matrix
tests/benchmarks/test_seed_tts_dataset_variants.py	Unit tests for new dataset classes
tests/benchmarks/test_bench_tts_cli.py	Unit tests for bench_tts.py
tests/dfx/perf/tests/test_runner_metadata.py	Tests for DFX metadata key exclusion

Removed files

Directory	Reason
benchmarks/qwen3-tts/	Superseded by benchmarks/tts/ (bench_tts.py + plot_results.py cover all use cases)
benchmarks/voxcpm/	Superseded by benchmarks/tts/ (offline path → bench_voxcpm_offline.py)

Alignment with sglang-omni

Uses the same seed-tts-eval dataset as sglang-omni. Adds TTFP and SIM/UTMOS which sglang-omni does not currently track.

Test plan

• pytest tests/benchmarks/test_bench_tts_cli.py -v — 5/5 pass (locally, no torch needed)
• pytest tests/benchmarks/test_seed_tts_dataset_variants.py -v — 8/9 pass locally; 9th (test_attach_sets_seed_tts_row_even_without_extra_body) requires real vllm, passes on H20
• JSON/YAML schema valid (all configs validated with python -m json.tool and yaml.safe_load)
• python benchmarks/tts/bench_tts.py --help — outputs help text, no import errors
• Manual smoke test on H20
• DFX runner integration test on H20 with both models

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

please help remove other unnecessary benchmarks

[linyueqian]

@Sy0307 During smoke testing of the benchmark framework on H20, VoxCPM2 crashes at concurrency=4 with an orchestrator thread crash. Concurrency 1 and 2 work fine.

Root cause trace:

(Worker pid=...) INFO VoxCPM2: CUDA Graph captured for scaffold (batch_size=3)
(APIServer pid=...) INFO [Orchestrator] Shutting down all stages
(APIServer pid=...) ERROR [AsyncOmniEngine] Orchestrator thread crashed
RuntimeError: {'request_id': '...', 'error': 'Orchestrator thread crashed'}

Looks like the VAE scaffold CUDA graph capture fires lazily during concurrent inference and races with the orchestrator. Worth a look if you have context on the VoxCPM2 runtime path.

[lishunyang12]

Review: Universal TTS Benchmark

This is a well-structured consolidation of per-model benchmarks into a single, model-agnostic framework. The YAML-registry approach (model_configs.yaml) for adding new models without code changes is a good design choice, and the removal of ~2900 lines of duplicated per-model benchmark code is a clear win.

Strengths

1. Clean architecture: bench_tts.py delegates to vllm bench serve --omni with model-aware defaults from YAML, keeping the CLI thin and testable. The _TASK_TO_DATASET mapping is simple and correct.

2. Correct bug fix in patch.py: Moving seed_tts_row = True before the if not ex: return guard is the right fix. Without this, SeedTTSTextSampleRequest (which has seed_tts_speech_extra=None) would skip PCM capture entirely, breaking WER/UTMOS for default_voice and voice_design tasks. The test in test_attach_sets_seed_tts_row_even_without_extra_body verifies this via source inspection, which is pragmatic.

3. DFX integration: test_tts.json separates perf and quality eval phases, which is thoughtful. The enabled field and eval_phase metadata keys, plus updating exclude_keys in run_benchmark.py, ensure these don't leak into CLI args.

4. Good test coverage: The conftest.py vllm stubs approach is creative and lets dataset tests run without a full vllm install.

Issues and suggestions

1. SeedTTSDesignDataset uses "instructions" key but SeedTTSDesignSampleRequest docstring says "voice_description"

In seed_tts_dataset.py, the sample() method of SeedTTSDesignDataset builds:

speech_extra: dict[str, Any] = {
    "instructions": row.voice_description,
    "task_type": "VoiceDesign",
    ...
}

But SeedTTSDesignSampleRequest's docstring says the dict carries voice_description. The test also checks for "voice_description" in extra -- but extra actually contains "instructions", not "voice_description". This test line would fail:

assert "voice_description" in extra  # extra has "instructions", not "voice_description"

Wait, looking more carefully the test builds the dataset and calls .sample(), so the extra dict will have "instructions" key. The assertion assert "voice_description" in extra should indeed fail. This looks like a real bug in the test.

Action needed: Either the key in speech_extra should be "voice_description" (matching the docstring and test), or the test assertion should check for "instructions" (matching the actual code). Please verify which key the Qwen3-TTS VoiceDesign endpoint expects and align all three (code, docstring, test).

2. Duplicate VoxCPM2 stage config

benchmarks/tts/stage_configs/voxcpm2.yaml and tests/dfx/perf/stage_configs/voxcpm2.yaml are identical. Consider having the DFX config reference the benchmark one (or vice versa) to avoid drift. If they must be separate (e.g., different GPU memory settings for CI vs benchmarking), add a comment explaining why.

3. bench_voxcpm_offline.py REPO_ROOT change looks correct but is fragile

The file was moved from benchmarks/voxcpm/vllm_omni/ (3 levels deep) to benchmarks/tts/ (2 levels deep), so parents[3] -> parents[2] is correct. But this kind of relative-path depth counting is brittle. Consider using a marker file lookup or deriving from git rev-parse --show-toplevel for robustness.

4. plot_results.py -- mean_e2el metric key may be wrong

The comparison table uses "mean_e2el" as the key for E2E latency:

("E2E (ms)", "mean_e2el", ".1f"),

But bench_tts.py's summary table references "mean_audio_rtf" and "mean_audio_ttfp_ms". Please verify "mean_e2el" is the correct key emitted by vllm bench serve --omni result JSON. If it should be "mean_e2el_ms" (with _ms suffix), the E2E column would silently show nan.

5. Minor: _SeedTTSDesignRow uses __import__("random") inline

In SeedTTSDesignDataset.load_data():

rng = __import__("random").Random(self.random_seed)

This works but is unusual. A normal import random at the top of the file would be cleaner and more readable.

6. voice_design entries in test_tts.json are missing extra_body for task_type

The voice_clone entries don't need extra_body because the dataset itself provides ref_audio/ref_text. The default_voice entries correctly include extra_body: {"voice": "Vivian", ...}. But the voice_design perf entry has no extra_body -- is the task_type: "VoiceDesign" already handled by the dataset class? If yes, this is fine. If the server needs task_type in the request body for routing, this could cause 400 errors at runtime. Worth a note or confirming in the test plan.

Overall

The framework design is solid and the consolidation is clearly needed. The main concern is the "instructions" vs "voice_description" key mismatch (point 1) which appears to be a real test bug. The rest are minor improvements. After addressing point 1, this should be good to go.

[linyueqian]

@lishunyang12 @hsliuustc0106 — addressed the review with some extra H20-sourced work. Push tip is e42f3ad9.

Review items

#	Item	Action
P1	instructions vs voice_description drift	The fix(bench/tts): use 'instructions' commit updated the code but left the docstring + test still asserting voice_description, so the unit test failed on H20. Aligned docstring + test on instructions.
P2	Duplicate voxcpm2.yaml / qwen3_tts.yaml under benchmarks/tts/stage_configs/	Removed the dir; bench_tts.py default now points at tests/dfx/perf/stage_configs/ so DFX nightly and the CLI share a single source of truth.
P3	Fragile parents[2] in bench_voxcpm_offline.py	Replaced with a _find_repo_root() walker that looks for the pyproject.toml + vllm_omni/ marker.
P4	mean_e2el silent-NaN in plot_results.py	Verified with H20 result JSONs: key is mean_e2el_ms. Updated both call sites.
P5	Inline __import__("random")	Replaced with the existing module-level import random.
P6	extra_body.task_type missing on voice_design DFX entry	Empirically verified on H20 — SeedTTSDesignDataset.sample() injects task_type=VoiceDesign into each request's speech_extra, so DFX works without extra_body. Added extra_body defensively anyway to match the default_voice style.
hsliuustc0106	"remove other unnecessary benchmarks"	The per-model TTS dirs (qwen3-tts/, voxcpm/) were already removed. Leaving benchmarks/qwen3-omni/ and benchmarks/fish-speech/ alone — they're separate model families, not TTS dupes. Can delete in a follow-up if you confirm which ones you meant.

Extras surfaced by H20 benchmarking

Ran voice_clone / voice_design concurrency sweeps on H20-3e 141GB and found:

Live-traffic bug: Qwen3-TTS-12Hz-*-CustomVoice checkpoints don't ship speaker_encoder weights, so voice_clone requests crash with ValueError: This checkpoint does not provide speaker_encoder weights. The original PR's model_configs.yaml listed voice_clone under CustomVoice and the DFX test_qwen3_tts entry used CustomVoice for voice_clone — both would fail in the nightly.

Fix:

• Removed voice_clone from 1.7B-CustomVoice.supported_tasks.
• Split the DFX matrix into test_qwen3_tts_base (voice_clone via 1.7B-Base) and test_qwen3_tts_customvoice (default_voice + voice_design via 1.7B-CustomVoice).

Concurrency-cliff regression coverage: the old perf phase only ran max_concurrency [1, 4], which is below the TTFP cliff that vllm-omni's codec-bs=1 exposes. Observed H20 numbers:

Task	Model	c=1	c=4	c=8	c=16	c=32
voice_clone	1.7B-Base	RTF 0.15 / TTFP 165ms	0.28 / 412ms	0.49 / 1701ms	0.72 / 3355ms	0.77 / 3772ms
voice_design	1.7B-CustomVoice	0.08 / 53ms	0.11 / 154ms	0.21 / 872ms	0.33 / 1801ms	0.38 / 1989ms

Both show a clean 4-6× TTFP jump from c=4 to c=8 and throughput saturating at c=4-8, which is the same pattern the NVIDIA-fork comparison doc flags.

Replaced the single perf phase with:

• latency (c=1) — tight TTFP + RTF
• throughput (c=8) — loose TTFP ceiling + throughput/RTF floor, so a codec-batching regression breaks CI
• quality (c=4) — unchanged, WER/SIM/UTMOS

Thresholds padded ~2× from the H20 means. default_voice thresholds currently reuse the voice_design numbers (same checkpoint, same pipeline) — happy to run a dedicated default_voice sweep if you'd prefer.

Additional polish

• bench_tts.py now persists _task / _concurrency into the saved result JSON so plot_results.py can build per-concurrency comparison tables (was in-memory only).
• Rebased onto upstream/main and resolved the test_tts.json add/add and qwen3-tts/vllm_omni/configs/*.yaml modify/delete conflicts.

[linyueqian]

Rebased on top of #2383 (merged) + migrated to the Pipeline + Deploy schema (2ddf7658).

Post-#2383 adaptations

Change	Reason
Drop stage_config_name: qwen3_tts.yaml from test_qwen3_tts_base and test_qwen3_tts_customvoice in test_tts.json	#2383 deleted tests/dfx/perf/stage_configs/qwen3_tts.yaml. Server now auto-loads vllm_omni/deploy/qwen3_tts.yaml by model.
Drop stage_config: field from every entry in model_configs.yaml	Unused after #2383 — Pipeline registry handles model → stage mapping.
Remove --stage-configs-dir flag and _DEFAULT_STAGE_CONFIGS_DIR constant from bench_tts.py	Dead flag after #2383 (never referenced in build_bench_args).
Delete tests/dfx/perf/stage_configs/voxcpm2.yaml	Dir removed by #2383; not restored because VoxCPM2 is not yet migrated to the new schema.
Drop test_voxcpm2 entry from test_tts.json	VoxCPM2 still uses legacy ModelPipeline / _parse_pipeline_yaml; no vllm_omni/deploy/voxcpm2.yaml exists. Will re-add as a follow-up once VoxCPM2 gets its deploy YAML.

H20 smoke after migration

Booted Qwen/Qwen3-TTS-12Hz-1.7B-Base on H20-3e with no --stage-configs-path / --deploy-config flags and fired bench_tts.py --task voice_clone --concurrency 1:

voice_clone  concurrency=1   RTF 0.173   TTFP 144ms   throughput 5.78 audio-s/wall-s

• Auto-discovery picked up vllm_omni/deploy/qwen3_tts.yaml as expected.
• Numbers are in the same ballpark as the pre-migration H20 sweep (RTF 0.153 / TTFP 165ms at c=1), well within the latency baselines (RTF <= 0.25, TTFP <= 350ms). The new deploy YAML's stage-0 max_num_seqs=10 + async_scheduling=true should only help throughput at higher concurrencies.

Baselines unchanged — full sweep recalibration can land in a follow-up if DFX nightly flags drift.

CI after push: pre-commit / DCO / build(3.11) / build(3.12) all green.

[linyueqian]

@yenuo26 @amy-why-3459 @congw729 ptal, thanks!

[hsliuustc0106]

is there a readme.md under tts/

[linyueqian]

Added benchmarks/tts/README.md in 798cea7e. One-page reference covering quick-start recipes (smoke / concurrency sweep / --wer-eval), the three task types and which checkpoints support each (incl. the -CustomVoice no-speaker_encoder gotcha), how to register a new TTS model via model_configs.yaml, the bundled-vs-external dataset matrix, DFX nightly wiring (latency / throughput / quality regimes), and an H20 concurrency-cliff reference table. Links to #2558 and #2383 for context.

[Sy0307]

The new throughput regime here does not actually assert throughput. In tests/dfx/perf/tests/test_tts.json, the throughput entries only gate median_audio_ttfp_ms and median_audio_rtf, but there is no baseline for audio_throughput itself. That means this regime is currently acting more like a high-concurrency latency/RTF check than a real throughput guard. If the intent is to catch codec batching regressions or throughput collapse under load, it would be better to add an explicit audio_throughput baseline as well.

Other contents LGTM.

[hsliuustc0106]

using this benchmark locally in one L20 48GB device:

[hsliuustc0106]

lgtm

[amy-why-3459]

Can we provide the vllm bench command in the README file? This would make it easier for users to modify the vllm bench parameters.

[linyueqian]

Added in eb30f559. New section "Raw vllm bench serve commands" with full copy-paste invocations for each of the three task types (voice_clone, default_voice, voice_design) showing the full flag set — --host, --port, --model, --dataset-name, --dataset-path, --seed-tts-locale, --num-prompts, --num-warmups, --extra-body, --max-concurrency, --request-rate, --percentile-metrics, --save-result. Plus a short note on appending --seed-tts-wer-eval for WER/SIM/UTMOS. Users can now tweak bench flags without reading through bench_tts.py.

[hsliuustc0106]

please also paste the benchmark summaries ttfp/ttft/tpot/rtf standard outputs in test results

[hsliuustc0106]

I think we can move this section to the top as the first option for users

[linyueqian]

H20 benchmark summaries (per review feedback)

All runs on H20-3e 141GB, num_prompts=20, num_warmups=2, dataset seed-tts / seed-tts-design. PR tip 969aead0. The vllm bench serve --omni stdout blocks (TTFT / TTFP / TPOT / RTF / throughput / latency) are pasted below so reviewers do not have to re-run the sweep locally.

voice_clone — Qwen/Qwen3-TTS-12Hz-1.7B-Base, seed-tts

concurrency = 1

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  19.92
Request throughput (req/s):              1.00
----------------End-to-end Latency----------------
Mean E2EL (ms):                          995.69
Median E2EL (ms):                        902.93
P99 E2EL (ms):                           1577.83
---------------Time to First Token----------------
Mean TTFT (ms):                          164.80
Median TTFT (ms):                        162.55
P99 TTFT (ms):                           209.90
================== Audio Result ==================
Total audio duration generated(s):       135.04
Audio throughput(audio duration/s):      6.78
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.15
Median AUDIO_RTF:                        0.16
P99 AUDIO_RTF:                           0.18
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    164.80
Median AUDIO_TTFP (ms):                  162.55
P99 AUDIO_TTFP (ms):                     209.90
==================================================

concurrency = 4

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             4
Benchmark duration (s):                  9.23
Request throughput (req/s):              2.17
----------------End-to-end Latency----------------
Mean E2EL (ms):                          1785.28
Median E2EL (ms):                        1563.50
P99 E2EL (ms):                           2748.23
---------------Time to First Token----------------
Mean TTFT (ms):                          412.33
Median TTFT (ms):                        401.16
P99 TTFT (ms):                           581.77
================== Audio Result ==================
Audio throughput(audio duration/s):      14.37
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.28
Median AUDIO_RTF:                        0.29
P99 AUDIO_RTF:                           0.32
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    412.33
Median AUDIO_TTFP (ms):                  401.16
P99 AUDIO_TTFP (ms):                     581.77
==================================================

concurrency = 8 (cliff)

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  9.00
Request throughput (req/s):              2.22
----------------End-to-end Latency----------------
Mean E2EL (ms):                          3071.54
Median E2EL (ms):                        3471.58
P99 E2EL (ms):                           4366.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1701.14
Median TTFT (ms):                        1805.45
P99 TTFT (ms):                           2911.19
================== Audio Result ==================
Audio throughput(audio duration/s):      14.99
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.49
Median AUDIO_RTF:                        0.44
P99 AUDIO_RTF:                           0.87
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    1701.14
Median AUDIO_TTFP (ms):                  1805.45
P99 AUDIO_TTFP (ms):                     2911.19
==================================================

concurrency = 32

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             32
Benchmark duration (s):                  8.92
Request throughput (req/s):              2.24
----------------End-to-end Latency----------------
Mean E2EL (ms):                          5929.22
Median E2EL (ms):                        5932.68
P99 E2EL (ms):                           8914.57
---------------Time to First Token----------------
Mean TTFT (ms):                          3772.46
Median TTFT (ms):                        3843.58
P99 TTFT (ms):                           7199.71
================== Audio Result ==================
Audio throughput(audio duration/s):      14.58
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.77
Median AUDIO_RTF:                        0.73
P99 AUDIO_RTF:                           1.34
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    3772.46
Median AUDIO_TTFP (ms):                  3843.58
P99 AUDIO_TTFP (ms):                     7199.71
==================================================

voice_design — Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice, seed-tts-design

concurrency = 1

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  13.77
Request throughput (req/s):              1.45
----------------End-to-end Latency----------------
Mean E2EL (ms):                          688.20
Median E2EL (ms):                        688.65
P99 E2EL (ms):                           884.63
---------------Time to First Token----------------
Mean TTFT (ms):                          53.27
Median TTFT (ms):                        52.89
P99 TTFT (ms):                           59.12
================== Audio Result ==================
Audio throughput(audio duration/s):      12.28
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.083
Median AUDIO_RTF:                        0.081
P99 AUDIO_RTF:                           0.112
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    53.27
Median AUDIO_TTFP (ms):                  52.89
P99 AUDIO_TTFP (ms):                     59.12
==================================================

concurrency = 4

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             4
Benchmark duration (s):                  5.13
Request throughput (req/s):              3.90
----------------End-to-end Latency----------------
Mean E2EL (ms):                          943.80
Median E2EL (ms):                        919.07
P99 E2EL (ms):                           1456.01
---------------Time to First Token----------------
Mean TTFT (ms):                          154.31
Median TTFT (ms):                        150.22
P99 TTFT (ms):                           228.11
================== Audio Result ==================
Audio throughput(audio duration/s):      33.04
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.113
Median AUDIO_RTF:                        0.110
P99 AUDIO_RTF:                           0.155
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    154.31
Median AUDIO_TTFP (ms):                  150.22
P99 AUDIO_TTFP (ms):                     228.11
==================================================

concurrency = 8 (cliff)

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             8
Benchmark duration (s):                  4.80
Request throughput (req/s):              4.17
----------------End-to-end Latency----------------
Mean E2EL (ms):                          1640.43
Median E2EL (ms):                        1797.72
P99 E2EL (ms):                           2333.79
---------------Time to First Token----------------
Mean TTFT (ms):                          872.26
Median TTFT (ms):                        843.18
P99 TTFT (ms):                           1205.98
================== Audio Result ==================
Audio throughput(audio duration/s):      33.99
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.210
Median AUDIO_RTF:                        0.188
P99 AUDIO_RTF:                           0.283
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    872.26
Median AUDIO_TTFP (ms):                  843.18
P99 AUDIO_TTFP (ms):                     1205.98
==================================================

concurrency = 32

============ Serving Benchmark Result ============
Successful requests:                     20
Failed requests:                         0
Maximum request concurrency:             32
Benchmark duration (s):                  4.66
Request throughput (req/s):              4.29
----------------End-to-end Latency----------------
Mean E2EL (ms):                          2732.84
Median E2EL (ms):                        2754.59
P99 E2EL (ms):                           4639.83
---------------Time to First Token----------------
Mean TTFT (ms):                          1989.31
Median TTFT (ms):                        2009.91
P99 TTFT (ms):                           4053.75
================== Audio Result ==================
Audio throughput(audio duration/s):      33.55
-----------------Real Time Factor-----------------
Mean AUDIO_RTF:                          0.378
Median AUDIO_RTF:                        0.330
P99 AUDIO_RTF:                           0.809
---------------Time to First Packet---------------
Mean AUDIO_TTFP (ms):                    1989.31
Median AUDIO_TTFP (ms):                  2009.91
P99 AUDIO_TTFP (ms):                     1989.31
==================================================

Notes

• TPOT / ITL are reported as 0.0 for both tasks because Qwen3-TTS is non-streaming at the token level (codec streaming is chunk-based); the serving bench only computes TPOT/ITL when token-by-token streaming is enabled.
• The codec-bs=1 cliff is clean from c=4 to c=8 in both tasks: voice_clone TTFP jumps 412ms → 1701ms (4.1×), voice_design TTFP jumps 154ms → 872ms (5.7×). Audio throughput saturates around c=4-8 in both cases.

[linyueqian]

@hsliuustc0106 is this OK to merge?

[amy-why-3459]

Could you add a nightly-test for the benchmark?

[yenuo26]

please add mark

[linyueqian]

Addressed in 8f06be05. Added module-level pytestmark = [pytest.mark.core_model, pytest.mark.cpu] so the file is picked up by the nightly core_model and cpu selector.

[yenuo26]

please don't use unittest.mock, pytest mock instead

[linyueqian]

Done in 8f06be05. Dropped from unittest.mock import patch; test_unsupported_task_exits now takes mocker as a fixture and calls mocker.patch.object(sys, "argv", [...]) instead of the with patch.object(...) context manager.

[yenuo26]

please add mark, like pytest.mark.core_model, pytest.mark.cpu

[linyueqian]

Done in 8f06be05. Added module-level pytestmark = [pytest.mark.core_model, pytest.mark.cpu] to tests/benchmarks/test_bench_tts_cli.py.

[yenuo26]

please add mark, like pytest.mark.core_model, pytest.mark.cpu

[linyueqian]

Done in 8f06be05. Added module-level pytestmark = [pytest.mark.core_model, pytest.mark.cpu] to tests/benchmarks/test_seed_tts_dataset_variants.py.

[yenuo26]

please don't use unittest.mock, pytest mock instead

[linyueqian]

Done in 8f06be05. Dropped from unittest.mock import MagicMock; the three dataset-class tests (test_seed_tts_text_dataset_omits_ref_audio, test_seed_tts_design_dataset_has_instructions, test_seed_tts_design_dataset_rejects_missing_description) now take mocker as a fixture and use mocker.MagicMock().

[yenuo26]

Could you add a nightly-test for the benchmark?

I think tts-test and omni-test are enough.

[amy-why-3459]

Could you add an L4 test case?

[lishunyang12]

@hsliuustc0106 PTAL

[hsliuustc0106]

lgtm, update the qwen3-tts recipe later, thank you