[gcanlin]

Purpose

Consolidate the seven redundant single-GPU MOSS-TTS Local 1.5 presets into standalone moss_tts_local.yaml, using the former optimized settings. Remove GPU-memory/MPS probing from pipeline dispatch and register the static pipeline directly.

CUDA uses MRV2, full-quota private MPS, 128 slots per stage, a 32 GiB Talker KV budget, prefix caching and dense codec graphs. Resolved settings match the former optimized profile on CUDA/CPU/NPU/XPU/ROCm/MUSA. The distinct 2/3-NPU topologies remain. Smaller CUDA GPUs or deployments without MPS must supply overrides; automatic capacity fallback is removed. Documentation and configuration tests follow the single default.

Test Plan

One H200 (143,771 MiB), GPU6; BF16, no quantization; full SeedTTS EN1088 voice cloning, 48 kHz stereo PCM, output length 256. One API process; server CPUs 48–79, client CPUs 80–95. Prewarm the 666 unique reference files, exclude the first complete C64 warmup round, and measure two full rounds at each concurrency. These are warm repeated-prompt results with prefix caching enabled where configured; TTFP is the client-side mean, not a tail-latency guarantee. MPS is enabled for all three preset comparisons.

vLLM Version: 0.31.0; Torch 2.13.0+cu130; CUDA 13.0; Transformers 5.14.1.
vLLM-Omni Commit: main b65107d61331327424910c72e7a2da137dcfb3df; this PR changes configuration/dispatch only.

CUDA_VISIBLE_DEVICES=6 OMP_NUM_THREADS=1 VLLM_WORKER_MULTIPROC_METHOD=spawn \
VLLM_OMNI_MOSS_REF_GRAPHS=0 taskset -c 48-79 vllm serve \
  OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
  --host 127.0.0.1 --port 8124 --trust-remote-code \
  --allowed-local-media-path /root/vllm-omni-workspace --api-server-count 1 \
  --init-timeout 3600 --stage-init-timeout 3600

# Prewarm the reference files; then the first complete C64 pass warms the prompt set.
VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000 VLLM_OMNI_BENCH_AUDIO_CHANNELS=2 \
taskset -c 80-95 python benchmarks/tts/bench_tts.py \
  --host 127.0.0.1 --port 8124 \
  --model OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
  --task voice_clone --locale en \
  --dataset-path /root/vllm-omni-workspace/seedtts_testset \
  --num-prompts 1088 --concurrency 64 64 64 128 128 --output-len 256 \
  --output-dir results/moss-local-unified -- --seed-tts-file-ref-audio

Test Result

Pooled throughput = total generated audio duration / total benchmark time. Parentheses show the two-round range. Select highest throughput subject to mean TTFP ≤130 ms.

Configuration	C64 audio-s/s	C64 TTFP ms	C128 audio-s/s	C128 TTFP ms
Former high-concurrency + MPS	363.8 (359.3–368.4)	73.8 (73.8–73.8)	435.0 (420.0–451.0)	147.3 (145.9–148.6)
Former low-latency + MPS	372.6 (371.7–373.5)	77.0 (76.0–78.0)	455.5 (455.1–455.9)	139.6 (137.5–141.6)
Main: former optimized	443.7 (441.4–446.0)	59.6 (58.3–60.8)	554.9 (552.7–557.1)	123.9 (122.7–125.2)
PR: implicit default dispatch	444.9 (443.3–446.5)	60.2 (59.9–60.6)	548.3 (538.0–559.3)	126.5 (123.5–129.5)

The optimized settings win at both concurrencies. This is a configuration cleanup; the PR does not add a new runtime speedup.

• All timed requests succeeded.
• Actual serving without --deploy-config: full benchmark plus first-frame length/order, stream/nonstream, cancellation and slot-recycling checks.
• 547 CPU tests passed (15 warnings, 17.09 s); full local pre-commit and diff checks passed.
• Six-platform resolved-config comparison is identical to the former optimized profile.
• Sampled peak device memory is approximately 87 GiB after excluding the 1.5 GiB benchmark reservation; includes loading and warmup.

AI assistance: Codex assisted with the configuration refactor, tests, benchmarking and PR description.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/vllm_omni_config.md.

Module owners: @lishunyang12 @alex-jw-brooks @hsliuustc0106

Routing: @lishunyang12 via module of the changed files, semantic router, CODEOWNERS; @alex-jw-brooks via module of the changed files, semantic router; @hsliuustc0106 via CODEOWNERS

@gcanlin, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

[Sy0307]

LGTM.

[gcanlin]

baseline:

vLLM-Omni 当前最佳性能

vLLM: v0.26.0

vLLM-Omni: https://github\.com/gcanlin/vllm\-omni/tree/encoder\-gpu

# Serving
VLLM_OMNI_MOSS_REF_ENCODER_DEVICE=cuda:7 vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
  --omni \
  --host 0.0.0.0 \
  --port 8123   \
  --trust-remote-code \
  --deploy-config vllm_omni/deploy/moss_tts_local.yaml \
  --allowed-local-media-path /root/vllm-omni-workspace

# Benchmark
SEED_TTS_EVAL_DEVICE=cuda:1 \
VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000 \
VLLM_OMNI_BENCH_AUDIO_CHANNELS=2 \
python benchmarks/tts/bench_tts.py \
--host 127.0.0.1 \
--port 8123 \
--model OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
--task voice_clone \
--locale en \
--dataset-path /root/vllm-omni-workspace/seed-tts-eval \
--num-prompts 1088 \
--concurrency 1 8 16 32 64 \
--output-len 256 \
--wer-eval \
--output-dir /root/vllm-omni-workspace/vllm-omni/results/

Model	Hardware	TFLOPS	显存	dataset 分布	部署方式	并发	WER	Mean RTF	P99 RTF	Request Throughput	Audio Throughput	Mean Latency	P99 Latency	TTFP（首包延迟）	P99 TTFP	显存峰值
MOSS-TTS-Local-v1.5 Streaming	NVIDIA H200 SXM	1,979（BF16 Tensor Core，稀疏）	141 GB HBM3e	SeedTTS EN full（1,088 requests/run）	1 GPU	1	3.14%	0.13	0.14	1.86 req/s	7.86 audio-s/s	538 ms	912 ms	56 ms	68 ms	~111257 MiB
						8	3.14%	0.22	0.26	8.43 req/s	36.40 audio-s/s	947 ms	1,776 ms	95 ms	137 ms
						16	3.06%	0.30	0.35	12.37 req/s	52.84 audio-s/s	1,285 ms	2,418 ms	127 ms	212 ms
						32	4.25%	0.46	0.53	16.41 req/s	70.22 audio-s/s	1,928 ms	3,644 ms	177 ms	323 ms
						64	4.28%	0.68	0.77	21.88 req/s	93.02 audio-s/s	2,862 ms	5,315 ms	277 ms	634 ms
					2 GPUs	1	3.14%	0.13	0.14	1.86 req/s	8.08 audio-s/s	538 ms	1,048 ms	51 ms	64 ms	GPU 0 : 83,661 MiB; GPU 1：30,539 MiB
						8	3.14%	0.16	0.19	11.48 req/s	49.14 audio-s/s	693 ms	1,200 ms	74 ms	106 ms
						16	3.06%	0.20	0.23	18.63 req/s	80.24 audio-s/s	852 ms	1,581 ms	95 ms	173 ms
						32	4.25%	0.27	0.32	27.73 req/s	118.14 audio-s/s	1,117 ms	1,980 ms	132 ms	292 ms
						64	4.28%	0.38	0.49	36.98 req/s	157.24 audio-s/s	1,603 ms	2,893 ms	197 ms	533 ms
	NVIDIA A800 80GB SXM	312（BF16/FP16 Tensor Core，稠密）	80 GB HBM2e	SeedTTS EN full（1,088 requests/run)	1 GPU	1	3.62%	0.21	0.24	1.11 req/s	4.83 audio-s/s	901 ms	1,822 ms	109 ms	124 ms	~73,000MiB
						8	4.23%	0.35	0.42	5.25 req/s	22.72 audio-s/s	1,514 ms	2,734 ms	161 ms	236 ms
						16	2.83%	0.48	0.57	7.68 req/s	33.13 audio-s/s	2,067 ms	3,720 ms	217 ms	444 ms
						32	3.71%	0.74	0.86	10.05 req/s	43.29 audio-s/s	3,147 ms	5,963 ms	300 ms	505 ms
						64	3.08%	1.15	1.34	12.89 req/s	55.04 audio-s/s	4,860 ms	8,797 ms	472 ms	1,049 ms
					2 GPUs	1	2.87%	0.20	0.23	1.19 req/s	5.09 audio-s/s	840 ms	1,495 ms	101 ms	116 ms	GPU 0 : 45,853 MiB; GPU 1：29,761 MiB
						8	3.79%	0.25	0.30	7.34 req/s	32.11 audio-s/s	1,084 ms	2,420 ms	126 ms	178 ms
						16	3.06%	0.30	0.36	12.29 req/s	53.27 audio-s/s	1,292 ms	2,365 ms	163 ms	317 ms
						32	3.71%	0.39	0.48	18.44 req/s	79.97 audio-s/s	1,678 ms	2,998 ms	228 ms	518 ms
						64	3.12%	0.58	0.76	23.96 req/s	103.61 audio-s/s	2,478 ms	4,279 ms	362 ms	864 ms