Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 24 additions & 44 deletions docs/configuration/stage_configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -626,52 +626,32 @@ the intended GPU, and warm the complete pipeline before measuring performance.

## MOSS-TTS Local 1.5 with Model Runner V2

`OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5` defaults to CUDA MRV2 using
the shared runtime introduced for Qwen3-TTS. To select the C64 profile without
MPS explicitly:
`OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5` uses one default deployment,
`moss_tts_local.yaml`. Omitting `--deploy-config` selects the same file:

```bash
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 --omni \
--deploy-config vllm_omni/deploy/moss_tts_local_mrv2.yaml
--deploy-config vllm_omni/deploy/moss_tts_local.yaml
```

This profile inherits the batching, codec graph buckets and 1-frame/15-frame
chunk geometry from `moss_tts_local_v1.yaml`, and selects V2 for both the Local
Talker and codec stages. The Local depth predictor exposes the MRV2 `mtp`
capabilities while retaining its V1 `talker_mtp` implementation, sampling
defaults and explicit request-seed handling. Codec chunks use the native data
plane; the internal Talker retains its final-only orchestrator output policy.

On CUDA, the profile bounds stage-0 prefill work to 512 tokens per iteration.
The codec retains its own CUDA Graph capture/replay, including batch buckets
through 64, but disables Inductor compilation (`compilation_config.mode: 0`)
of the stateful decoder to avoid lengthy compilation at startup. This is
independent of `enforce_eager`; the profile keeps `enforce_eager: false`.
It explicitly selects `cudagraph_mode: FULL`, which works without Inductor;
the default `FULL_AND_PIECEWISE` would otherwise normalize to `NONE` and clear
the codec's capture buckets when compilation is disabled.

For sustained C128 serving on a large-memory CUDA GPU, select the
separate `moss_tts_local_mrv2_high_concurrency.yaml` profile. It uses 128
stream slots per stage, a 32 GiB Talker KV budget, and the codec's
`triton_slot` backend with Inductor compilation and codec-owned CUDA graphs.
The bounded KV budget leaves room for codec state and graphs; it does not
guarantee that 128 maximum-length prompts fit simultaneously. See the
[MOSS recipe](gh-file:recipes/OpenMOSS/MOSS-TTS.md#local-15-mrv2-and-slot-attention)
for activation, backend comparisons, memory requirements and benchmark commands.

Omitting `--deploy-config` selects the C128 system profile on CUDA GPUs with
at least 140 GiB memory and MPS available. Smaller GPUs or a failed memory
query use C64 with utilization-based memory budgets; missing MPS selects
C64 without MPS. Selecting `moss_tts_local.yaml` explicitly uses C64 with MPS.
Use `moss_tts_local_v1.yaml` for the previous V1 deployment.
NPU, XPU, ROCm and MUSA overrides retain V1. These profiles do not enable
MRV2 for MOSS Delay, Realtime or Nano. Local 1.5 outputs 48 kHz stereo audio;
set `VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000` and
`VLLM_OMNI_BENCH_AUDIO_CHANNELS=2` when benchmarking raw PCM.

Event-driven orchestration remains independently selectable with
`VLLM_OMNI_EVENT_DRIVEN_ORCH=0` or `1`. Keep the runner and deployment identical
when comparing these modes. Model-runner selection does not change the
orchestration default. MPS, codec dispatch and graph settings belong to the
deployment profile; the automatic CUDA default selects MPS when available.
On CUDA, both stages use MRV2 and 128 stream slots on one GPU, with full-quota
private MPS, a 32 GiB Talker KV budget, prefix caching and Triton attention.
The Talker bounds prefill to 512 tokens and uses FULL CUDA graphs. The codec
uses `triton_slot`, Inductor compilation, codec-owned graphs through batch
128 and dispatch coalescing at 16 rows or 6 ms. Initial and steady chunks
remain 1 and 15 codec frames; the Stage0 first-frame path follows the Talker
capture sizes and safe-request/executor requirements.

These CUDA settings are validated on H200 and require memory for both stages
and their graph/state pools, plus `nvidia-cuda-mps-control` on `PATH`. Smaller
GPUs and services without MPS should provide deployment overrides for the
capacities, capture sizes, KV budget and `platforms.cuda.cuda_mps`. The
pipeline no longer changes configuration based on a GPU-memory or MPS probe.

Other platforms use V1 in the same file; NPU keeps its codec graph override.
The 2/3-NPU files retain their distinct multi-device topology. These changes
do not enable MRV2 for MOSS Delay, Realtime or Nano. Local 1.5 emits 48 kHz
stereo; benchmark raw PCM with `VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=48000` and
`VLLM_OMNI_BENCH_AUDIO_CHANNELS=2`. See the
[MOSS recipe](gh-file:recipes/OpenMOSS/MOSS-TTS.md#local-15-unified-deployment-and-slot-attention)
for the execution path and configuration overrides.
28 changes: 14 additions & 14 deletions examples/online_serving/text_to_speech/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -569,15 +569,15 @@ python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.

## MOSS-TTS Local Transformer v1.5

For a single H200, the optional
[`moss_tts_local_h200.yaml`](../../../vllm_omni/deploy/moss_tts_local_h200.yaml)
For a single H200, the default
[`moss_tts_local.yaml`](../../../vllm_omni/deploy/moss_tts_local.yaml)
deployment places both the talker and codec on logical GPU 0:

```bash
CUDA_VISIBLE_DEVICES=0 VLLM_OMNI_EVENT_DRIVEN_ORCH=1 \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
--omni --trust-remote-code \
--deploy-config vllm_omni/deploy/moss_tts_local_h200.yaml \
--deploy-config vllm_omni/deploy/moss_tts_local.yaml \
--disable-log-stats
```

Expand All @@ -586,18 +586,18 @@ Run from the repository root and select an available physical GPU with
disables detailed per-request statistics logging to reduce CPU overhead at
high concurrency. Remove `--disable-log-stats` when those statistics are needed;
keep these settings identical when comparing performance.
This preset configures 256 request slots per stage,
This preset configures 128 request slots per stage,
a 32 GiB talker KV cache, talker CUDA Graph buckets through 512 scheduled tokens,
and codec batch buckets through 256. It requires more than 80 GiB of GPU memory;
request capacity also depends on input and generated lengths. The default
[`moss_tts_local.yaml`](../../../vllm_omni/deploy/moss_tts_local.yaml) remains
available for smaller deployments.

The preset enables the optional codec backend with
`hf_overrides.codec_attention_backend: triton` on stage 1. It preserves the
streaming ring-cache mask and uses BF16 attention with 64-dimensional heads;
other attention shapes use PyTorch SDPA. Set the backend to `sdpa` to use the
default attention implementation. Codec terminal tails share execution only
and codec batch buckets through 128. It requires room for both stages and
their graph/state pools, plus `nvidia-cuda-mps-control` on `PATH`.
For smaller deployments, override the capacities, graph buckets, KV budget
and MPS setting in a custom deploy file.

The preset enables the codec backend with
`hf_overrides.codec_attention_backend: triton_slot` on stage 1, accessing
request-slot ring state directly with BF16 attention. Set the backend to
`sdpa` in a custom deploy to compare with the PyTorch attention path.
Codec terminal tails share execution only
when they already map to the same padded CUDA Graph; returned audio retains
each request's actual length.

Expand Down
Loading
Loading