[Sy0307]

Purpose

Optimize CosyVoice3 packed streaming with tighter packed-attention query launch bounds, cached ISTFT, and bounded CUDA graphs for HiFT's deterministic waveform decoder. This PR is rebased onto merged #8422 (e372aa3): the shared MRV2 decode-bound and multimodal-padding fixes, their CPU/FA3 regressions, and the single Buildkite FA3 step are inherited unchanged from main.

For the packed codec, use the actual query length for the attention launch grid and graph only deterministic HiFT decode. F0, noise generation, phase carry and request state remain outside capture. The cache holds at most 32 exact-shape graphs and 128 shape counters, falls back to eager execution at capacity, and returns an owned waveform so subsequent requests cannot overwrite it. ISTFT reuses the existing cached overlap envelope implementation.

The new cosyvoice3_packed_streaming_optimized_standard.yaml inherits the existing streaming profile, changes AR/codec admission capacities from 64/16 to 32/32, and enables the two codec optimizations. It requires standard sampling and streaming requests. Flow remains at 10 steps/BF16; HiFT and F0 remain FP32. Both new environment switches default off. The measured throughput results apply to this opt-in profile.

HiFT graph capture remains lazy: the third use of an exact shape pays for an additional eager warmup and capture. The YAML now documents this latency and the graph pool's memory headroom requirement; gpu_memory_utilization does not impose a hard cap on this codec graph pool. Moving common-shape capture before readiness is deferred to a separately validated latency change.

Inherited MRV2 behavior is unchanged: with an unspecified capture-time query bound, FULL_AND_PIECEWISE and FULL_DECODE_ONLY retain the uniform decode bound. Pure FULL, which has no separate decode routine and also serves mixed batches, retains the token-bucket worst-case bound, including uniform dummy capture batches. Explicit bounds remain intact.

Test Plan

vLLM Version: 0.30.0

vLLM-Omni Commit: PR base e372aa3a4137bca513827b37e6080b908cd977c7 (merged #8422). Historical performance/quality measurements used 423f34326ed420e5acf0b1fb862a1b5ffb7e0fa7 plus the optimization patch. SGLang-Omni comparison revision: 257826d526b718851a5323b8dab72a11522fb0de.

Environment: Linux, one H200, torch 2.13.0+cu130, CUDA runtime 13.0 in a CUDA 13.2.1 container. Rebase regressions used physical GPU 6; historical generation and SGLang controls used the same physical GPU 2. WER/speaker scoring ran separately and is excluded from generation timing.

From the repository root:

python -m pytest -q \
  tests/worker_v2/test_omni_model_state.py tests/config/test_environment_variables.py \
  -m 'core_model and cpu' --run-level=core_model -o addopts=

# Select an available Hopper GPU before CUDA device remapping.
python -m pytest -q \
  tests/worker_v2/test_capture_query_bound_gpu.py tests/worker_v2/test_decode_query_bound_gpu.py \
  tests/model_executor/models/cosyvoice3/test_cosyvoice3_full_batch.py::test_packed_stream_attention_matches_chunk_causal_reference \
  tests/model_executor/models/cosyvoice3/test_cosyvoice3_flow_graph.py \
  tests/model_executor/models/cosyvoice3/test_cosyvoice3_hift_graph.py \
  -m 'core_model and cuda' --run-level=core_model -o addopts=

The inherited H100 test-ready.yml job runs both test_capture_query_bound_gpu.py and test_decode_query_bound_gpu.py; there is exactly one FA3 step, with source dependencies. These files and the shared runner/CPU regression are byte-identical to main. Existing packed attention/Flow numerical coverage is retained; HiFT output ownership and eager fallback are collected by the existing model executor CUDA job. CPU coverage includes the active embedding boundary and reviewed environment inventory.

To serve the opt-in profile with a cached checkpoint and local Seed-TTS references:

vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni   --host 127.0.0.1 --port 19700 --trust-remote-code   --deploy-config vllm_omni/deploy/cosyvoice3_packed_streaming_optimized_standard.yaml   --hf-overrides '{"cosyvoice3_sampling_mode":"standard"}'   --allowed-local-media-path "$SEED_TTS_ROOT"

An existing repository client can exercise streaming performance with the same profile and warmup/concurrency settings:

vllm bench serve --omni --host 127.0.0.1 --port 19700   --model FunAudioLLM/Fun-CosyVoice3-0.5B-2512   --backend openai-audio-speech --endpoint /v1/audio/speech   --dataset-name seed-tts --dataset-path "$SEED_TTS_ROOT" --seed-tts-locale en   --num-prompts 1088 --num-warmups 64 --max-concurrency 32 --request-rate inf   --seed 42 --extra-body '{"stream":true,"response_format":"pcm","stream_format":"audio","seed":42}'   --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun   --save-result --result-dir ./cosy-stream-c32

Repeat with concurrency 64 and a separate output directory. This is a service exercise using maintained benchmark infrastructure; the historical table below used a local comparison driver and separate Whisper/WavLM evaluators. Those experiment-only bridges and raw evidence are retained locally, not added to this PR, so the command above is not claimed to reproduce the archived measurements exactly.

Test Result

Fresh validation after rebasing onto merged #8422 (8 changed files, +178/-4; tests: 2 files, +43/-2):

• CPU: 48 passed, 14 warnings, 9.95 s.
• H200 CUDA: 8 passed, 16 warnings, 26.67 s, physical GPU 6. Both inherited FA3 files and the retained codec regressions passed.
• All applicable changed-file pre-commit hooks pass, including Ruff, mypy, YAML, environment/test marks, SPDX and forbidden imports; git diff --check passes.

Historical H200 throughput and quality

EN1088 voice cloning, all 1,088 rows per round, 64 warmup requests, C32/C64, two independent service starts for the chosen profile and SGLang. Standard sampling: temperature 0.7, top-p 0.8, top-k 20, repetition penalty 1.21, seed 42, 2,048 token cap. CPU affinity was 0-47,96-143. Each framework retained its native streaming chunks. Full-response results are excluded.

The table pools total generated audio seconds divided by total measured wall time across the two starts; QPS uses the same wall time.

Concurrency	SGL audio-s/s	PR profile audio-s/s	Gain	SGL QPS	PR profile QPS	QPS gain
C32	61.59	74.33	+20.70%	12.967	14.165	+9.24%
C64	68.52	111.12	+62.18%	14.647	21.877	+49.36%

Chosen-profile per-start ranges: C32 73.892–74.790, C64 107.358–115.325 audio-s/s. C32's second start is about 19.98% above the pooled SGL control, so this does not establish that every start exceeds 20%.

A single-start screening comparison against the correctness-repaired existing AR64/codec16 profile gives 70.035 → 74.790 at C32 and 86.338 → 115.325 at C64. That is a configuration-plus-optimization comparison, not an independent ablation or a repeated-mainline speedup claim. The unrepaired baseline has severely degraded transcription quality and is not used as a comparable-quality throughput control.

Concurrency	SGL / PR mean audio (s)	SGL / PR first PCM (s)	SGL / PR WER	SGL / PR speaker SIM ×100
C32	4.750 / 5.248	1.348 / 1.667	1.9956% / 2.1431%	70.5429 / 70.1804
C64	4.678 / 5.079	2.632 / 1.892	1.8717% / 2.1057%	70.4554 / 70.3017

C32's audio throughput gain partly reflects longer generated audio; QPS improves only 9.24%, and first PCM is slower. First-packet sizes differ between the implementations. WER is slightly higher and speaker SIM slightly lower by point estimate, so there is no zero-quality-cost claim. Paired reference-cluster bootstrap intervals include zero: WER differences +0.1474 pp [-0.2763,+0.5866] / +0.2340 pp [-0.1481,+0.6574]; SIM differences -0.3625 [-0.7638,+0.0314] / -0.1537 [-0.5315,+0.2394], for C32/C64 respectively.

Maximum sampled whole-device GPU memory (1 s NVML polling, including engine reservation and CUDA contexts): repaired existing profile 64,079 MiB, chosen profile 65,245 / 66,957 MiB, SGL 135,000 / 133,716 MiB across starts. These are sampled device maxima, not exact allocator peaks or isolated graph-cache memory costs.

The complete screening/confirmation set contains 16 formal rounds, 17,408 successful generation requests, zero generation failures, with every row scored by Whisper large-v3 plus official Seed-TTS text normalization and the original WavLM similarity checkpoint. Historical aggregate measurements have not been rerun on the newly aligned PR base. Explicit FULL uncompiled serving is also outside the validated end-to-end configuration; the PR validates the default MRV2 mode and targeted FULL attention regressions.

AI assistance: OpenAI Codex assisted with optimization analysis, implementation, regression tests, validation, and this PR description. The validation results and measurement limitations are reported above.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/model_integration.md, docs/design/module/ar_runtime.md.

Module owners: @tzhouam @gcanlin @fake0fan

Routing: @tzhouam via module of the changed files, semantic router, CODEOWNERS; @gcanlin via module of the changed files, semantic router; @fake0fan via module of the changed files

@Sy0307, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-20261001.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 118f646c8771 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot: three questions on the performance claim

@Sy0307 this PR reads as a performance or value claim:

• title: [Model] Optimize CosyVoice3 packed streaming with bounded HiFT graphs

Before the full evidence checklist, three short questions:

1. Bottleneck — what is the current bottleneck, and which profile, trace or per-stage measurement shows it?
2. Value — what does the change buy the user or the system (latency, throughput, memory, cost), and at which workload?
3. A/B or ablation — is there a same-workload, same-head/config comparison that isolates each main claim on its own? For stacked optimizations, one number per item rather than a blended delta.

When you answer, the evidence that settles it is: base and head SHA, hardware, model, workload, warm-up and repeat count, mean or percentiles with their spread, and a correctness/quality-equivalence signal; an end-to-end claim also needs stage attribution.

[vllm-omni-review-bot]

Omni ReviewBot review

Scan:

Category	Result
Tests / verification	6 finding(s) below
Security	no finding reported
Docs / comments	3 finding(s) below
Behavior / compatibility	no finding reported
Correctness	no finding reported

Validated:

• [resolved] 'Eager path breaks when forward_full stops deriving the bound from page_table width': RaggedRowAttention.init always computes max_seqlen_q (packed_dit.py:57); the object.new path in _solve sets every field the kernel path reads (heads, head_dim, cache_seqlens, cu_seqlens_q, page_table, max_seqlen_q).
• [resolved] Concern 'query_start_loc slice breaks other consumers': supports_embed_input_ids_query_start_loc is set only by cosyvoice3.py:723 (plus tests), and the v1 twin already slices (embedding_kwargs["query_start_loc"] = self.query_start_loc.cpu[: num_reqs + 1].tolist(), gpu_model_runner.py:1645) — v2 now matches the v1 contract; residual checked: no other in-repo consumer of the full padded list.
• [resolved] 'Graph replay overwrites a previous request's waveform': run() returns state.output.clone() with the aliasing comment, and the new test snapshots earlier outputs and re-asserts torch.equal after later decodes.
• [claim-verified] 'AR/codec admission 64/16 -> 32/32' — cosyvoice3_full_response.yaml:33 (stage-0 max_num_seqs 64), cosyvoice3_packed_streaming.yaml:18 (stage-1 16); new yaml sets 32/32 and stage lists merge per stage_id (stage_config.py:818 _merge_stage_lists, 'a thin overlay does not discard inherited runtime contracts'), so the inherited connector codec_chunk_frames survives.
• [claim-verified] Env counts: MODEL_SPECIFIC 75→77 and PROMOTE 46→48 match exactly the two _MODEL_PROMOTE additions (COSYVOICE3_CACHED_ISTFT, COSYVOICE3_HIFT_GRAPH), alphabetically placed.
• [claim-verified] 'cache holds at most 32 exact-shape graphs and 128 shape counters, falls back to eager at capacity, returns an owned waveform' — hift_graph.py:33 (max_graphs=32), :36-37 (dict+OrderedDict), :60-61 (capacity eager fallback), :85 (output.clone()) match the description verbatim.

Of 14 candidates, 5 survive as publishable comments and 9 are merged away or dropped. The one major asks the new HiFTDecodeGraphs to handle CUDA-graph capture failure (try/except with per-key disable, eager fallback, logging, and the sibling depth_graph.py pool/error-mode settings), since an unguarded capture exception fails live streaming requests in exactly the connector-thread environment the in-tree sibling documents. Four minors, all central to the PR: the new guard's separate_routine() check changes decode-capture bounds for every worker_v2 model with only FA3/CosyVoice3 evidence behind it (kept as a scoping question); the new deploy yaml's standard-sampling requirement is enforced only by a comment while runtime.py defaults to ras (question form); enable_decode_graphs() silently overrides an explicit COSYVOICE3_CACHED_ISTFT=0 despite the inventory promoting the vars as independent (ask is one comment/log line); and the boundary-fix regression test asserts the slice on a MagicMock rather than the real CosyVoice3.embed_input_ids path that used to raise. Drops are six self-verified no-issue/confirmation records (indices 2, 3, 4, 5, 9, 10) that request no change; dups merge the three second-site restatements (1->6, 7->0, 8->12) into their stronger survivors. The kept set engages the PR's primary changes — the MRV2 replay-bound fix, the boundary fix, and the new HiFT graph feature — rather than secondary files.

Verdict: REQUEST CHANGES

CI at 2cd21a984fa4 (2026-10-02T08:55:42.071038+00:00): all required and operator-watched checks reported green. Observed Buildkite: buildkite/vllm-omni-amd-ci (passed), buildkite/vllm-omni (passed), buildkite/vllm-omni-intel-ci (passed), and 2 more.

Findings

• **[P2] The new guard (omni_model_state.py:433) evaluates self.vllm_config.compilation…** — tests/worker_v2/test_omni_model_state.py:60The new guard (omni_model_state.py:433) evaluatesself.vllm_config.compilation_config.cudagraph_modewheneverfor_capture=Trueandmax_query_len is None, but this module's shared _make_statefactory setsvllm_config = SimpleNamespace(model_config=...)with nocompilation_config(:60). It's only latent today because no test here calls prepare_attn (repo grep: test callers are test_capture_query_bound_gpu.py:73/103/106, which build their own vllm_config; production caller omni_model_runner.py:591). Extend_make_stateto stubcompilation_config(a SimpleNamespace carrying acudagraph_mode`) so the factory can't produce states that AttributeError on the capture path, and the CPU-side guard behavior becomes drivable from this module.

Evidence: Added by this diff, vllm_omni/worker_v2/model_states/omni_model_state.py:430-434: if ( / for_capture / and input_batch.max_query_len is None / and not self.vllm_config.compilation_config.cudagraph_mode.separate_routine() / ): — the guard dereferences compilation_config. Unchanged by this diff, present in the PR-time tree, tests/worker_v2/test_omni_model_state.py:60: state.vllm_config = SimpleNamespace(model_config=SimpleNamespace(subtalker_sampling_params={})) — no compilation_config attribute. Repo-wide grep for prepare_attn at PR-time head: only test callers are tests/worker_v2/test_capture_query_bound_gpu.py:73 prepare = OmniModelState.prepare_attn, :103 and :106 (state built via OmniModelState.__new__ with the diff adding state.vllm_config = c), plus production vllm_omni/worker_v2/omni_model_runner.py:591 attn_metadata = self.model_state.prepare_attn(; zero matches in test_omni_model_state.py.

• [P2] At capacity the fallback is permanent for every new shape: the check at hift_gr… — vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hift_graph.py
  At capacity the fallback is permanent for every new shape: the check at hift_graph.py:60-61 returns eager BEFORE the self.seen recurrence counter is updated (lines 62-67), so a shape first seen after the 32-graph cap is reached can never hit the 3-recurring threshold for capture; and nothing ever evicts from self.graphs — the only trim is self.seen.popitem(last=False) at line 65, which affects the recency table only. A mixed-tail-length streaming server that accumulates 32 distinct shapes silently loses decode graphing for all newer shapes for the process lifetime. Since self.seen already tracks recency in an OrderedDict, consider evicting the least-recently-seen resident graph before falling back.

Evidence: hift_graph.py:60-61 if len(self.graphs) >= self.max_graphs: / return self.decode(mel, source, finalize=finalize) — returns before the counter bookkeeping at lines 62-65 (count = self.seen.pop(key, 0) + 1 ... self.seen.popitem(last=False)), so post-cap shapes never accumulate hits; hift_graph.py:80 self.graphs[key] = state is the only mutation of self.graphs in the 85-line file (verified by full read). Eviction consumers: none — repo-wide grep shows the only external users are hifigan.py:974 self._decode_graphs = HiFTDecodeGraphs(self._decode_eager) (construction) and the new test, which at tests/model_executor/models/cosyvoice3/test_cosyvoice3_hift_graph.py:35 hift._decode_graphs.max_graphs = 1 then line 41 assert len(hift._decode_graphs.graphs) == 1 codifies that an off-cap shape goes eager while the resident graph is kept. (hift_graph.py is a new file added by this diff; the grep hits in higgs_audio_v3_code2wav.py are an unrelated same-named attribute in a different class.)

Suggestion: if len(self.graphs) >= self.max_graphs:
for old in self.seen:
if old in self.graphs:
del self.graphs[old]
break

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[vllm-omni-review-bot]

[P1] This diff's new HiFTDecodeGraphs.run() captures the HiFT decode CUDA graph with…

This diff's new HiFTDecodeGraphs.run() captures the HiFT decode CUDA graph with no failure path (hift_graph.py:74-76: graph = torch.cuda.CUDAGraph() / with torch.cuda.graph(graph, pool=self.pool):): capture happens lazily on a live request (each shape runs eager only for its first two occurrences, hift_graph.py:66-67, and hifigan.py decode() routes here via runner.run(x, s, finalize)), so if capture raises — OOM, workspace setup, or stream activity from the engine's connector threads — the exception propagates out of decode() and fails the streaming request, and nothing is logged. The closest in-tree sibling already solved this exact environment: minimax_music3/depth_graph.py:107-137 wraps warmup+capture in try/except, adds the shape to a _disabled set, logs a warning, and captures with pool=current_platform.get_global_graph_pool() and capture_error_mode="thread_local" — its comment (depth_graph.py:117-120) explains the engine's connector threads keep their own streams busy and would fail a global-mode capture. That is exactly the environment the enabling profile runs: the only profile setting COSYVOICE3_HIFT_GRAPH=1 is the new cosyvoice3_packed_streaming_optimized_standard.yaml:22, whose chain configures SharedMemoryConnector (cosyvoice3_packed_streaming.yaml:7-8 connector_of_shared_memory:). Match the sibling: try/except around warmup+capture with a per-key disabled set, eager fallback, warning log, the engine's global graph pool, thread_local error mode, and an info log on successful capture (needs self.disabled = set() in init and a key in self.disabled check in the miss path).

Evidence: In the diff: hift_graph.py:74-75 graph = torch.cuda.CUDAGraph() / with torch.cuda.graph(graph, pool=self.pool): — the entire 85-line new file contains no try/except and no logger, and capture is reached from the live path (hift_graph.py:66-67 if count < 3: / return self.decode(mel, source, finalize=finalize)). Sibling, unchanged by this diff, present in the PR-time tree: depth_graph.py:129-130 except Exception as exc: # noqa: BLE001 - any capture failure must fall back, not crash / self._disabled.add(pairs); depth_graph.py:116 and 121 pool=current_platform.get_global_graph_pool(), / capture_error_mode="thread_local",; depth_graph.py:117-120 comment # The engine's connector threads keep their own streams busy while this runs.. Enabling profile, new in this diff: cosyvoice3_packed_streaming_optimized_standard.yaml:22 COSYVOICE3_HIFT_GRAPH: "1"; its connector chain, unchanged: cosyvoice3_packed_streaming.yaml:7-8 connectors: / connector_of_shared_memory:.

Suggestion: try:
static_mel, static_source = mel.clone(), source.clone()
stream = torch.cuda.Stream(device=mel.device)
stream.wait_stream(torch.cuda.current_stream(mel.device))
with torch.cuda.stream(stream):
self.decode(static_mel, static_source, finalize=finalize)
torch.cuda.current_stream(mel.device).wait_stream(stream)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(
graph,
pool=current_platform.get_global_graph_pool(),
capture_error_mode="thread_local",
):
output = self.decode(static_mel, static_source, finalize=finalize)
except Exception as exc:
self.disabled.add(key)
logger.warning("HiFT decode CUDA graph capture failed for %s; staying eager: %s", mel.shape, exc)
return self.decode(mel, source, finalize=finalize)
state = _DecodeGraph(graph, static_mel, static_source, output)
self.graphs[key] = state

[vllm-omni-review-bot]

[P2] The new guard (omni_model_state.py:430-434: `for_capture and input_batch.max_qu…

The new guard (omni_model_state.py:430-434: for_capture and input_batch.max_query_len is None and not self.vllm_config.compilation_config.cudagraph_mode.separate_routine()) skips the worst-case-bucket widening whenever the config-level cudagraph_mode is a separate-routine mode, while production capture passes for_capture only for per-batch FULL-routine dummy runs (omni_model_runner.py:598: for_capture=dummy_run and batch_desc.cg_mode == CUDAGraphMode.FULL) — so for FULL_DECODE_ONLY / FULL_AND_PIECEWISE deployments the decision comes entirely from the config. The change reaches every v2 stage whose model declares Omni lifecycle flags, not just CosyVoice3: OmniModelState is built for any such model (model_states/init.py:37-45), and e.g. the qwen3_tts talker sets has_preprocess = True (qwen3_tts_talker.py:386) under model_runner: v2 (qwen3_tts.yaml:20), as do the higgs/qwen3-omni mrv2 deploys — their decode-capture metadata switches from the widened bucket bound to the uniform dummy bound. This PR's replay evidence is the qwen3/FA3-shaped synthetic plus CosyVoice3 WER; can you name one non-CosyVoice3 (ideally non-FA3) capture validation run at this head, or state in the guard comment why the uniform dummy bound is backend-independent? minimax_music3.yaml:54 ('FULL_DECODE_ONLY only ever dispatches a graph for a uniform decode batch') suggests it is — making that explicit here would close the scoping question.

Evidence: In the diff — vllm_omni/worker_v2/model_states/omni_model_state.py:430-433: if ( for_capture and input_batch.max_query_len is None and not self.vllm_config.compilation_config.cudagraph_mode.separate_routine() ):. Unchanged by this diff, present in the PR-time tree — vllm_omni/worker_v2/omni_model_runner.py:598: for_capture=dummy_run and batch_desc.cg_mode == CUDAGraphMode.FULL,; vllm_omni/worker_v2/model_states/init.py:37-40: uses_omni_lifecycle = any( getattr(model, flag, False) is True for flag in ("has_preprocess", "has_postprocess", "have_multimodal_outputs") ) with :45 return OmniModelState(vllm_config, model, encoder_cache, device) (so construction is lifecycle-flag-gated, not 'every worker_v2 model'); vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py:385-386: self.have_multimodal_outputs = True / self.has_preprocess = True; vllm_omni/deploy/qwen3_tts.yaml:20: model_runner: v2; vllm_omni/deploy/minimax_music3.yaml:54: # captured. FULL_DECODE_ONLY only ever dispatches a graph for a uniform.

[vllm-omni-review-bot]

[P2] The new vllm_omni/deploy/cosyvoice3_packed_streaming_optimized_standard.yaml:2…

The new vllm_omni/deploy/cosyvoice3_packed_streaming_optimized_standard.yaml:2 requires --hf-overrides '{"cosyvoice3_sampling_mode": "standard"}' and stream=true, but nothing in the diff or tree enforces it: runtime.py:63 defaults cosyvoice3_sampling_mode to 'ras', and runtime.py:64-65 only validate the string, silently selecting RAS otherwise. Repo-wide, the only readers of cosyvoice3_sampling_mode are those runtime.py lines and tests; every caller of cosyvoice3_standard_sampling (cosyvoice3.py:743/1031/1137/1178, tts_adapters/cosyvoice3.py:156) just branches on the boolean, and no code references this profile by name. Serving this YAML without the flag therefore silently runs the throughput profile under RAS sampling — the same silent config/quality mismatch class this PR's WER numbers (67% -> 1.4%) warn about. (Sibling cosyvoice3_full_response_high_concurrency_standard.yaml:1-3 shares the comment-only convention, so a guard belongs in the shared load path.) Can you confirm a guard exists elsewhere, or fail loudly at load when this profile runs with ras?

Evidence: Added by this diff, vllm_omni/deploy/cosyvoice3_packed_streaming_optimized_standard.yaml:2: # Use --hf-overrides '{"cosyvoice3_sampling_mode": "standard"}' and stream=true. — comment-only; the file sets only COSYVOICE3_* env gates and default_sampling_params. Unchanged by this diff, present in the PR-time tree: vllm_omni/model_executor/models/cosyvoice3/runtime.py:63 mode = getattr(config, "cosyvoice3_sampling_mode", "ras") and runtime.py:65 raise ValueError("cosyvoice3_sampling_mode must be 'ras' or 'standard'") — default 'ras', string validation only. Repo-wide grep confirms cosyvoice3_sampling_mode occurs only in runtime.py, tests, and the two YAML headers, the profile name packed_streaming_optimized_standard appears in no file content (no load-path guard keys on it), and all callers of cosyvoice3_standard_sampling — cosyvoice3.py:743 standard_sampling = cosyvoice3_standard_sampling(self.config), cosyvoice3.py:1031/1137/1178, tts_adapters/cosyvoice3.py:156 — are plain if-branches with no raise or warning on packed-streaming + ras.

[vllm-omni-review-bot]

[P2] The widening skip at vllm_omni/worker_v2/model_states/omni_model_state.py:433 k…

The widening skip at vllm_omni/worker_v2/model_states/omni_model_state.py:433 keys on CUDAGraphMode.separate_routine(), an enum method defined only in the external vllm dependency (the test imports it from vllm.config.compilation); no vllm source is readable in the review environment, so the membership this guard needs (FULL_AND_PIECEWISE/FULL_DECODE_ONLY -> True, FULL -> False) is inferred solely from in-repo references — four upstream-copied comment blocks (e.g. vllm_omni/worker/gpu_model_runner.py:1005) plus this PR's own test, which derives decode from the same predicate and therefore cannot catch a wrong membership. Since this predicate alone decides whether decode captures keep the uniform Q=1 dummy bound — the path that would otherwise re-bake the mismatched FA3 GQA layout this PR fixes — please confirm the membership against the pinned vllm source, or record the assumed membership in the guard's comment.

Evidence: IN DIFF (new code): vllm_omni/worker_v2/model_states/omni_model_state.py:433 and not self.vllm_config.compilation_config.cudagraph_mode.separate_routine() — the widening skip keys on the upstream predicate; tests/worker_v2/test_capture_query_bound_gpu.py:35 decode = graph_mode.separate_routine() and :37 runtime_mode = CUDAGraphMode.FULL if decode else CUDAGraphMode.NONE — test derives its shapes from the same predicate the guard consults, so a wrong membership flips both together undetected. UNCHANGED BY THIS DIFF, PRESENT IN PR-TIME TREE: tests/worker_v2/test_capture_query_bound_gpu.py:10 from vllm.config.compilation import CUDAGraphMode — the enum lives in the external vllm dependency; vllm_omni/worker/gpu_model_runner.py:1005-1006 # If cudagraph_mode.decode_mode() == FULL and / # cudagraph_mode.separate_routine(). This means that we are using — upstream-copied doc tying separate_routine() to 'different graphs and/or modes for mixed prefill-decode batches vs. uniform decode batches' (corroborates, does not prove, the assumed membership). NEGATIVE CHECK: repo-wide grep for separate_routine returns only call/comment sites (omni_model_state.py:433, test:35, gpu_model_runner.py:1005-1008, gpu_generation_model_runner.py:654-657, npu_model_runner.py:137-140, npu_generation_model_runner.py:644-647) — no definition of separate_routine exists in the tree, and no installed vllm source was found under /usr/local/lib, /usr/lib, /opt, or /home/ubuntu/.local.

[vllm-omni-review-bot]

[P2] enable_decode_graphs() (hifigan.py:973, added here) unconditionally calls `se…

enable_decode_graphs() (hifigan.py:973, added here) unconditionally calls self.enable_cached_istft(), but the load path (cosyvoice3_code2wav.py:773-777) enables cached ISTFT only when COSYVOICE3_CACHED_ISTFT=1 — so COSYVOICE3_HIFT_GRAPH=1 alone turns on cached ISTFT even against an explicit COSYVOICE3_CACHED_ISTFT=0, without the 'cached ISTFT enabled' log (that log fires only in the explicit branch), although the inventory promotes the two as independent MODEL_SPECIFIC vars (environment_variable_inventory.py:151,157) and the new yaml sets both. The coupling itself is required (HiFTDecodeGraphs docstring: decode 'must have warmed its ISTFT envelope before capture'), so keep it but make the override visible: a comment at this call, or an info log when the implicit enable happens.

Evidence: hifigan.py:969-974 (added by this diff): def enable_decode_graphs(self) -> None: ... self.enable_cached_istft() — unconditional at line 973. cosyvoice3_code2wav.py:773-777 (added by this diff): if os.getenv("COSYVOICE3_CACHED_ISTFT", "0") == "1": / self.hift.enable_cached_istft() then if os.getenv("COSYVOICE3_HIFT_GRAPH", "0") == "1": / self.hift.enable_decode_graphs() — the graph flag alone reaches the unconditional enable, and the cached-ISTFT log at line 775 is inside the explicit branch only. hift_graph.py:28-29 (added by this diff): "decode must have warmed its ISTFT envelope before capture" — coupling required, keep. environment_variable_inventory.py:151,157 (added by this diff): both "COSYVOICE3_CACHED_ISTFT", and "COSYVOICE3_HIFT_GRAPH", listed under _MODEL_PROMOTE as independent vars.

Suggestion: # Replay needs the deterministic cached-ISTFT path, so this
# deliberately overrides an explicit COSYVOICE3_CACHED_ISTFT=0.
self.enable_cached_istft()

[vllm-omni-review-bot]

[P2] The new regression test test_mm_embeddings_exclude_zero_length_graph_padding_ro…

The new regression test test_mm_embeddings_exclude_zero_length_graph_padding_rows pins the producer slice but asserts it on a MagicMock: _make_state builds the model with model = MagicMock() (tests/worker_v2/test_omni_model_state.py:42), so line 96 inspects embed_input_ids.call_args without ever reaching real consumer code. The original failure path of this bugfix — a padded graph batch feeding the real CosyVoice3.embed_input_ids, whose validation raises on non-covering boundaries (cosyvoice3.py:1244-1245 raise ValueError("CosyVoice3 input embedding boundaries must cover all scheduled tokens")) — is therefore covered only piecewise (this test for the slice; test_cosyvoice3_model_helpers.py:677-786 for the consumer with hand-built boundaries). Extend this test (or add a sibling) to feed the sliced boundaries from this padded batch into the real CosyVoice3.embed_input_ids and assert the rearranged output, pinning the composed path that used to raise.

Evidence: tests/worker_v2/test_omni_model_state.py:42 model = MagicMock() and :58 state.model = model (helper _make_state, unchanged by the diff) — hence the assert at :96 assert state.model.embed_input_ids.call_args.kwargs["query_start_loc"] == [0, 2, 5] inspects only a MagicMock and the kwarg never reaches real consumer code; the real consumer validation is unchanged by this diff, present in the PR-time tree: vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py:1244-1245 if not boundaries or boundaries[0] != 0 or boundaries[-1] != total_tokens: / raise ValueError("CosyVoice3 input embedding boundaries must cover all scheduled tokens"); the piecewise real-consumer coverage cited is tests/model_executor/models/cosyvoice3/test_cosyvoice3_model_helpers.py:677 def test_embed_input_ids_preserves_interleaved_request_boundaries(order): calling model.embed_input_ids with hand-built query_start_loc=boundaries at :707.

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] This diff's new HiFTDecodeGraphs.run() captures the HiFT decode CUDA graph with… — vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hift_graph.py:75

This diff's new HiFTDecodeGraphs.run() captures the HiFT decode CUDA graph with no failure path (hift_graph.py:74-76: graph = torch.cuda.CUDAGraph() / with torch.cuda.graph(graph, pool=self.pool):): capture happens lazily on a live request (each shape runs eager only for its first two occurrences, hift_graph.py:66-67, and hifigan.py decode() routes here via runner.run(x, s, finalize)), so if capture raises — OOM, workspace setup, or stream activity from the engine's connector threads — the exception propagates out of decode() and fails the streaming request, and nothing is logged. The closest in-tree sibling already solved this exact environment: minimax_music3/depth_graph.py:107-137 wraps warmup+capture in try/except, adds the shape to a _disabled set, logs a warning, and captures with pool=current_platform.get_global_graph_pool() and capture_error_mode="thread_local" — its comment (depth_graph.py:117-120) explains the engine's connector threads keep their own streams busy and would fail a global-mode capture. That is exactly the environment the enabling profile runs: the only profile setting COSYVOICE3_HIFT_GRAPH=1 is the new cosyvoice3_packed_streaming_optimized_standard.yaml:22, whose chain configures SharedMemoryConnector (cosyvoice3_packed_streaming.yaml:7-8 connector_of_shared_memory:). Match the sibling: try/except around warmup+capture with a per-key disabled set, eager fallback, warning log, the engine's global graph pool, thread_local error mode, and an info log on successful capture (needs self.disabled = set() in init and a key in self.disabled check in the miss path).

Evidence: In the diff: hift_graph.py:74-75 graph = torch.cuda.CUDAGraph() / with torch.cuda.graph(graph, pool=self.pool): — the entire 85-line new file contains no try/except and no logger, and capture is reached from the live path (hift_graph.py:66-67 if count < 3: / return self.decode(mel, source, finalize=finalize)). Sibling, unchanged by this diff, present in the PR-time tree: depth_graph.py:129-130 except Exception as exc: # noqa: BLE001 - any capture failure must fall back, not crash / self._disabled.add(pairs); depth_graph.py:116 and 121 pool=current_platform.get_global_graph_pool(), / capture_error_mode="thread_local",; depth_graph.py:117-120 comment # The engine's connector threads keep their own streams busy while this runs.. Enabling profile, new in this diff: cosyvoice3_packed_streaming_optimized_standard.yaml:22 COSYVOICE3_HIFT_GRAPH: "1"; its connector chain, unchanged: cosyvoice3_packed_streaming.yaml:7-8 connectors: / connector_of_shared_memory:.

Suggestion: try:
static_mel, static_source = mel.clone(), source.clone()
stream = torch.cuda.Stream(device=mel.device)
stream.wait_stream(torch.cuda.current_stream(mel.device))
with torch.cuda.stream(stream):
self.decode(static_mel, static_source, finalize=finalize)
torch.cuda.current_stream(mel.device).wait_stream(stream)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(
graph,
pool=current_platform.get_global_graph_pool(),
capture_error_mode="thread_local",
):
output = self.decode(static_mel, static_source, finalize=finalize)
except Exception as exc:
self.disabled.add(key)
logger.warning("HiFT decode CUDA graph capture failed for %s; staying eager: %s", mel.shape, exc)
return self.decode(mel, source, finalize=finalize)
state = _DecodeGraph(graph, static_mel, static_source, output)
self.graphs[key] = state

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

[linyueqian]

Looks good to me. The prepare_attn change is the narrowing I was hoping for when the capture bound first went in: the worst-case query bound now only applies when FULL graphs also serve mixed batches, and separate decode routines keep their uniform Q=1 bound, so FA3's captured layout matches the runtime schedule. The parametrized bound test covers FULL, FULL_AND_PIECEWISE and FULL_DECODE_ONLY against the real FA3 backend, and it is wired into the ready CUDA job.

The rest reads correct to me:

• query_start_loc now stops at num_reqs + 1, so zero-length graph padding rows no longer reach embed_input_ids.
• In packed streaming, every query segment is at most static_chunk_size (the layout splits rows by that span), so using it as max_seqlen_q is a valid bound and much tighter than the page table width.
• The HiFT graphs are opt-in, bounded (32 graphs, one shared pool), return an owned clone, and fall back to eager for unseen shapes, CPU and nested capture.

Two small notes inline, neither blocking. CI is green apart from the NPU job still running.

[linyueqian]

Capture happens on the serving path the third time a shape is seen, so that request pays for one eager warmup plus the capture. With streaming chunks the set of shapes is small, so this is fine as is. If the first-request latency ever matters for this profile, capturing the common chunk shapes at load (as the Higgs codec graphs now do) would move the cost before readiness. It would also help to mention in the YAML comment that the graph pool is outside gpu_memory_utilization.

[linyueqian]

Thanks for narrowing this. One thing worth stating in the comment or the PR body for future readers: a profile running cudagraph_mode: FULL (for example the Higgs MRv2 H200 profile) still gets the widened bound for its decode graphs, since it has no separate decode routine. That matches the previous behavior there, I just want it to be explicit that only the separate-routine modes change.