[MrDongsls]

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Higgs Audio v3 runs on Model Runner V2 by default and passes the #9387 parity bar.

2. What is the highest-leverage variable for that outcome?

Whether V2 matches V1 on quality and performance for this model, measured on the same vLLM build.

3. How will this work move that variable?

Upstream vLLM may drop Model Runner V1 (vllm-project/vllm#63373). Higgs v3 already had an MRV2 path, but it was reachable only through eight yaml-only flags in the H200 profile. This PR makes model_runner: v2 self-sufficient, flips the default profile, and compares V2 with V1.

4. Whose customer or user needs does this address, and what are they?

People who serve bosonai/higgs-audio-v3-tts-4b and need to stay on a vLLM-Omni that drops V1. They need v3 to run on V2 the way it runs on V1, with a V1 opt-out while V1 still exists.

Purpose

Upstream vLLM plans to remove Model Runner V1 (vllm-project/vllm#63373). Issue #9387 tracks the core TTS models. Its table lists Higgs Audio as a v3 opt-in profile with no v2-generation MRV2 path. This PR promotes v3's MRV2 path from the opt-in higgs_multimodal_qwen3_mrv2_h200.yaml to the shipped default. The v2-generation port is staged separately and follows the codec-stage ruling in #9387.

Before, the default and both perf profiles ran v1, and a bare model_runner: v2 on the default yaml raised at startup. After, the MRV2 feature switches self-enable from the runner selection, the default yaml selects v2, and higgs_multimodal_qwen3_high_throughput.yaml and higgs_multimodal_qwen3_low_latency.yaml remain v1 as the opt-out.

Changes:

• The eight audio_mrv2_* and audio_async_* switches become tri-state config fields that default to the model runner selection. model_runner: v2 on the default yaml now works without the H200 profile's hf_overrides block.

• The external decode graph decision moves from the model state into the talker's __init__. This also fixes a latent case where v2 with enforce_eager: true skipped the MLP graph disarm because the flag was set after init.

• deploy/higgs_multimodal_qwen3.yaml selects v2 with the H200-validated stage shapes scaled to 16 sequences. The connector streaming geometry follows the codec graph windows, so the first chunk is now one 8-frame window, about 320 ms, where V1 shipped about 40 ms. Steady-state cadence is unchanged.

• NPU, XPU, ROCm and MUSA pin v1 through the platforms: section. XPU rejects v2 stages at deploy load, and NPU generation workers reject v2 at init.

• The v2 sampler supports temperature, top_p, top_k and seed. Other sampling parameters fail at startup when set non-neutrally in default_sampling_params, and per request with a message naming the v1 opt-out. A follow-up issue will track penalties.

• Under model_runner: v1, explicitly set MRV2 switches log a warning instead of being silently ignored, and the default profile's async_scheduling: true — a v2 requirement that no shipped V1 profile exercises — logs a warning naming the V1 profiles.

• The platforms: section can scope connector settings; the Higgs fallback platforms restore the shipped V1 streaming geometry.

• Tests cover the runner matrix across platforms, the flipped profile shape, flag resolution, the startup fail-fast, and e2e under the default, the V1 opt-out and the H200 params, including a streaming abort and a long input. The dfx entry gains a c=4 latency point.

• Docs: docs/configuration/stage_configs.md, deploy/README_higgs_audio_v3.md, and the recipe.

Test Plan

vLLM Version: 0.32.0

V1 is the parent commit with the old default yaml. V2 is this PR. Both run the same vLLM build.

• Quality: seed-tts-eval on seed-tts-text/en, one server per runner, same requests and seeds.
• Performance: the dfx entry on H20 at c=1, 4 and 16. TTFA on V2 measures the first 8-frame chunk against V1's 1-frame chunk; e2el separates the cadence change from a real regression.
• Streaming, abort and long input: tests/e2e/online_serving/test_higgs_audio_v3.py under the default and the V1 opt-out params. The ROCm pool runs the default yaml's plain-text param to exercise the v1 fallback.
• Invalid-parameter handling: tests/dfx/reliability/invalid_param_test/test_invalid_audio_speech.py boots the default yaml, so its 4xx expectations move to the v2 path with this PR and get re-validated in the same CI run.

Test Result

Local CPU suites on this PR: 757 passed across tests/model_executor/models/higgs_audio_v3/ and tests/config/test_config_factory.py, and 15 passed across the worker, stage-input-processor and adapter tests. test_codec_snake_kernel fails the same way on main in this environment and is not touched here.

Measured on 1x H20 96GB, matched A/B — same machine, same vllm 0.32.0 build, same datasets and seeds, no env overrides. V1 is the pre-flip checkout 4daeba8b4 running its shipped default (eager); V2 is this PR at c195ebb (the commit after it touches tests only). The comparison is shipped default against shipped default. Each point below is one measurement per server run, two server runs per side; run-to-run spread is given where it matters.

Quality — seed-tts-eval EN hard, 1088 voice_clone prompts at concurrency 4; WER by whisper-large-v3, speaker similarity by wavlm-base-plus, plus UTMOS. Zero failed evaluations in all four runs.

metric	V1 run1 / run2	V2 run1 / run2	V2−V1 (per run)
WER	0.0178 / 0.0160	0.0165 / 0.0163	−0.0013 / +0.0003
SIM	0.8802 / 0.8804	0.8798 / 0.8798	−0.0004 / −0.0006
UTMOS	3.3587 / 3.3587	3.3593 / 3.3593	+0.0005 / +0.0005

V1's own two WER runs differ by 0.0018, more than either cross-runner difference, so WER is within run-to-run variation in both runs. SIM and UTMOS repeat to the fourth decimal within a side, so the small cross-side differences are consistent rather than noise: V1 is higher on SIM by 0.0004–0.0006 and V2 is higher on UTMOS by 0.0005 — 0.03–0.07% of the value, inside the ±0.002 SIM band, with the floors (WER ≤ 0.05, SIM ≥ 0.75) holding on every run.

Performance — the dfx entry on seed-tts-text prompts; medians, min–max over the two runs per side:

concurrency	metric	V1	V2	V2/V1
1	TTFA (median_audio_ttfp_ms)	259.6–265.0	88.8–91.2	0.34×
1	RTF (median_audio_rtf)	0.558–0.560	0.107–0.108	0.19×
4	TTFA	287.6–289.9	100.8–103.7	0.35×
4	RTF	0.586–0.594	0.125–0.126	0.21×
4	audio/s	6.60–6.69	31.5–32.0	4.8×
16	TTFA	334.5–335.7	144.4–148.8	0.43×
16	RTF	0.676–0.677	0.181	0.27×
16	audio/s	23.1–23.2	87.2–87.4	3.8×

Run-to-run spread on both sides stays within 3% on TTFA and 1.5% on RTF and throughput. TTFA reads V2's first chunk (8 codec frames, ~320 ms of audio) against V1's 1-frame chunk.

Memory — startup accounting, identical across both runs (default yaml, stage-0 util 0.6): weights 7.61 GiB on both sides; available KV 49.01 GiB on V1 and 48.73 GiB on V2 (−0.6%); consumed 7.81 vs 7.97 GiB; V2 adds a 0.06 GiB CUDA-graph pool. Under the dfx load matrix (util 0.3/0.2, c=1/4/16, 5 s sampling) peak device memory is 37.2 GiB on V1 and 31.1 GiB on V2.

Streaming behavior — the e2e suite under dummy weights, shared matrix: 43/43 on the V2 branch across plain_text (V2 default), v1_optout_high_throughput (V1 escape hatch) and mrv2_h200 (tuned profile); 13/13 on the v1 baseline checkout.

Not run:

• NPU, XPU and MUSA launches.
• The H200 profile beyond test collection.

Not changed:

• The V1 opt-out profiles and the H200 profile.
• V1 behavior. The old default shape is available on _high_throughput.yaml and _low_latency.yaml.

[jiangmengyu18]

Explicitly neutral defaults such as logprobs: null, prompt_logprobs: null, allowed_token_ids: null, bad_words: [], and logit_bias: {} are rejected at startup, although this PR’s request path accepts them. Complete the neutral-value handling or share normalized validation between startup and requests.

[MrDongsls]

Fixed in c195ebb: startup validation now passes None/[]/{} for those keys, matching the request path. Non-neutral values still fail with the v1 opt-out named.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-30T17:26:13.145226Z We couldn’t complete this review. Comment @codex review to try again.	3ec115c	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[amy-why-3459]

Thanks for the detailed parity plan. Making the eight MRV2 switches tri-state and resolving them once into a frozen HiggsMrv2Options is a nice cleanup: the model code no longer reads raw config attributes, explicit False on the two structural flags fails loudly under v2, and v1 warns instead of silently ignoring set switches. Moving the external-decode-graph decision into the talker __init__ also closes the enforce_eager: true + v2 ordering gap you describe.

A few things worth a look:

1. Unrelated .claude/skills/review-pr/* changes. Four files under .claude/skills/review-pr/ ("smallest fix direction" to "root-cause fix direction") are part of this diff. They don't relate to Higgs v3 / MRV2; could you split them into their own PR so this one stays reviewable as a model-runner change?

2. Connector geometry is not restored for the V1 platform fallbacks. The platforms: overrides put ROCm/NPU/XPU/MUSA back on model_runner: v1 and restore the stage-0 V1 shape (eager, no async scheduling, prefix caching), and the comment says the fallback keeps "the shipped V1 shape". But the top-level connector_of_shared_memory changes (codec_left_context_frames: 25 -> 8, codec_right_holdback_frames: 4 -> 0, initial_codec_chunk_frames: 1 -> 8) are not platform-scoped, so those V1 fallbacks now also stream with the V2 codec geometry. If that's intended, the comment should say so and the ROCm pool run should cover audio quality with it; if not, those keys need a per-platform restore (or a note if platforms: can't override connectors).

3. model_runner: v1 on the default file is an untested combination. Someone who keeps higgs_multimodal_qwen3.yaml but overrides model_runner: v1 (CLI or a copied file, which the header suggests) gets enforce_eager: false, async_scheduling: true, enable_prefix_caching: false, FLASH_ATTN, and the V2 codec geometry on the V1 runner. Since every MRV2 switch resolves off under v1, is async_scheduling: true supported by the V1 Higgs path? A startup check, or a test that boots the default yaml with model_runner: v1, would catch a regression here.

4. Startup sampling check only covers deploy defaults. _validate_mrv2_default_sampling_params checks default_sampling_params at deploy-merge time, and add_request rejects per-request values, which is good. One gap: the neutral-value table only has scalar entries, so a default like logit_bias: {} or bad_words: [] passes (fine), but logprobs: 0 would be rejected even though it requests no logprobs on most paths. If that's intentional, a comment would help; otherwise add logprobs/prompt_logprobs with their neutral value to _MRV2_NEUTRAL_SAMPLING_VALUES.

5. Minor: stage-0 compilation_config (cudagraph_mode: FULL, capture sizes up to 16) and stage-1 hf_overrides (codec_graph_*) are inherited by every platform fallback. With enforce_eager: true and v1 they should be ignored, but a quick note that they're CUDA/V2-only would save the next reader from wondering.

The TTFA change for the first chunk (about 320 ms vs about 40 ms on the V1 profiles) is called out clearly in the description; I'll leave whether that's the right default to the maintainers tracking #9387.

[amy-why-3459]

Took a look at the three new commits since my last pass (7e7c4242, c928ba1e, 16accccf). They're small and mostly documentation; a couple of notes:

• higgs_audio_v3_talker.py – new v1 + async_scheduling warning. Makes sense as a guardrail. Since it lives in the talker __init__, it will be emitted once per worker/rank (and again on every re-instantiation), so with TP > 1 the log gets noisy. logger.warning_once(...) (available on vLLM's init_logger loggers) would keep the signal without the repetition. It might also help to say what the user should do (e.g. "set async_scheduling: false or use model_runner: v2") rather than only that the combination isn't exercised.
• stage_config.py – logprobs / prompt_logprobs rationale. The explanation for keeping them out of _MRV2_NEUTRAL_SAMPLING_VALUES is clear and matches the request-path behaviour from c195ebb2. If there isn't one already, a tiny unit test asserting that logprobs: 0 is still rejected at startup (while logprobs: null passes) would lock this intent in, so a future "helpful" addition of 0 to the neutral table gets caught.
• higgs_multimodal_qwen3.yaml – fallback comment. Good to call out that stage-0 compilation_config and stage-1 codec_graph_* are inert on the v1 fallbacks. Minor: the last sentence ("the fallbacks stream with the v2 chunk shape above") could mention whether that chunk shape has been sanity-checked on the V1 runner, since that's the one setting that does carry over.

No functional concerns with the rest of this increment.

[amy-why-3459]

Thanks for the follow-ups. I went through the four new commits since my last pass (346e325d, 0fbc9b38, 97ede951, 1278c535). The warning_once change, the clearer warning text, and the new logprobs: 0 startup test all look good. A few notes on the new per-platform connectors override:

• _apply_platform_overrides and non-mapping overrides. The outer connectors value is checked to be a dict, but each per-connector override isn't. A YAML typo such as connector_of_shared_memory: null or a scalar would fail inside _get_recursively_merged_dict with an AttributeError (.items() on None), not a clear ValueError. Adding an isinstance(override, dict) check next to the unknown-name check would keep the error messages consistent.
• Interaction with base_config overlays. _merge_platforms merges stages by stage_id, but every other platform key, now including connectors, is replaced shallowly ({**bp, **op}). If an overlay deploy file sets platforms.<plat>.connectors for just one key, the base file's full V1 geometry restore for that platform is dropped, not merged. That may be fine for now since nothing inherits from this file yet, but it's worth either deep-merging connectors there too or noting the shallow behaviour. A related point: the platform restore runs after top-level connectors are merged, so an overlay that changes top-level codec_* frames would be silently overridden on the non-CUDA fallbacks by the hard-coded 25/4/1.
• YAML duplication. The same three-key connector block now appears four times (rocm/npu/xpu/musa). A YAML anchor (&v1_connector_geometry / *v1_connector_geometry) would keep them from drifting apart if the V1 geometry ever changes.
• Test coverage. The new assertions cover the restore and the unknown-name error. A small check that _apply_platform_overrides leaves other connector extra keys intact (to prove it really is a deep merge, not a replace) would protect the merge semantics.

No functional concerns with the docs/comment trim or the talker warning change.

[MrDongsls]

Thanks @amy-why-3459 — everything from both passes is in:

1. Split out; the .claude wording edits are gone from the branch.
2. connectors is now platform-scopable — deep-merged per named connector, unknown names rejected — and the Higgs fallback platforms restore the shipped V1 geometry (25/4, 1-frame first chunk): 346e325d.
3. Boot-tested the flipped default on H20 (model_runner: v1, i.e. v1 + async_scheduling: true + the v2 connector shape): the warning fires exactly once and both a plain and a streaming request serve 200, so it stays informational. Also switched to warning_oncewith actionable wording:1278c53`.
4. Kept logprobs=0 rejected: it still requests the sampled token's logprob and the request path rejects any explicit value, so startup matches it. Stated on the table and locked by a test: 97ede95.
5. One-line note in the yaml, as suggested.

[amy-why-3459]

Thanks for the quick turnaround. I looked only at the new commit since my last pass (9f92aa02). It covers the earlier notes well: the per-connector isinstance(override, dict) check gives a clear ValueError, _merge_platforms now deep-merges connectors by name through _merge_connectors, the YAML anchor removes the four copies, and the new tests check both the deep-merge semantics and the mapping error.

A few small notes:

• Clearing connectors from an overlay. Because _merge_platforms now always deep-merges, an overlay that writes platforms.<plat>.connectors: null (or {}) keeps the base file's connector restores instead of removing them. That's probably the right default, but there's no longer a way for an overlay to opt out of the inherited V1 geometry for a platform. A short comment on _merge_platforms saying this is intended would help future readers.
• Overlay that nulls a single connector. If an overlay sets connector_of_shared_memory: null under a platform, _get_recursively_merged_dict replaces the base mapping with None, and the new check in _apply_platform_overrides then raises must be a mapping. The error is clear, but it points at the merged result, not at the overlay file that caused it. Not a blocker, just worth knowing.
• Anchor aliasing. npu, xpu, and musa now alias the rocm mapping. That's safe as long as the loader (load_yaml_config + to_dict) returns independent containers per alias and nothing mutates the platform connectors dict in place. _get_recursively_merged_dict copies, so this looks fine today. A one-line assertion in the test that rocm and xpu restores are distinct objects (or that mutating one doesn't affect the other) would guard against a future in-place edit.
• Still open from the last pass. The non-CUDA restores still hard-code 25/4/1, so an overlay that changes the top-level codec_* frames is silently overridden on rocm/npu/xpu/musa. Fine to leave as is if that's intended, but a comment next to &v1_connector_geometry noting it would make the precedence explicit.

[amy-why-3459]

Looked only at the new commit since my last pass (46bcf19c). It picks up the remaining notes: the _merge_platforms docstring now states that an overlay can't clear inherited connector restores, the comment next to &v1_connector_geometry makes the precedence over top-level codec_* explicit, and the test now asserts that the anchor resolves to distinct containers.

One small note on the new test:

• The mutation check doesn't exercise aliasing. rocm and xpu come from two separate load_deploy_config(deploy_path) calls, so they are independent objects no matter how the anchor is resolved, and the == 25 assertion would pass even if the loader returned shared containers. The raw[...] is not raw[...] line above it is the part that actually guards the anchor. If you want the mutation check to add coverage, mutate the single raw dict instead (e.g. change raw["platforms"]["rocm"]["connectors"]["connector_of_shared_memory"]["extra"]["codec_left_context_frames"] and assert the xpu entry is unchanged), so a shared container would show up as a leak. Not a blocker.

Nothing else from me on this delta.

[amy-why-3459]

Looked only at the new commit since my last pass (bf5d3b0b). This resolves the note on the anchor test: the mutation now goes through a single resolve_deploy_yaml parse, so if the &v1_connector_geometry anchor ever resolved to a shared container, writing codec_left_context_frames = 99 on the rocm entry would show up on the xpu entry and the == 25 assertion would fail. Mutating raw is also safe for the rest of the test, since resolve_deploy_yaml re-reads the YAML on every call (no caching), so the later load_deploy_config(deploy_path) for cuda still sees the original values.

Nothing else from me on this delta.