[Model] Make Higgs Audio v3 run Model Runner V2 by default

docs/configuration/stage_configs.md

@@ -571,6 +571,37 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/depl
 !!! important
     We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
 
+## Higgs-Audio V3 with Model Runner V2
+
+Higgs-Audio V3 runs `model_runner: v2` by default. The MRV2 feature switches
+(`audio_mrv2_*`, `audio_async_*`) are tri-state fields of `HiggsAudioV3Config`
+that self-enable from the runner selection, so a deploy yaml needs only the
+`model_runner` key — the explicit `hf_overrides` block the old opt-in profile
+carried is now a tuning override, not a requirement. `audio_async_payload` and
+`audio_async_prompt_mode` are structural under v2 and cannot be disabled.
+
+Select one of these deployment profiles:
+
+| Profile | Runner | Intended use |
+| --- | --- | --- |
+| `higgs_multimodal_qwen3.yaml` | V2 (default) | Shipped default; H200-validated shape scaled to 16 sequences |
+| `higgs_multimodal_qwen3_mrv2_h200.yaml` | V2 | Large-card tuning (64-sequence capacities, MPS) |
+| `higgs_multimodal_qwen3_high_throughput.yaml` | V1 | V1 opt-out: eager Stage 0 + local MLP CUDA graph |
+| `higgs_multimodal_qwen3_low_latency.yaml` | V1 | V1 opt-out: `FULL_DECODE_ONLY` external decode graph |
+
+The default's `platforms:` section keeps V1 on NPU, XPU, ROCm and MUSA
+(XPU rejects any v2 stage at deploy load; NPU generation workers — Higgs
+Stage 1 — reject v2 at init). The V2 shape pins the connector streaming
+geometry (25-frame chunks, 8-frame left context, 8-frame first chunk) that
+the Stage-1 codec CUDA graph windows are sized against.
+
+Sampling surface under v2: the talker's self-owned sampler supports
+`temperature`/`top_p`/`top_k`/`seed` only. Penalties, logprobs, min_tokens,
+min_p, allowed/bad token ids, and logit_bias are text-generation machinery
+the captured sampling step does not wire in; a non-neutral unsupported value
+in `default_sampling_params` fails at startup, and per-request use raises.
+Remove the parameter or run the stage with `model_runner: v1`.
+
 ## Qwen3-TTS with Model Runner V2
 
 Qwen3-TTS runs the native CUDA Model Runner V2 pipeline by default on vLLM

recipes/BosonAI/Higgs-Audio-V3-TTS.md

@@ -108,29 +108,33 @@ python examples/online_serving/text_to_speech/higgs_audio_v3/batch_speech_client
   and reference code row count without logging text or audio content. They do not
   change the existing fallback behavior; legacy counts describe the current span.
 - Deploy config: `vllm_omni/deploy/higgs_multimodal_qwen3.yaml` (auto-discovered from `model_type`).
+    - Runs `model_runner: v2` by default; the MRV2 feature switches self-enable from the runner selection (no `hf_overrides` needed).
     - `max_num_seqs=16` for both stages.
     - Stage 0 and Stage 1 default to the same device (`0`) for single-GPU serving.
-    - Stage 0 intentionally keeps `enforce_eager=true`. This preserves the Higgs-specific local MLP CUDA graph path, which is the current high-throughput default.
-    - Stage 1 remains `enforce_eager=true` for the codec decoder.
+    - Stage 0 is non-eager with `FULL` CUDA graph capture and `async_scheduling: true` (the validated MRV2 shape, scaled from the H200 profile); Stage 1 remains `enforce_eager=true` with stage-local codec CUDA graphs.
+    - Streaming geometry: 25-frame chunks with 8-frame left context; the first chunk is one 8-frame codec-graph window (~320 ms). The V1 profiles ship a 1-frame first chunk.
+    - Sampling under v2 supports `temperature`/`top_p`/`top_k`/`seed` only; penalties, logprobs, min_tokens, min_p, allowed/bad token ids, and logit_bias require `model_runner: v1`.
+    - Non-CUDA platforms (NPU/XPU/ROCm/MUSA) fall back to `model_runner: v1` through the file's `platforms:` section.
 - Deploy profiles:
-    - High throughput: `vllm_omni/deploy/higgs_multimodal_qwen3_high_throughput.yaml`.
-    Use this for medium/high concurrency. The auto-discovered
-    `higgs_multimodal_qwen3.yaml` is kept as a compatibility/default alias for
-    this profile.
-    - Low latency: `vllm_omni/deploy/higgs_multimodal_qwen3_low_latency.yaml`.
-    Use this for low-concurrency serving (for example c1-c4). It sets Stage 0
+    - MRV2 default: `vllm_omni/deploy/higgs_multimodal_qwen3.yaml` (the auto-discovered default above).
+    - High throughput (V1 opt-out): `vllm_omni/deploy/higgs_multimodal_qwen3_high_throughput.yaml`.
+    Use this for medium/high concurrency on the V1 runner; Stage 0 stays eager
+    to keep the Higgs-specific local MLP CUDA graph path.
+    - Low latency (V1 opt-out): `vllm_omni/deploy/higgs_multimodal_qwen3_low_latency.yaml`.
+    Use this for low-concurrency V1 serving (for example c1-c4). It sets Stage 0
     `enforce_eager=false` and enables vLLM `FULL_DECODE_ONLY` CUDA graph through
     YAML `compilation_config`; no environment variable is required.
+    - H200-tuned MRV2: `vllm_omni/deploy/higgs_multimodal_qwen3_mrv2_h200.yaml` (64-sequence capacities, larger capture sizes, MPS).
     - Profile details: `vllm_omni/deploy/README_higgs_audio_v3.md`.
-- Performance note:
-    - Do not switch the auto-discovered default Stage 0 profile to vLLM
-    `FULL_DECODE_ONLY` CUDA graph without an end-to-end throughput and
-    audio-quality revalidation. On the H20 SeedTTS c16/full-dataset benchmark,
-    the eager Stage 0 path with Higgs local MLP CUDA graph reproduced
-    ~35 audio_s/s (`1088/1088` OK per run, three runs around 134-136s). A
-    separate H20 low-concurrency smoke after the FULL_DECODE audio-feedback
-    capture fix showed lower c1/c4 latency, so the FULL_DECODE profile is kept
-    as an explicit low-latency option rather than the throughput default.
+- Performance note (V1 opt-out profiles):
+    - Do not switch a V1 opt-out Stage 0 profile to vLLM `FULL_DECODE_ONLY`
+    CUDA graph without an end-to-end throughput and audio-quality
+    revalidation. On the H20 SeedTTS c16/full-dataset benchmark, the eager
+    Stage 0 path with Higgs local MLP CUDA graph reproduced ~35 audio_s/s
+    (`1088/1088` OK per run, three runs around 134-136s). A separate H20
+    low-concurrency smoke after the FULL_DECODE audio-feedback capture fix
+    showed lower c1/c4 latency, so the FULL_DECODE profile is kept as the
+    explicit low-latency option rather than the throughput pick.
 - Known limitations:
     - Stage 1 (code2wav) must use `enforce_eager=true` (`@torch.inference_mode` incompatible with graph capture).
-    - Stage 0 full-decode CUDA graph is experimental; sampler, delay-state updates, staging, and request postprocess remain outside the graph.
+    - On the V1 low-latency profile, the Stage 0 full-decode CUDA graph is experimental; sampler, delay-state updates, staging, and request postprocess remain outside the graph.

tests/config/test_config_factory.py

@@ -3082,7 +3082,8 @@ def test_higgs_audio_v3_rocm_uses_triton_attention(self):
         deploy_path = Path(get_deploy_config_path("higgs_multimodal_qwen3.yaml"))
 
         base = load_deploy_config(deploy_path)
-        assert base.stages[0].attention_backend == "FLASHINFER"
+        assert base.model_runner == "v2"
+        assert base.stages[0].attention_backend == "FLASH_ATTN"
 
         rocm = _apply_platform_overrides(base, platform="rocm")
         assert rocm.stages[0].attention_backend == "TRITON_ATTN"
@@ -3092,6 +3093,147 @@ def test_higgs_audio_v3_rocm_uses_triton_attention(self):
         stages = merge_pipeline_deploy(pipeline, rocm)
         assert stages[0].yaml_engine_args["attention_backend"] == "TRITON_ATTN"
 
+    def test_higgs_audio_v3_non_cuda_platforms_fall_back_to_v1(self):
+        """Non-CUDA platforms must keep the V1 runner after the MRV2 flip.
+
+        XPU rejects any v2 stage at deploy load, NPU generation workers
+        (Higgs stage 1) reject v2 at init, and ROCm/MUSA v2 is Phase 3
+        (#9387). The platforms section pins v1 and restores the shipped V1
+        stage-0 shape (eager, no async scheduling, prefix caching) and the
+        shipped V1 connector streaming geometry, so the fallback stays
+        valid when this file's base flips to model_runner: v2.
+        """
+        deploy_path = Path(get_deploy_config_path("higgs_multimodal_qwen3.yaml"))
+
+        for platform in ("rocm", "npu", "xpu", "musa"):
+            deploy = _apply_platform_overrides(load_deploy_config(deploy_path), platform=platform)
+            assert deploy.model_runner == "v1", platform
+            assert all(stage.model_runner is None for stage in deploy.stages), platform
+            stage0 = deploy.stages[0]
+            assert stage0.enforce_eager is True, platform
+            assert stage0.async_scheduling is False, platform
+            assert stage0.engine_extras["enable_prefix_caching"] is True, platform
+            # The connector restore keeps the shipped V1 streaming geometry:
+            # the v2 chunk shape exists only for the codec CUDA-graph windows.
+            extra = deploy.connectors["connector_of_shared_memory"]["extra"]
+            assert extra["codec_left_context_frames"] == 25, platform
+            assert extra["codec_right_holdback_frames"] == 4, platform
+            assert extra["initial_codec_chunk_frames"] == 1, platform
+            # The restore is a deep merge: untouched extra keys survive.
+            assert extra["codec_chunk_frames"] == 25, platform
+            assert extra["codec_streaming"] is True, platform
+
+        # The yaml anchor must resolve to independent containers: mutate the
+        # rocm entry of one parse and the xpu entry of the same parse must
+        # not change.
+        from vllm_omni.config.stage_config import resolve_deploy_yaml
+
+        raw = resolve_deploy_yaml(deploy_path)
+        assert raw["platforms"]["rocm"]["connectors"] is not raw["platforms"]["xpu"]["connectors"]
+        raw_rocm = raw["platforms"]["rocm"]["connectors"]["connector_of_shared_memory"]["extra"]
+        raw_xpu = raw["platforms"]["xpu"]["connectors"]["connector_of_shared_memory"]["extra"]
+        raw_rocm["codec_left_context_frames"] = 99
+        assert raw_xpu["codec_left_context_frames"] == 25
+
+        # The v2 default path keeps the graph-coupled chunk shape.
+        cuda = _apply_platform_overrides(load_deploy_config(deploy_path), platform="cuda")
+        extra = cuda.connectors["connector_of_shared_memory"]["extra"]
+        assert extra["codec_left_context_frames"] == 8
+        assert extra["codec_right_holdback_frames"] == 0
+        assert extra["initial_codec_chunk_frames"] == 8
+
+        # A platform connector override must name a defined connector.
+        deploy = load_deploy_config(deploy_path)
+        deploy.platforms["rocm"]["connectors"] = {"no_such_connector": {"extra": {}}}
+        with pytest.raises(ValueError, match="no_such_connector"):
+            _apply_platform_overrides(deploy, platform="rocm")
+
+        # And each override must itself be a mapping.
+        deploy = load_deploy_config(deploy_path)
+        deploy.platforms["rocm"]["connectors"] = {"connector_of_shared_memory": None}
+        with pytest.raises(ValueError, match="must be a mapping"):
+            _apply_platform_overrides(deploy, platform="rocm")
+
+        rocm = _apply_platform_overrides(load_deploy_config(deploy_path), platform="rocm")
+        assert rocm.stages[0].attention_backend == "TRITON_ATTN"
+
+        xpu = _apply_platform_overrides(load_deploy_config(deploy_path), platform="xpu")
+        assert xpu.stages[0].attention_backend == "FLASH_ATTN"
+        assert xpu.stages[0].gpu_memory_utilization == 0.4
+        assert xpu.stages[1].gpu_memory_utilization == 0.15
+        assert xpu.stages[1].enforce_eager is True
+
+    def test_platform_connector_overrides_merge_with_base_config(self):
+        from vllm_omni.config.stage_config import _merge_platforms
+
+        base = {
+            "rocm": {
+                "model_runner": "v1",
+                "connectors": {
+                    "connector_of_shared_memory": {
+                        "extra": {"codec_left_context_frames": 25, "initial_codec_chunk_frames": 1}
+                    }
+                },
+            }
+        }
+        overlay = {"rocm": {"connectors": {"connector_of_shared_memory": {"extra": {"initial_codec_chunk_frames": 2}}}}}
+        merged = _merge_platforms(base, overlay)["rocm"]
+        assert merged["model_runner"] == "v1"
+        extra = merged["connectors"]["connector_of_shared_memory"]["extra"]
+        assert extra["codec_left_context_frames"] == 25
+        assert extra["initial_codec_chunk_frames"] == 2
+
+    def test_higgs_audio_v3_v2_rejects_unsupported_default_sampling_params(self):
+        """Non-neutral unsupported sampling defaults fail at startup under v2.
+
+        The v2 path's self-owned sampler supports temperature/top-k/top-p/seed
+        only; a yaml carrying e.g. repetition_penalty=1.1 must fail at deploy
+        merge (with the v1 opt-out named), not on every request. Neutral
+        engine defaults (repetition_penalty=1.0) pass, and falsy neutral
+        values (logprobs: null, bad_words: [], logit_bias: {}) pass too,
+        matching the request-path checks in model_state.py add_request.
+        """
+        deploy_path = Path(get_deploy_config_path("higgs_multimodal_qwen3.yaml"))
+        pipeline = resolve_pipeline_config("higgs_multimodal_qwen3")
+
+        # Shipped defaults are neutral: merge succeeds.
+        stages = merge_pipeline_deploy(pipeline, load_deploy_config(deploy_path))
+        assert all(stage.yaml_engine_args["use_v2_model_runner"] for stage in stages)
+
+        # Explicitly neutral None/collection values are no-ops, not failures.
+        deploy = load_deploy_config(deploy_path)
+        deploy.stages[0].default_sampling_params.update(
+            logprobs=None,
+            prompt_logprobs=None,
+            allowed_token_ids=None,
+            bad_words=[],
+            logit_bias={},
+        )
+        merge_pipeline_deploy(pipeline, deploy)
+
+        deploy = load_deploy_config(deploy_path)
+        deploy.stages[0].default_sampling_params["repetition_penalty"] = 1.1
+        with pytest.raises(ValueError, match="model_runner: v1"):
+            merge_pipeline_deploy(pipeline, deploy)
+
+        # Non-neutral collection values fail the same way as scalars.
+        deploy = load_deploy_config(deploy_path)
+        deploy.stages[0].default_sampling_params["logprobs"] = 5
+        with pytest.raises(ValueError, match="model_runner: v1"):
+            merge_pipeline_deploy(pipeline, deploy)
+
+        # 0 looks neutral but still requests a logprob; keep it rejected so
+        # startup matches add_request.
+        deploy = load_deploy_config(deploy_path)
+        deploy.stages[0].default_sampling_params["logprobs"] = 0
+        with pytest.raises(ValueError, match="model_runner: v1"):
+            merge_pipeline_deploy(pipeline, deploy)
+
+        # The same penalties are legal on the v1 opt-out path.
+        deploy_v1 = _apply_platform_overrides(load_deploy_config(deploy_path), platform="rocm")
+        deploy_v1.stages[0].default_sampling_params["repetition_penalty"] = 1.1
+        merge_pipeline_deploy(pipeline, deploy_v1)
+
     def test_qwen3_omni_cuda_uses_thinker_rotary_custom_op(self):
         deploy_path = Path(get_deploy_config_path("qwen3_omni_moe.yaml"))
         pipeline = resolve_pipeline_config(
@@ -3175,6 +3317,7 @@ def test_fish_speech_npu_uses_ascend_kv_block_size(self):
         [
             ("qwen3_tts_mrv2.yaml", "qwen3_tts"),
             ("moss_tts_local.yaml", "moss_tts_local"),
+            ("higgs_multimodal_qwen3.yaml", "higgs_multimodal_qwen3"),
         ],
     )
     @pytest.mark.parametrize("platform", ["cuda", "npu", "xpu", "rocm", "musa"])

tests/dfx/perf/tests/test_higgs_audio_v3.json

@@ -56,6 +56,22 @@
           }
         }
       },
+      {
+        "task": "default_voice",
+        "eval_phase": "latency",
+        "dataset_name": "seed-tts-text",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "benchmarks/build_dataset/seed_tts_smoke",
+        "num_prompts": [
+          100
+        ],
+        "max_concurrency": [
+          4
+        ],
+        "seed_tts_locale": "en",
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration"
+      },
       {
         "task": "default_voice",
         "eval_phase": "throughput",