[Sy0307]

Purpose

Enable Higgs Audio v3 streaming on Model Runner V2 with model-owned delayed-codebook sampling, request state and owned audio outputs. The opt-in H200 profile uses asynchronous scheduling, FULL backbone graphs, sampler graphs and exact codec graph shapes.

Protect the FULL mixed-prefill capture query bound. An evenly split dummy batch can capture a bound smaller than a later long clone prompt. OmniModelState.prepare_attn uses the token bucket for unconstrained capture, preserving explicit bounds and runtime metadata. Real FA3 capture/replay now matches eager. Reverting the fix fails the explicit bound assertion, but does not reproduce the historical 81.6-second output numerically on the current backend; that historical causal attribution remains unproven. The generation limit is unchanged.

The generic runner retains the default path for models without the new optional model-state/sampler hooks. The current Higgs path supports temperature/top-k/top-p and rejects unsupported text penalties, logprobs, grammar and speculative sampling. The current sampler retains fill_exponential_rows for per-request seeded exponential noise in eager and graph sampling paths. The model also declares GPU-resident audio codes for deferred prefix-cache handling. Historical measurements below describe their recorded source snapshot, not a fresh qualification of this merged revision. The legacy and native whole-utterance adapters now share their codec conversion.

Test Plan

• Base: 817f5d0e4d12751020a2ed8d7886b821eb605467, the latest main fetched before this comparison. Measurements were collected at 0e679d523d419470bacd6046d9ceb6c4e151a589; the subsequent scope cleanup is described below.
• One H200, physical GPU 6; vLLM 0.30.0, PyTorch 2.13.0 / CUDA 13.0, Transformers 5.14.1, Triton 3.7.1.
• Same Higgs v3 4B weights, full Seed-TTS EN1088, streaming plain TTS. Both stages capped at 64, prefix caching off, MPS on, same configured stage memory fractions and CPU affinity.
• C32/C64, 1088 timed requests each, 64 excluded warmup requests, three independent starts per side. Order: before/after, after/before, before/after. 13,056 timed requests; all succeeded.
• Seed 42, temperature 1, top-p 0.95, top-k 50, repetition penalty 1, max 2048 tokens. First payload is 8 frames / 320 ms / 15,360 bytes for every measured request; subsequent chunks 25 frames, left context 8, right holdback 0.
• Baseline retains main's V1 eager, synchronous FLASHINFER path (use_trtllm_attention=false), with capacity and output protocol normalized. Candidate uses the submitted MRV2 profile. This is a complete code/configuration comparison, not isolated kernel attribution or a claim of identical output quality.

Test Result

vLLM-Omni before / after

Concurrency	Version	Audio-s/s	Requests/s	Mean first audio (ms)	P95 first audio (ms)	Mean completion (s)
C32	Main / V1	46.39	10.77	409.60	467.98	2.714
C32	PR / MRV2	197.88	45.44	118.08	135.48	0.692
C64	Main / V1	71.89	16.50	551.19	692.60	3.563
C64	PR / MRV2	277.86	63.71	168.89	253.40	0.973

• C32: throughput +326.6%, mean first audio -71.2%, P95 first audio -71.0%. Individual-start audio-s/s: main 50.21, 40.88, 49.41; PR 197.52, 197.35, 198.77.
• C64: throughput +286.5%, mean first audio -69.4%, P95 first audio -63.4%. Individual-start audio-s/s: main 75.54, 67.28, 73.41; PR 279.47, 273.87, 280.32.

Throughput pools emitted audio duration over measured wall time; latency pools all request records. All planned rounds are retained. Baseline round 2 produced two unusually long outputs (61.64 s at C32 and 30.12 s at C64), which affect wall time and widen its throughput range. They remain in the performance results and were separately scored; no failed-quality output was silently discarded. The other two paired throughput improvements were 293.4%/302.3% at C32 and 269.9%/281.9% at C64, so the direction is not dependent on that round. These are repeated short runs, not a long-duration soak test.

All timed 1 Hz telemetry samples reported SM/memory clocks of 1980/3201 MHz. Sampled peak device memory varied from 92.3 to 137.0 GiB on the baseline and 94.6 to 95.4 GiB on the candidate across windows; these are device samples, not allocator-level peak attribution, and no isolated memory-saving claim is made.

Historical quality and clone regression

The prespecified first paired start was fully scored: 1088 texts per side and concurrency, 4,352 scored WAVs, zero skips, same Whisper large-v3 and official Seed-TTS normalization. Confidence intervals use 10,000 paired resamples by text. Later performance repetitions are not counted as additional quality evidence.

Concurrency	Main WER	PR WER	Paired difference 95% CI (percentage points)
C32	7.238%	1.818%	[-6.174, -4.685]
C64	12.671%	1.759%	[-11.974, -9.805]

The baseline has appreciable recognition errors under this high-concurrency configuration. The table therefore describes observed end-to-end behavior, not an equal-quality microbenchmark; do not interpret the speedup as solely arithmetic efficiency. WER does not establish speaker similarity or perceptual equivalence.

• Clone capture-fix validation: full EN1088 at C1 and C32, 1088/1088 successful requests each, zero 81.6-second outputs, maxima 8.92/8.88 seconds. All outputs scored with no skips; official WER 1.552%/1.668%.
• Same original 128 C1 rows: WER 25.380% -> 0.784%. All 14 historical long outputs now last 2.32-8.92 seconds.
• The historical measured tree repeated clone C1/C32 with 128 rows each, covering all 14 historical failures: all 256 succeeded, zero long outputs or scoring skips, WER 0.784%/0.899%.
• A short C32 large-v3 ASR outlier remained in the aggregate. A different ASR model, Whisper-small without a target-text prompt, recovered its complete correct sentence from the same saved WAV. We did not replace the unfavorable original score.
• Shadow prefill numerical check: minimum observed cosine similarity 0.07891 -> 0.999981. Shadow execution changes KV state and is used only as numerical evidence, never for throughput or quality claims.
• Historical regression suite: 278 passed, 19 warnings in 7.78s. The six new capture-bound cases give 3 failures / 3 passes without the fix and all pass with it.
• Applicable pre-commit checks passed except mypy, which reports the same 25 production and 41 test diagnostics as the clean base, with zero additional diagnostics. This is not reported as a clean mypy pass. No new model-specific Python example was added.

Separate external reference

Historical external reference, same model/protocol	C32 / C64 audio-s/s	C32 / C64 mean first audio (ms)	C32 / C64 P95 first audio (ms)
Other framework, revision 7dc8909e78534a07400e86f781a969437ecc70e3, three independent starts	127.58 / 161.02	155.57 / 240.70	183.40 / 286.19

This is a separate historical reference, not the "before" denominator above and not a fresh interleaved comparison against this final tree. In particular, not every tail-latency metric improves by 20% against that reference. The primary comparison in this PR is main versus this change.

Scope cleanup validation

The cleanup removes unused experimental RNG/seed handling and the opt-in
scheduler-level deferred-code cache, while retaining standard whole-utterance
payload delivery. It shares whole-utterance codec conversion and adds independent
expected-output coverage for short rows, negative/special codes and tail trimming.
Sampler graph parity checks, output ownership copies and streaming window logic
remain intact. CPU regression: 286 passed, 2 CUDA cases deselected. The first
unfiltered run failed those two CUDA cases because GPU visibility was disabled;
the marker-selected CPU run passed. No new mypy diagnostics were introduced.

Performance and audio-quality numbers above remain measurements of 0e679d5,
not a fresh GPU benchmark of the cleanup commit. Shared GPUs were occupied, so
no GPU integration or performance rerun is claimed for this cleanup.

Step-overhead reductions (c922f2a)

At C64, profiles of the MRV2 decode step showed small eager launches and
per-request host work around the captured backbone and sampler graphs. After
this revision a steady-state step is ~5.9 ms with ~4.75 ms of GPU kernel time
(~81% busy), so the stage is now GPU-bound. This revision removes that overhead without
changing the sampling contract:

• Sampler-graph replay writes live rows into the captured static inputs
  (parameters copied only when metadata changes, noise drawn straight into
  the graph input) instead of padding through temporaries on every call.
• Seeded per-request exponential noise is drawn in one Triton launch that is
  bit-identical to torch's per-generator exponential_() (values and
  generator offsets are tested equal), instead of one launch per request.
• Terminal-row restore, prefill reset and audio-tail validation (~20 eager
  kernels) and pure-decode token/feedback embeddings are one launch each
  (tested equal to the eager code).
• Audio-mode step classification is vectorized over per-slot prompt facts;
  async-chunk codec payloads for a step use one host conversion per row
  (tested equal to the per-request builder); output rows use one
  torch.split; the custom-process signature check is cached.

No flag or deployment configuration changes.

Same H200 (physical GPU 7), back to back, C64 streaming, EN1088, 3 rounds each	Audio-s/s	Per round	Mean first audio (ms)	P95 first audio (ms)	Mean completion (s)
Previous head b63b5d0	278.68	273.25 / 279.81 / 282.99	166.0	240.4	0.968
This revision	329.59	316.58 / 337.02 / 335.18	162.2	269.3	0.811

Throughput +18.3%, mean completion -16.2%; 6,528 timed requests, all
succeeded. P95 first audio rose in this pair (240 -> 269 ms); earlier
repetitions of the same code measured 237-310 ms, so no P95 improvement is
claimed. Quality (round 2 of each side, Whisper large-v3, official
normalization): 1,088/1,088 scored per side, no skips, zero outputs over 20 s; WER 1.828% -> 1.789% (not a claimed quality change).

Tests: Higgs model, adapter, stage-input-processor, worker_v2 and omni-connector suites on the H200 (CUDA cases included): 1,307 passed, 34 skipped (plus the new async-chunk parity test); the 3 failures are Bagel connector tests that need network access / mooncake libraries and fail identically on b63b5d0. Changed-file pre-commit hooks pass except mypy, which reports 2 existing str | None diagnostics at omni_connector_payload_transport.py:1112/1120 (unchanged lines); no new diagnostics.

Repository scope

Benchmark reports, raw evidence JSON, the matched V1 benchmark configuration and
reference-client bridge are excluded from this PR diff and retained as local
experiment artifacts. The results and limitations are summarized in this PR
body. The production MRV2 deployment configuration and regression tests remain.

Three implementation flags (audio_sampler_no_sync, audio_mixed_direct_sampling,
audio_skip_text_head) have been removed; existing eligibility guards select
those Higgs paths automatically. Sampler/codec CUDA graph overrides remain
optional: MRV2 defaults to enabled, V1 defaults to disabled, and an explicit
true/false overrides either default without mutating checkpoint configuration. The capture query-bound fix is unconditional for unconstrained
capture because correctness must not depend on a performance flag. It repairs
the mismatch between the replay domain and the captured attention bound, while
preserving actual sequence boundaries and the generation limit. This Omni
compatibility fix addresses behavior in the vLLM 0.30 capture path; an upstream
fix is the preferable long-term home. No GPU or quality rerun is claimed for
this artifact-only removal.

Default selection and compatibility

The H200 YAML no longer repeats the five optimization switches. Only Higgs
sampler/codec graph overrides remain. The shared V1 runner uses a model
capability (requires_cpu_input_tail_ids, declared by Higgs only) instead of
reading a Higgs optimization flag; models without that capability retain the
previous path. Automatic sampling/text-head optimizations can also apply to
eligible Higgs V1 batches, so this is not a claim that Higgs V1 execution is
unchanged. No default or implementation is changed in other model classes.

Validation for this change: 307 CPU tests passed, 2 CUDA tests deselected;
all applicable changed-file pre-commit hooks passed, including mypy. Tests cover
runner-dependent graph defaults, explicit overrides, shared-config preservation,
text-head fallback and capability-gated CPU tail metadata. No fresh GPU audio
quality or performance result is claimed for this revision.

Earlier review fixes and validation (2026-09-29)

Historical validation head 0e74cb0feb19b5f0401b135affa7dbb027485d8a, runtime change ee97d6f8c, stacked on shared prerequisite e4689f3a8 (#8184). Historical performance figures above have not been remeasured on this revision.

Removed unused V1 prompt classification from the shared runner, preallocated stable decode state, removed stdout diagnostics and clarified unsupported configuration errors.

Fixed in ee97d6f8c; that validation used head 0e74cb0fe, which also merged the documentation-only shared prerequisite. OmniGPUWorkerBase.compile_or_warm_up_model invokes an optional model-owned capture_auxiliary_graphs after normal engine warmup and before readiness. The MRv2 generation worker invokes the same helper after its generation warmup. This covers V1 opt-in as well as MRv2; models without the method retain their existing path.

Higgs warms both forced-audio modes and greedy/random sampler specializations for configured request buckets, restores device RNG and decode state, and clears temporary output snapshots. Codec capture uses exact batch/frame shapes. Capture time and allocated-memory delta are logged. Serving cache misses take the ordinary eager path and never create new graphs. Startup capture errors propagate and prevent readiness rather than silently moving work into the first request.

On physical H200 GPU 0, the final source captured 44 sampler graphs (0.46 s with compiler caches already populated; 96.5 MiB allocated delta) and 159 codec graphs (15.01 s; 201.8 MiB) before API startup complete. An earlier fresh compiler-cache run took 15.44/17.91 s respectively. These are capture-time observations, not steady-state TTFP or peak-memory measurements. The final real-checkpoint E2E passed six requests, including four concurrent streams and a true nonstream request; abort-after-first-audio followed by another request also passed. H200 regression suite: 338 passed. Graph prewarming does not eliminate cold reference-encoder or other backend initialization.

Real attention backend coverage and red/green evidence

Replaced the mock-only capture-bound test with a Hopper FA3 integration test. It runs OmniModelState.prepare_attn through the real upstream metadata builder and FlashAttentionImpl, captures a FULL graph from 256 tokens distributed over 64 requests, then replays a 212-token prefill with ragged/padded companions and compares output against eager. The test passes on H200.

Reverting to DefaultModelState.prepare_attn makes the test fail because capture retains max_query_len=4 instead of 256. The failure is the explicit launch-bound assertion, not a numerical mismatch. Separate probes with the current vLLM 0.30 FA3 backend did not reproduce the historical 81.6-second audio failure, even with the bound reverted. Consequently this closes the real-backend execution coverage gap, but is not the requested numerical red/green reproduction of the historical incident. That causal claim remains unproven on this backend/version and must not be overstated.

The local guard mirrors upstream's capture-time worst-case max_seq_len treatment for the query axis and preserves runtime/explicit bounds. An upstream change would require corresponding backend coverage; this PR does not claim that an upstream fix has been submitted.

Quality review

Final-runtime paired evaluation: full repository Seed-TTS EN1088 voice cloning at C8, seeds 42 and 43, matched weights, text/reference order, codec windows, temperature 1, top-k 50, top-p 0.95 and repetition penalty 1. All 4,352 requests completed and were scored, with no skipped utterances.

Seed	V1 mean utterance WER	MRv2 WER	Difference (pp)	Paired 95% interval (pp)
42	1.5792%	1.6615%	+0.0823	[-0.1300, +0.3048]
43	1.7461%	1.7648%	+0.0187	[-0.1973, +0.2444]
Combined	1.6626%	1.7132%	+0.0505	[-0.1133, +0.2260]

Combined bootstrap resamples utterances keeping both seeds together. Mean WavLM cosine was 0.879397 / 0.879554 (V1 / MRv2); paired difference interval [-0.000423, +0.000760]. This is a WavLM proxy, not official UniSpeech SV. No output reached 80 seconds; the maximum across arms was 9.88 seconds. This comparison does not establish a statistically significant regression, but two seeds with one run per arm/seed do not prove equivalence or zero quality cost. WER is mean utterance WER, not pooled token WER. These replace the earlier prototype seed-42 result. Inference used physical H200 GPU 0/2 and scoring GPU 1/3; GPU 4/5 were not used.

Unified sampler contract and topology

Higgs and MiniCPM now use the same ModelState.custom_sampler registration path and OmniSampler adapter. The separate Higgs sample_omni_output hook is removed. HiggsSampler.sample_step returns OmniSamplingOutput with sampled/rejected counts, model-owned audio payload, hidden-state requirements and the CPU-snapshot finalizer; ordinary samplers and synthetic warmup requests retain standard sampling. The runner no longer looks up a model-specific finalization hook. The shared contract and its tests landed on main through MiniCPM #8222. This branch now merges main 5af3a1652, so the shared implementation is inherited rather than repeated in the Higgs diff; the original staged-write fast path is preserved.

The unused V1 model-specific prompt classification was removed in the preceding fix. A separate V1 Talker + MRv2 codec deployment previously passed six real-model E2E requests, providing a concrete stage-migration example for #8184.

Sampler-refactor validation (2026-09-30)

Historical validation head: 3952eda8a, with vLLM 0.30.0, after merging main including MiniCPM #8222.

• Final integrated Higgs, MiniCPM, mainline MOSS per-row sampling, forward-metadata and worker_v2 regressions: 1136 passed, 9 skipped, 22 warnings. The earlier sampler-only validation passed 282 tests; these suites overlap and the counts are not additive.
• Final-head real Higgs checkpoint on physical H200 GPU 3: 6/6 Seed-TTS EN functional E2E requests passed (one cold stream, four concurrent streams, one true nonstream response). All saved audio was decodable, non-silent and finite.
• Changed-file pre-commit and whitespace checks passed; all new commits are signed off. This round also corrects two existing worker acknowledgement return annotations to include the None result on non-primary ranks, without changing runtime behavior.
• Previous head 20be501f9 passed CUDA Buildkite 16364: https://buildkite.com/vllm/vllm-omni/builds/16364. Final-head CUDA Buildkite 16367 passed all 19 executed steps (3 ignored, 0 failed): https://buildkite.com/vllm/vllm-omni/builds/16367. Its commit_id matches 3952eda8a; final-head DCO passed.

These probes validate the sampler refactor's functionality; they do not repeat the full WER/speaker evaluation above or establish new performance gains. The historical numerical red/green reproduction limitation remains as documented. The three fixed inline threads have been replied to and resolved, and follow-ups have been posted to both reviewers. The historical numerical red/green reproduction remains unproven; explicit non-chunk deployment validation is completed in the follow-up below. Reviewer approval is still pending.

Latest review follow-up: full-response compatibility (5daf4d888)

Based on main e34f8c4f4, with vLLM 0.30.0. Fixes the V1 async_chunk: false regression reported in #8226 (review).

The terminal/invalid post-sample audio row previously had shape [0], which made the full-payload accumulator replace all earlier [N, 8] codec rows with an empty value. Both synchronous and asynchronous snapshot exports now preserve shape [0, 8], leaving prior rows intact. The sync input adapter also accepts the runtime MultimodalPayload Mapping rather than only dict. This retains V1 support and the existing full-payload connector; default profiles and sampling parameters are unchanged.

Validation on physical H200 GPUs 3 and 6 (GPUs 4/5 excluded):

Deployment	Requests	Checks
V1, async_chunk: false	6/6	Fresh server; single + four concurrent + nonstream WAV
V1, async_chunk: true	6/6	Fresh server; single + four concurrent PCM streams + nonstream WAV
MRv2, async_chunk: false	6/6	Fresh server; single + four concurrent + nonstream WAV
MRv2, async_chunk: true	6/6	Fresh server; single + four concurrent PCM streams + nonstream WAV

All saved audio is decodable, finite and non-silent. The explicit non-chunk deployments disable deployment-level chunking, independently of the HTTP stream flag. These Seed-TTS EN samples are functional smoke coverage, not a new full quality or performance evaluation.

• Related model, stage-input and forward-metadata regressions: 225 passed. The new terminal/invalid-row and Mapping regressions fail when the two production fixes are reverted (17 failed / 16 passed in the selected control suite).
• Permanent V1 non-chunk E2E: 1 passed both locally on H200 and in CUDA CI on H100, comprising three concurrent requests; wired into the ready CUDA pipeline as TTS · Higgs v3 Full Response Test on one H100. It uses --no-async-chunk on the default V1 deployment.
• Correction: the initial production-file pre-commit run failed on two overlong memory-log strings, despite the subsequent test-file-only run passing. Commit eea9b2061 splits these strings without changing the AST or log contents. The CI-equivalent whole-tree pre-commit scan and all applicable changed-file hooks (including mypy), Buildkite YAML validation, whitespace and signed-off identity checks now pass locally; refreshed GitHub checks are pending.
• Sampler and codec capture logs now include allocated delta, reserved delta and reserved total. On the MRv2 streaming H200 start, the codec reserved delta was 8224 MiB, versus 201.8 MiB allocated. The H200 YAML now calls for at least 10 GiB of additional free headroom for codec graph pools after KV sizing; this is profile-specific, not a universal peak-memory bound.
• The profile documents that batch-wide disabled top-k/top-p produces None metadata and falls back to eager sampling; the default tensor-valued 50/0.95 path retains warmed graphs. No additional graphs or runtime recapture were added.

CUDA Buildkite 16404 passed: 20 executed steps passed, 3 ignored, 0 failed. Its commit_id matches 5daf4d888c092046a692b94c382e1a68f3a4a814: https://buildkite.com/vllm/vllm-omni/builds/16404. This includes the new H100 full-response E2E, Model Executor, Engine&Model Executor and Qwen3-TTS compatibility jobs. DCO also passed. All three new inline discussions have English replies and are resolved; linyueqian approved the reviewed head on 2026-09-30; approval remains in place after the string-only fix. Historical attention-bound numerical red/green reproduction remains unproven on the current backend, as described above.

CUDA CI fixture follow-up (2026-09-30)

The last failing CUDA run (16359, head 3e44114ce) failed test_cfm_graph_replay_matches_eager_for_uncached_and_cached_shapes in the MiniCPM regression suite. Its _MiniDiT fixture wrote only the current-chunk prefix of a torch.empty attention-output buffer, then compared the unwritten old-cache tail as well. Later runs were green, but the latent uninitialized-memory comparison remained.

Commit ce609c763 makes the fixture write the old-cache tail, matching the documented current-first cache layout. The regression now poisons output buffers with NaNs and checks full eager/graph parity, finiteness and preservation of the old cache, without relaxing tolerances or changing production code. On physical H200 GPU 6 (UUID GPU-62d67334-2671-f5f0-76f8-a9c784c8544c), reverting the fixture fix while retaining NaN poisoning deterministically fails on all 160 unwritten tail elements; the corrected file passes 41 tests with 4 skips, and the targeted test passes again in an independent process. All applicable changed-file hooks and the CI-equivalent whole-tree pre-commit scan pass locally. Refreshed checks for ce609c763 are pending.

Historical failure: https://buildkite.com/vllm/vllm-omni/builds/16359

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/ar_runtime.md, docs/design/module/model_integration.md.

Module owners: @tzhouam @fake0fan @Gaohan123

Routing: @tzhouam via module of the changed files, semantic router, CODEOWNERS; @fake0fan via module of the changed files; @Gaohan123 via module of the changed files

@Sy0307, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit ce609c763b89 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot: three questions on the performance claim

@Sy0307 this PR reads as a performance or value claim:

• claim: The other two paired throughput improvements were 293.4%/302.3% at C32 and 269.9%/281.9% at C64, so the direction is not dependent on that round.

Before the full evidence checklist, three short questions:

1. Bottleneck — what is the current bottleneck, and which profile, trace or per-stage measurement shows it?
2. Value — what does the change buy the user or the system (latency, throughput, memory, cost), and at which workload?
3. A/B or ablation — is there a same-workload, same-head/config comparison that isolates each main claim on its own? For stacked optimizations, one number per item rather than a blended delta.

When you answer, the evidence that settles it is: base and head SHA, hardware, model, workload, warm-up and repeat count, mean or percentiles with their spread, and a correctness/quality-equivalence signal; an end-to-end claim also needs stage attribution.

[Sy0307]

@BeatSeat, the new H200 FA3 FULL-graph test covers 256-token/64-request capture, a 212-token prefill replay and comparison against eager; reverting the fix fails the launch-bound assertion, while the historical numerical mismatch remains unreproduced on vLLM 0.30.0. Sampler and codec graphs now capture before readiness, with capture-time and allocated-memory logging and no serving-path recapture. Could you recheck this coverage and advise on the remaining numerical reproduction; the bound fix is still Omni-local, with no upstream patch claimed?

[Sy0307]

@linyueqian, the V1 runner no longer contains Higgs-specific prompt classification, sampler/codec capture now runs before readiness, and the config error points to the H200 profile. The sampler hooks are unified; final-head regressions, real-model E2E and CUDA build 16367 passed. Could you re-review these fixes; I am leaving explicit async_chunk: false deployment validation open because the streaming-profile nonstream HTTP test does not establish it?

[linyueqian]

Thanks for the quick turnaround. I re-ran this head (3952eda) on one H200 against main 5af3a16, fresh server per condition, 3 runs per cell. The three things I raised are fixed:

• The shared V1 runner no longer carries Higgs logic; it only forwards capability-gated CPU tail ids.
• Sampler and codec graphs are captured once before /health (44 sampler and 159 codec graphs on each of 3 starts, none while serving; a counter-instrumented copy saw every serving-path sample replay a captured graph). The first request after ready is now 2.2 s to first audio, down from 39 s, and the parity print is gone.
• The MRv2 error on the default YAML now points at the H200 profile.

MRv2 also got faster since my last run: c1 8.09 audio-s/s at 75 ms TTFP, c8 56.0 audio-s/s at 81 ms (was 40.3 and 109 ms). V1 on the default YAML matches main (c1 2.16 vs 2.25, c8 13.3 vs 13.1 audio-s/s) and tests/e2e/online_serving/test_higgs_audio_v3.py passes 12/12. All 13 touched test files pass on GPU (195/195). Qwen3-TTS on its MRv2 profile shows no regression (c8 51.6 vs 53.3 audio-s/s, WER 1.16% on both).

On @BeatSeat's WER question: the earlier 1.45% vs 1.16% was a 32-item sample. On all 1088 Seed-TTS EN items at c8 with two seeds, MRv2 minus V1 is -0.12 pp with a paired bootstrap 95% CI of [-0.29, +0.04], so there is no quality gap.

Requesting changes for one new regression, inline: with async_chunk: false on V1, every request now returns HTTP 200 with empty audio and no error, while main returns audio for the same config. The PR description already lists this deployment as not validated, so it just needs the fix or an explicit guard before merge. Two smaller notes inline as well.

[linyueqian]

Thanks, 5daf4d8 fixes it. I re-ran the head on one H200 against main 004a571, fresh server per condition.

• V1 with async_chunk: false now returns full audio for streaming and non-streaming (8/8 each, WER 1.12%, same as main), and 64/64 at c8 in both modes with WER in line with main (paired bootstrap p=0.27 streaming, 0.74 non-streaming). The non-chunk outputs now have the same sizes as the default async_chunk path.
• Regression smoke: V1 default 8/8 at WER 1.12%, and 64/64 at c8 over three fresh runs (1.74 / 0.72 / 1.01% vs main 2.17 / 1.16 / 1.01%). MRv2 H200 profile 8/8 at WER 1.12%, 64/64 at c8 with 1.16%, first packet p50 about 82 ms, 46.3 audio-s/s.
• The new reserved-memory log lines are useful: the codec capture shows about 8.0 GiB reserved against 0.2 GiB allocated, which matches the headroom note in the H200 YAML.
• tests/model_executor/models/higgs_audio_v3/ passes on GPU, 212/212, including the new terminal and invalid row test.

One small thing to keep an eye on, not blocking: in the first cold wave of c8 V1 streaming, "The primary coil has fifty turns." picked up a few extra trailing words on this head in two runs, which I did not see on main. Later waves were clean, so this may just be warmup noise.

Looks good to me.