[CI] Refresh Voxtral-4B-TTS perf baselines from a clean window

[linyueqian]

Summary

Nightly CI has reported a Voxtral-4B-TTS perf regression on H100 (#5967): audio_throughput -12.20% and median_audio_rtf -6.74%. It is not a regression. The baselines it compares against were averaged over a window that includes three nights when Voxtral output was broken, and the resulting audio_throughput target is unreachable by construction.

This refreshes all eight Voxtral H100 baselines in tests/dfx/perf/tests/test_tts.json to the plain mean of the 31 clean nightlies (2026-07-08 .. 2026-08-09). Metadata only, no runtime code changes.

Why the current baselines are wrong

#5231 set them as the plain mean of the seven nightlies 2026-07-05..07-11. All eight values reproduce exactly as that mean. Three of those nights sit inside a nine-night window when Voxtral emitted unrelated, over-long audio:

• [Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs #4527 moved the inter-stage codec feedback under codes.audio while tts_preprocess still read top-level audio, so decode lost its previous audio-code embeddings.
• Fix Voxtral TTS feedback and short ASR checks #4954 fixed it and closed [Bug]: Weekly CI, voxtral_tts, output audio doesn't match input #4949 ("output audio doesn't match input"). Its own comment in voxtral_tts.py names [Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs #4527.

mean_audio_duration_s, stress cell (request_rate=2.0, n=100):

night	06-27	06-28	06-29	06-30 .. 07-06	07-07	07-08	07-09
mean audio s	5.05	5.16	7.47	7.36 - 7.37	7.35	5.13	5.24

Boundary nightlies map to 1b318d11d..c45ac74d4 (contains #4527) and 7ab50965e..267b49b88 (heads with #4954). The anomaly is Voxtral-only: Qwen3-TTS Base/CustomVoice, VoxCPM2 and Higgs mean_audio_duration_s are flat across the same window.

The stress baseline is unreachable, not merely unobserved

100 prompts at request_rate=2.0 means 99 inter-arrival gaps, so the arrival span alone is 49.5 s and wall clock cannot be shorter (observed 50.63-50.96 s). At the correct 5.138 s/prompt, audio_throughput is capped at 100 x 5.138 / 49.5 = 10.380 even against an infinitely fast server. Hitting 11.7605 requires 5.821 s/prompt at that bound; the largest per-prompt mean in 31 clean nights is 5.333 s, and only the #4527 window reaches 7.47 s.

Clean-window statistics (31 nightlies):

cell / metric	old baseline	clean mean +/- sd	clean range	08-09 value	vs old	vs clean mean
rr=2.0 audio_throughput	11.7605	10.1186 +/- 0.2003	9.6387 - 10.4642	10.3253	-12.20%	+2.04%
c=1 median_audio_rtf	0.1241	0.1277 +/- 0.0030	0.1209 - 0.1365	0.1325	-6.74%	+3.77%

The flagged night is one of the better ones on record.

Before / after

Recomputing the pass/fail verdict for every clean nightly against the same measured values, old baselines vs new:

threshold	metric	old	new
10%	rr=2.0 audio_throughput	31/31 nightlies flagged	0/31
10%	c=8 median_audio_rtf	11/31	1/31
10%	all 8 metrics x 31 nightlies	62/248	12/248
5%	all 8 metrics x 31 nightlies	103/248	25/248

The stress-cell check fired on every single clean night. Note this is not "loosen until green": 12 pairs still flag at 10% after the change, mostly c=8 median_audio_ttfp_ms (6/31, the one genuinely noisy cell at 10.2% CV), so real outlier nights stay detectable.

New baselines

cell	metric	old	new
c=1 n=20	median_audio_ttfp_ms	82.1886	82.6344
c=1 n=20	median_audio_rtf	0.1241	0.1277
c=8 n=80	median_audio_ttfp_ms	160.9168	170.3737
c=8 n=80	median_audio_rtf	0.1907	0.2076
c=8 n=80	audio_throughput	39.2332	37.0834
rr=2.0 n=100	median_audio_ttfp_ms	100.1265	101.0293
rr=2.0 n=100	median_audio_rtf	0.1437	0.1498
rr=2.0 n=100	audio_throughput	11.7605	10.1186

Plain mean over the clean window, the estimator used in #5231 and #5774. Medians differ by at most 2.5%, so the choice is not load-bearing.

Testing

Ran the CI harness against the patched spec on 1x H20 (voxtral entry only):

pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file <spec>
3 passed in 278.73s

• c=1 n=20: 20/20 completed, 0 failed
• c=8 n=80: 80/80 completed, 0 failed
• rr=2.0 n=100: 100/100 completed, 0 failed
• every emitted result JSON carries the patched value, hardware-nested H100 bucket resolved to the per-sweep scalar
• is_hardware_nested_baseline and resolve_baseline_for_sweep(raw, sweep_index=0) return the patched scalars for all three cells

H20 independently corroborates the ceiling: the stress cell measured mean_audio_duration_s = 5.384 with audio_throughput = 10.576, and 2.0 x 5.384 = 10.77. Absolute throughput on H20 is not comparable to an H100 baseline, so those numbers are reported for completeness only; the H100 baselines come from the archived nightlies.

Live audio length is 4.9-5.4 s/prompt across both runs, matching the clean regime and never the 7.37 s the old baseline was built from.

Follow-ups, not in this PR

• Voxtral has no WER/accuracy cell in test_tts.json. The [Core] Performance fix for orchestrator bottleneck: separates inter-stage from client outputs #4527 breakage produced unrelated speech for nine nights and the perf dashboard read it as an improvement; only the weekly transcript check ([Bug]: Weekly CI, voxtral_tts, output audio doesn't match input #4949) caught it. A WER cell would have flagged it on night one.
• [Bug]: Nightly CI, Qwen/Qwen3-Omni-30B-A3B-Instruct, performance metrics regressed by more than 10% compared to the baseline in some scenarios #5966 (Qwen3-Omni mean_audio_rtf) is the same class of duration-normalized metric, and its per-night mean_audio_duration_s is visibly bimodal in the same raw data. Worth the same clean-window check before treating it as a slowdown.
• [CI]For NPU CI, Nest perf baselines by hardware label (H100/A3) #5402 removed the --assert-baseline comparison from assert_result in run_benchmark.py, so the in-repo test no longer has any perf-comparison logic. It only ever printed rather than failed, but the flag is now referenced nowhere.

Closes #5967

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[congw729]

lgtm

[Gaohan123]

LGTM. Thanks