[CI/Build] Refresh A3 and MiniCPM-o H100 performance baselines

[congw729]

Summary

Refresh 69 performance baseline values across 8 files: 54 A3 values and 15 MiniCPM-o H100 values.

All values use equal-weight nightly arithmetic means, rounded once to four decimals, without an additional margin. Benchmark parameters, deployment settings and regression tolerances are unchanged.

• A3: retain the previously reviewed sampling windows: Sep 13–19, 2026 by default; Aug 20–25 for MiniCPM-o C1 QPS/E2EL/audio TTFP/RTF and C8 audio TTFP. MiniCPM-o A3 C1 TTFT and all C4 metrics use Sep 13–15 and Sep 18–19, five nights total; Sep 16–17 are excluded because they used the temporary realtime backend. The remaining four MiniCPM-o C8 metrics use Sep 18–19 and have limited two-night coverage.
• MiniCPM-o H100: use all seven ordinary HTTP Seed-TTS runs per concurrency from Sep 22–28, 2026, covering C1/P32, C4/P64 and C8/P128. Duplex results are excluded because their interaction and timing boundaries differ.

MiniCPM-o baseline increases

A3 MiniCPM-o: historical comparison and window selection

The August–September audit, covering successful standard HTTP Seed-TTS runs through Sep 23, identified distinct performance periods. Values below are equal-weight nightly means; latency is in milliseconds.

Reference / period	Nights	C1 QPS ↑	C1 E2EL ↓	C1 audio TTFP ↓	C1 RTF ↓	C8 audio TTFP ↓
Stored baseline	—	0.5383	1857.2154	986.4666	0.4423	3352.7515
Aug 20–25 — selected	6	0.4813	2086.7028	1023.3301	0.4659	3151.4020
Aug 26 and 30 — transition	2	0.5202	1922.3492	991.2393	0.4344	6497.8394
Aug 31–Sep 15 — high latency	13	0.3881	2576.8780	1514.9553	0.5908	9922.3433
Sep 18–21 — vLLM 0.29	4	0.4613	2176.0199	1262.0670	0.4940	3721.0238
Sep 23 — vLLM 0.30	1	0.4716	2120.2826	1264.8660	0.4811	3753.6184

The selected window balances historical coverage and regression sensitivity:

• Why adjust the stored values? All 26 successful standard nights in the audit had lower C1 QPS and higher C1 E2EL than the stored references. Even the selected early six-night window averages 10.59% lower QPS and 12.36% higher E2EL. This supports recalibrating those references to an observed multi-night level.
• Why Aug 20–25? It provides six consecutive successful nights before the later serving/reference-audio changes and high-latency period, meeting the five-night minimum. The transition period has only two nights; the later observations provide four nights on vLLM 0.29 and one on 0.30, which are kept separate.
• Why retain all six nights? No slower observations were removed, including Aug 23 C1 and Aug 25 C8. C1 audio TTFP and RTF increase by 3.74% and 5.34%; their selected-window medians, 1022.1196 ms and 0.4596, are close to the adopted means of 1023.3301 ms and 0.4659. No additional margin is applied.
• Why exclude the slower later periods? Averaging across them would substantially loosen the references. Every selected value remains stricter than the Sep 18–21 and Sep 23 observations shown above. C8 audio TTFP also improves 6.01% against the stored baseline.

These values serve as historical performance references. The comparison supports the chosen sampling window; the causes of subsequent performance shifts remain incompletely explained.

H100: the stored baselines predate #6628, which enabled the HTTP path to use each request’s reference audio. This changes the conditioning workload and supports refreshing the reference, although the available results do not isolate its contribution to each performance difference.

The largest increases are:

Metric	Stored baseline	Updated baseline	Change
C1 audio TTFP	458.1553 ms	585.7387 ms	+27.85%
C8 audio TTFP	2068.6233 ms	2508.4384 ms	+21.26%
C8 audio RTF	1.0324	1.1787	+14.17%

The full seven-day window includes performance improvements during the week. C8 audio TTFP fell from approximately 2.99–3.09 seconds on Sep 22–24 to 2.04–2.20 seconds on Sep 25–28. Its seven-night baseline is therefore looser than the later observations and will be less sensitive to regressions from that improved level. No slower days were selectively removed.

These archived means describe the selected windows. Exact software revisions are incomplete for the H100 records, so they do not establish a fixed-version performance comparison or fully explain the increases. The C8 RTF < 1 optimization goal remains outstanding.

Validation and sources

• Previously verified 54 A3 values are unchanged by the H100 update.
• Independently recomputed all 15 H100 values from 21 unique raw results: 1,568 completed requests, zero reported failures and complete audio metric counts.
• JSON parsing, change-scope checks and git diff --check passed. No new inference or WER runs.

Sources: A3 September archive, A3 August archive, H100 archive, and H100 update details.

Qwen3-Omni random-workload baselines remain omitted after #8057. Wan2.2 retains the archived means with the metric-name and latency-unit conversion from #7737.

AI assistance: Codex assisted with archive analysis, baseline verification and PR wording.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR was classified as CI work.

CI owner: @yenuo26

@congw729, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[yenuo26]

LGTM

[Gaohan123]

LGTM. Thanks

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit b907547aca25 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[congw729]

@yenuo26 Can be merged

[yenuo26]

Does this need to be adjusted to ascending?

[congw729]

This has been updated since your review. After checking the daily history, C1 E2E was revised from 2457.6655 to 2086.7028 ms, using the equal-weight mean of all six successful Aug 20–25 nights. This avoids pooling the later high-latency period, with no extra margin added. The PR description now includes the sampling window and all six raw results.

AI assistance: Codex helped verify the values and draft this reply.

[yenuo26]

Does this need to be adjusted to descending?

[congw729]

This has also been updated since your review. C1 QPS was revised from 0.4100 to 0.4813, using the same Aug 20–25 six-night mean as C1 E2E. All six successful nights are included, with no extra margin added. It remains below the original 0.5383 baseline, and the selected historical data and calculation are documented in the PR description.

AI assistance: Codex helped verify the values and draft this reply.

[yenuo26]

@congw729 please resolve conflicts

[congw729]

@yenuo26 Could you please take another look? The five MiniCPM-o A3 baselines now use the Aug 20–25 six-night means, and the PR description and source links are updated. Conflicts with main are resolved; raw-data recomputation, pre-commit and CI checks passed. Thanks!

[natureofnature]

I verified all 69 changed baseline values against the dashboard data: 54 A3 values and 15 H100 values match the stated nightly averages. The benchmark parameters are unchanged.

The A3 sampling windows, including the historical references and limited two-night C8 coverage, are explicitly documented.

Before merging, please address update the PR description to include the 15 H100 changes and remove the statement that GPU baselines are unchanged

[natureofnature]

The updated description now covers both the A3 and H100 changes, and DCO passes. All 69 baseline values match the recomputed dashboard averages.

One small documentation correction: please restore the explicit sampling window for MiniCPM-o A3 C1 TTFT and all C4 metrics: Sep 13–15 and Sep 18–19, five nights total. These do not use the full Sep 13–19 default window. For example, C1 TTFT averages 1091.2690 ms across all seven nights, versus the configured 328.1820 ms across the selected five.

No additional code or numerical issues found. This verification covers the archived dashboard data, not a new inference run. Once the sampling note is restored and the remaining CI checks pass, this looks ready to merge.

[congw729]

@natureofnature Restored the explicit Sep 13–15 and Sep 18–19 (five nights) window for MiniCPM-o A3 C1 TTFT and all C4 metrics in the PR description. It also explains that Sep 16–17 are excluded because they used the temporary realtime backend. All baseline values remain unchanged.

AI assistance: Codex assisted with this documentation correction and reply.

[natureofnature]

LGTM

[yenuo26]

LGTM