[CI/Build] Update performance baselines and fill missing metrics by congw729 · Pull Request #8957 · vllm-project/vllm-omni

congw729 · 2026-10-15T03:29:06Z

Purpose

Refresh stored performance references and fill missing common metrics so maintainers can track repository performance against documented measurements.

This PR updates 124 existing values and adds 257 values across 15 JSON files, for 381 changed values:

Hardware	Existing values refreshed	Values added	Total
H100	97	140	237
A3	11	115	126
H200	16	2	18
Total	124	257	381

A value is one metric for one hardware/configuration/sweep slot. Added references cover measured throughput, mean latency, audio first-packet latency/RTF and mean memory. Existing median/P99 metric keys are retained.

The H200 changes cover four HunyuanImage configurations: TI2I, TP4, TP2+SP2 and TP2+CfgP2. All 16 existing values improve against their previous references; the two additions are latency_mean for TP2+SP2 and TP2+CfgP2. TI2I memory fields remain absent because the archived measurements are zero placeholders.

Only baseline blocks and two provenance descriptions change. This PR preserves benchmark inputs, deployment settings, metric definitions, hardware marks and quality gates; inference code is unchanged. The benchmarking design needs no contract update.

Calibration policy

References	Measurement window	Comparable days per changed value
H100	October 8–14, 2026	5–7
A3	October 8–14, 2026	6 (October 12 absent)
H200 HunyuanImage	October 10–16, 2026	7

Require at least five comparable days per metric/slot. Average measurements within each day, then weight each day equally; round to four decimals with no added margin. Match hardware, workload, deployment and sweep slot against the source revision. Exclude failed/incomplete runs, missing measurements, zero placeholders and incomplete new arrays.

Existing references are refreshed when the window mean is equal or better: higher throughput, lower latency/RTF/memory. 68 regressing existing values retain their previous references. The one approved exception is MiniCPM-o chat on A3 (Hardware=910), c=4, n=64, mean_ttft_ms: 470.0053 → 623.6599 ms (+32.69%), using six days.

Deferred coverage

These cases are excluded from the 381 changed values:

Case	Deferred values	Evidence needed
Qwen3-Omni multi-replica, H100	14 audio TTFP/RTF values	At least five comparable days after the fixed-Ethan/sender changes in #8826/#8833; currently one day verified. The four existing TPOT references are unchanged.
MiniCPM-o OmniInteract, H200	18 performance values	Window-on results matching #9033: `sliding_window_mode=basic`, 7000/5000 tokens. The October 9–14 measurements used window off. The IA_QTF1 ≥ 0.2 gate is unchanged.
HunyuanVideo I2V single-device, H100	3 values: throughput, mean E2E latency and mean memory	Tiled-VAE results matching #9066. The existing calibration used the untiled configuration. The separate cached TP2 I2V configuration retains its three added references.

For Qwen multi-replica, build #17479 on October 17 used commit b164fea797531c1fa10f8b87829632cad006b9fc with the matching 3×H100, Ethan configuration. All seven scenarios completed all 662 requests. The seven archived JSONs match the Buildkite artifact hashes. As checked on October 20, this is the only comparable run day. P99 audio TTFP was 6.1287 s at c=24 and 8.6798 s at c=32; repeated measurements are needed to assess variation before calibration. See the recalibration requirements and pinned archive.

Qwen3-Omni multi-replica: October 17 measurements, calibration pending

Scenario	Mean audio TTFP (ms)	Mean audio RTF	P99 audio TTFP (ms)	Mean / P99 audio duration (s)
c=8, n=128	486.5029	0.1837	1846.8562	184.4556 / 327.5769
c=16, n=128	462.9634	0.2111	1342.5591	170.7125 / 327.5769
c=24, n=128	1179.7345	0.2994	6128.7063	169.1262 / 327.5769
c=32, n=128	1809.3504	0.3593	8679.8426	171.5863 / 327.5769
r=0.1, n=10	402.8773	0.1024	472.4795	28.4809 / 32.2905
r=0.5, n=40	437.4673	0.1168	857.5149	27.0069 / 34.9529
r=1, n=100	1052.2592	0.1864	2107.0255	27.7921 / 42.6137

All requests meet their configured Thinker output lengths (900 tokens for the concurrency sweep, 100 for multimodal workloads). Talker output terminates naturally in this configuration; the observed audio-duration statistics are recorded above. These measurements are excluded from the changed baseline values.

Voxtral TTS and Qwen-Image-Edit-2511 remain excluded; their existing baselines are unchanged.

Test Plan

vLLM Version: use the archived run environments. The raw JSON does not record complete vLLM/PyTorch/CUDA versions; environment details remain limited to the linked source runs.

vLLM-Omni Commit: current PR head d77856777e0208073f029e4258ffa543ac893fc5; original reference comparison d6ed4f4844fbc41573ffc03247e5b2ff0a38d322. The source commits for the measurements are listed below.

Source archives: H100/A3 calibration uses kanban revision 5a70cec1; H200 HunyuanImage uses revision 5aba1fee. The HunyuanImage runs completed all 882 measured requests across 28 runs.

Source matching checks embedded baselines or structured test/config metadata against source-revision workloads. Only startup timeout options (stage-init-timeout, init-timeout) are ignored. Inline HunyuanImage datasets/deployments are checked against their source contents and the runner's temporary-file conversion. The fixed-length non-async Qwen3-Omni case additionally checks recorded Thinker/Talker outputs against its 900/1536-token contract.

Test Result

All 381 values were independently recomputed from raw results with source hashes, dates, hardware, day counts and rounding checked. The current GitHub diff and all table rows have been cross-checked: 124 refreshed + 257 added values, 15 files, two descriptions.
All 30 configuration JSON files parse, and added arrays cover their configured sweeps. Workload settings and quality gates are preserved. git diff --check and local pre-commit checks on the affected files passed.
A merge-tree check against main 37ef70ad39d94d81e624e3c9a6702b64072d322f was conflict-free and preserved main's workload settings, including the new OmniInteract window and single-device HunyuanVideo VAE tiling.
GitHub pre-commit, DCO and Python 3.11/3.12 build checks passed on d77856777. CI status is tracked in the PR Checks tab.
No new GPU/NPU benchmark or L2 run was launched as part of this update. The calibration uses the pinned historical measurements; output quality and performance on the current PR head have not been re-measured.

AI assistance: Codex analyzed the archives, prepared baseline edits, checked the calculations and drafted this description.

Contributor intent (required human answers)

1. What outcome do you want to achieve?

Keep performance references current and fill gaps so repository maintainers can better track repository performance.

2. What is the highest-leverage variable for that outcome?

The performance benchmark baseline values and coverage of common mean metrics.

3. How will this work move that variable?

Refresh GPU and A3 references from recent nightly results and add the confirmed missing metrics. Existing reference values are updated only when the measured result is equal or better, with one explicitly approved A3 MiniCPM-o TTFT exception.

4. Whose customer or user needs does this address, and what are those needs?

Repository maintainers need to track current repository performance more effectively.

Changes by file

File	Existing refreshed	New values	Hardware
`test_cosmos3_vllm_omni.json`	11	0	H100
`test_hunyuan_image3_it2i.json`	3	0	H200
`test_hunyuan_image_tp2_cfgp2.json`	4	2	A3, H200
`test_hunyuan_image_tp2_sp2.json`	5	2	A3, H200
`test_hunyuan_image_tp4.json`	5	0	H200
`test_hunyuanvideo15_i2v_vllm_omni.json`	0	3	H100
`test_hunyuanvideo15_t2v_vllm_omni.json`	0	9	A3, H100
`test_lingbot_video_vllm_omni.json`	0	3	H100
`test_minicpmo_4_5.json`	20	0	A3, H100
`test_minimax_h3_vllm_omni.json`	9	0	H100
`test_qwen3_omni_async_chunk.json`	4	48	A3, H100
`test_qwen3_omni_no_async_chunk.json`	2	90	A3, H100
`test_qwen_image_vllm_omni.json`	9	24	A3, H100
`test_tts.json`	40	76	A3, H100
`test_wan22_i2v_vllm_omni.json`	12	0	A3, H100

Baseline values

T/B identifies the one-based test/benchmark entry in each JSON. Each row is one metric at one resolved sweep slot. Request throughput is req/s; audio throughput is audio seconds/s; _ms is milliseconds; legacy latency_* is seconds; memory is MB; audio RTF is dimensionless. Old and New compare stored reference values. Added values have no previous reference.

test_cosmos3_vllm_omni.json

T1/B1: test_cosmos3_t2i_official_demo_2gpu / 1024x1024_steps4 / random
T2/B1: test_cosmos3_t2v_official_demo_2gpu / 1280x720_frames189_steps4 / random
T3/B1: test_cosmos3_i2v_official_demo_2gpu / 1280x720_frames189_steps4 / random-mm
T4/B1: test_cosmos3_v2v_official_demo_2gpu / 1280x720_frames189_steps4 / random-mm

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=3	H100	mean_e2el_ms	843.0	837.8462	-0.61%	7
1/1 · c=1 n=3	H100	mean_peak_memory_mb	80000.0	19571.1429	-75.54%	7
2/1 · c=1 n=3	H100	mean_e2el_ms	32766.5	25428.4168	-22.40%	7
2/1 · c=1 n=3	H100	mean_peak_memory_mb	28762.4762	27080.0	-5.85%	7
2/1 · c=1 n=3	H100	request_throughput	0.0307	0.0394	+28.34%	7
3/1 · c=1 n=3	H100	mean_e2el_ms	33100.9	25860.7593	-21.87%	7
3/1 · c=1 n=3	H100	mean_peak_memory_mb	29441.1429	27080.0	-8.02%	7
3/1 · c=1 n=3	H100	request_throughput	0.0305	0.0387	+26.89%	7
4/1 · c=1 n=3	H100	mean_e2el_ms	33925.0	26717.7842	-21.24%	7
4/1 · c=1 n=3	H100	mean_peak_memory_mb	29443.1429	27792.0	-5.61%	7
4/1 · c=1 n=3	H100	request_throughput	0.0299	0.0367	+22.74%	7

test_hunyuan_image3_it2i.json

T1/B1: test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / custom

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=16 n=96	H200	latency_mean	79.0385	38.3398	-51.49%	7
1/1 · c=16 n=96	H200	latency_p99	101.852	53.1847	-47.78%	7
1/1 · c=16 n=96	H200	throughput_qps	0.1874	0.39	+108.11%	7

test_hunyuan_image_tp2_cfgp2.json

T1/B1: test_hunyuan_image_tp2_cfgp2 / 1024x1024_steps8 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=10	A3	latency_mean	—	5.4893	new	6
1/1 · c=1 n=10	H200	latency_mean	—	2.1515	new	7
1/1 · c=1 n=10	H200	latency_p99	4.7469	2.4108	-49.21%	7
1/1 · c=1 n=10	H200	peak_memory_mb_max	101100	47865.7143	-52.66%	7
1/1 · c=1 n=10	H200	peak_memory_mb_mean	101100	47865.7143	-52.66%	7
1/1 · c=1 n=10	H200	throughput_qps	0.21	0.465	+121.43%	7

test_hunyuan_image_tp2_sp2.json

T1/B1: test_hunyuan_image_tp2_sp2 / 1024x1024_steps8 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=10	A3	latency_mean	—	5.7228	new	6
1/1 · c=1 n=10	A3	throughput_qps	0.1733	0.1749	+0.92%	6
1/1 · c=1 n=10	H200	latency_mean	—	2.345	new	7
1/1 · c=1 n=10	H200	latency_p99	5.1025	2.6683	-47.71%	7
1/1 · c=1 n=10	H200	peak_memory_mb_max	97402	48702.0	-50.00%	7
1/1 · c=1 n=10	H200	peak_memory_mb_mean	97402	48702.0	-50.00%	7
1/1 · c=1 n=10	H200	throughput_qps	0.2	0.4264	+113.20%	7

test_hunyuan_image_tp4.json

T1/B1: test_hunyuan_image_tp4 / 1024x1024_steps8 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=10	H200	latency_mean	4.6953	2.247	-52.14%	7
1/1 · c=1 n=10	H200	latency_p99	4.8601	2.5891	-46.73%	7
1/1 · c=1 n=10	H200	peak_memory_mb_max	56912.0	47020.0	-17.38%	7
1/1 · c=1 n=10	H200	peak_memory_mb_mean	56912.0	47020.0	-17.38%	7
1/1 · c=1 n=10	H200	throughput_qps	0.213	0.4463	+109.53%	7

test_hunyuanvideo15_i2v_vllm_omni.json

T2/B1: test_hunyuanvideo15_i2v_cachedit_tp2_vae_patch2 / 832x480_frames33_steps4 / random-mm

T/B · slot	Hardware	Metric	Old	New	Δ	Days
2/1 · c=1 n=10	H100	mean_e2el_ms	—	6035.9394	new	7
2/1 · c=1 n=10	H100	mean_peak_memory_mb	—	43518.0	new	7
2/1 · c=1 n=10	H100	request_throughput	—	0.1656	new	7

test_hunyuanvideo15_t2v_vllm_omni.json

T1/B1: test_hunyuanvideo15_t2v_single_device / 832x480_frames33_steps4 / random
T2/B1: test_hunyuanvideo15_t2v_cachedit_tp2_vae_patch2 / 832x480_frames33_steps4 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=10	H100	mean_e2el_ms	—	8016.2097	new	7
1/1 · c=1 n=10	H100	mean_peak_memory_mb	—	77912.0	new	7
1/1 · c=1 n=10	H100	request_throughput	—	0.1247	new	7
2/1 · c=1 n=10	A3	mean_e2el_ms	—	14332.837	new	6
2/1 · c=1 n=10	A3	mean_peak_memory_mb	—	44251.2	new	6
2/1 · c=1 n=10	A3	request_throughput	—	0.0698	new	6
2/1 · c=1 n=10	H100	mean_e2el_ms	—	4153.3162	new	7
2/1 · c=1 n=10	H100	mean_peak_memory_mb	—	42470.2	new	7
2/1 · c=1 n=10	H100	request_throughput	—	0.2413	new	7

test_lingbot_video_vllm_omni.json

T1/B1: test_lingbot_video_single_device / 320x192_frames9_steps2 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=3	H100	mean_e2el_ms	—	2005.6932	new	7
1/1 · c=1 n=3	H100	mean_peak_memory_mb	—	20744.0	new	7
1/1 · c=1 n=3	H100	request_throughput	—	0.4985	new	7

test_minicpmo_4_5.json

T1/B1: test_minicpmo_4_5 / seed-tts

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=32	A3	mean_ttft_ms	328.182	324.7352	-1.05%	6
1/1 · c=1 n=32	H100	mean_audio_rtf	0.2504	0.159	-36.50%	7
1/1 · c=1 n=32	H100	mean_audio_ttfp_ms	585.7387	362.9356	-38.04%	7
1/1 · c=1 n=32	H100	mean_e2el_ms	1042.7617	660.8259	-36.63%	7
1/1 · c=1 n=32	H100	mean_ttft_ms	156.5006	121.6802	-22.25%	7
1/1 · c=1 n=32	H100	request_throughput	0.9615	1.525	+58.61%	7
1/1 · c=4 n=64	A3	mean_audio_rtf	1.4737	1.3127	-10.92%	6
1/1 · c=4 n=64	A3	mean_e2el_ms	6480.6866	5760.4772	-11.11%	6
1/1 · c=4 n=64	A3	mean_ttft_ms	470.0053	623.6599	+32.69%	6
1/1 · c=4 n=64	A3	request_throughput	0.6381	0.6969	+9.21%	6
1/1 · c=4 n=64	H100	mean_audio_rtf	0.6281	0.2941	-53.18%	7
1/1 · c=4 n=64	H100	mean_audio_ttfp_ms	1396.1575	795.4822	-43.02%	7
1/1 · c=4 n=64	H100	mean_e2el_ms	2669.8806	1238.4447	-53.61%	7
1/1 · c=4 n=64	H100	mean_ttft_ms	272.8529	255.7893	-6.25%	7
1/1 · c=4 n=64	H100	request_throughput	1.4882	3.1855	+114.05%	7
1/1 · c=8 n=128	H100	mean_audio_rtf	1.1787	0.4743	-59.76%	7
1/1 · c=8 n=128	H100	mean_audio_ttfp_ms	2508.4384	1115.6453	-55.52%	7
1/1 · c=8 n=128	H100	mean_e2el_ms	4882.2845	1965.672	-59.74%	7
1/1 · c=8 n=128	H100	mean_ttft_ms	283.2613	256.2622	-9.53%	7
1/1 · c=8 n=128	H100	request_throughput	1.6245	4.036	+148.45%	7

test_minimax_h3_vllm_omni.json

T1/B1: test_minimax_h3_t2v_4gpu / 1344x768_frames209_steps8 / random
T2/B1: test_minimax_h3_ti2v_4gpu / 1344x768_frames209_steps8 / random-mm
T3/B1: test_minimax_h3_v2v_4gpu / 1344x768_frames209_steps8 / random-mm
T4/B1: test_minimax_h3_t2v_dlo_4gpu / 1344x768_frames209_steps8 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=3	H100	mean_peak_memory_mb	54219.6	46034.6667	-15.10%	7
1/1 · c=1 n=3	H100	request_throughput	0.0211	0.0267	+26.54%	7
2/1 · c=1 n=3	H100	mean_peak_memory_mb	54622.8	46339.3333	-15.16%	7
2/1 · c=1 n=3	H100	request_throughput	0.0212	0.0248	+16.98%	7
3/1 · c=1 n=3	H100	mean_e2el_ms	105750.8	76815.3987	-27.36%	7
3/1 · c=1 n=3	H100	mean_peak_memory_mb	55180.8	46830.0	-15.13%	7
3/1 · c=1 n=3	H100	request_throughput	0.0077	0.0155	+101.30%	7
4/1 · c=1 n=3	H100	mean_peak_memory_mb	36218.4	29538.6667	-18.44%	7
4/1 · c=1 n=3	H100	request_throughput	0.0212	0.0263	+24.06%	7

test_qwen3_omni_async_chunk.json

T1/B1: test_qwen3_omni_chunk / random-mm
T1/B2: test_qwen3_omni_chunk / random-mm
T1/B3: test_qwen3_omni_chunk / random-mm
T1/B4: test_qwen3_omni_chunk / random-mm
T1/B5: test_qwen3_omni_chunk / random-mm
T1/B6: test_qwen3_omni_chunk / random-mm

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · r=0.1 n=10	A3	mean_audio_ttfp_ms	—	1424.6187	new	6
1/1 · r=0.1 n=10	A3	mean_e2el_ms	—	35032.2415	new	6
1/1 · r=0.1 n=10	A3	mean_tpot_ms	—	27.2798	new	6
1/1 · r=0.1 n=10	A3	mean_ttft_ms	—	529.783	new	6
1/1 · r=0.1 n=10	A3	request_throughput	—	0.0758	new	6
1/1 · r=0.1 n=10	H100	mean_audio_rtf	0.1098	0.1063	-3.19%	7
1/1 · r=0.1 n=10	H100	mean_audio_ttfp_ms	—	418.0683	new	7
1/1 · r=0.1 n=10	H100	mean_e2el_ms	—	2875.1345	new	7
1/1 · r=0.1 n=10	H100	mean_tpot_ms	—	4.9271	new	7
1/1 · r=0.1 n=10	H100	mean_ttft_ms	—	302.6793	new	7
1/1 · r=0.1 n=10	H100	request_throughput	—	0.0976	new	7
1/2 · r=0.5 n=40	A3	mean_audio_ttfp_ms	—	2331.3428	new	6
1/2 · r=0.5 n=40	A3	mean_e2el_ms	—	52017.2332	new	6
1/2 · r=0.5 n=40	A3	mean_tpot_ms	—	30.9782	new	6
1/2 · r=0.5 n=40	A3	mean_ttft_ms	—	431.6208	new	6
1/2 · r=0.5 n=40	A3	request_throughput	—	0.31	new	6
1/2 · r=0.5 n=40	H100	mean_audio_rtf	0.1335	0.1296	-2.92%	7
1/2 · r=0.5 n=40	H100	mean_audio_ttfp_ms	—	472.818	new	7
1/2 · r=0.5 n=40	H100	mean_e2el_ms	—	3450.9627	new	7
1/2 · r=0.5 n=40	H100	mean_tpot_ms	—	5.3421	new	7
1/2 · r=0.5 n=40	H100	mean_ttft_ms	—	292.8627	new	7
1/2 · r=0.5 n=40	H100	request_throughput	—	0.483	new	7
1/3 · r=1.0 n=100	A3	mean_audio_ttfp_ms	—	7125.4738	new	6
1/3 · r=1.0 n=100	A3	mean_e2el_ms	—	87154.8693	new	6
1/3 · r=1.0 n=100	A3	mean_tpot_ms	—	44.8081	new	6
1/3 · r=1.0 n=100	A3	mean_ttft_ms	—	1282.0529	new	6
1/3 · r=1.0 n=100	A3	request_throughput	—	0.5098	new	6
1/3 · r=1.0 n=100	H100	mean_audio_ttfp_ms	—	759.7934	new	7
1/3 · r=1.0 n=100	H100	mean_e2el_ms	—	4944.8597	new	7
1/3 · r=1.0 n=100	H100	mean_tpot_ms	—	6.5413	new	7
1/3 · r=1.0 n=100	H100	mean_ttft_ms	—	460.0396	new	7
1/3 · r=1.0 n=100	H100	request_throughput	—	0.9562	new	7
1/4 · r=0.1 n=10	A3	mean_tpot_ms	—	24.5161	new	6
1/4 · r=0.1 n=10	A3	mean_ttft_ms	—	463.8431	new	6
1/4 · r=0.1 n=10	A3	request_throughput	—	0.0972	new	6
1/4 · r=0.1 n=10	H100	mean_e2el_ms	756.8415	733.5836	-3.07%	7
1/4 · r=0.1 n=10	H100	mean_tpot_ms	—	4.6254	new	7
1/4 · r=0.1 n=10	H100	mean_ttft_ms	—	275.2829	new	7
1/4 · r=0.1 n=10	H100	request_throughput	—	0.0993	new	7
1/5 · r=0.5 n=40	A3	mean_tpot_ms	—	27.363	new	6
1/5 · r=0.5 n=40	A3	mean_ttft_ms	—	347.8392	new	6
1/5 · r=0.5 n=40	A3	request_throughput	—	0.484	new	6
1/5 · r=0.5 n=40	H100	mean_tpot_ms	—	5.1925	new	7
1/5 · r=0.5 n=40	H100	mean_ttft_ms	—	231.253	new	7
1/5 · r=0.5 n=40	H100	request_throughput	—	0.4966	new	7
1/6 · r=1.0 n=100	A3	mean_tpot_ms	—	36.0758	new	6
1/6 · r=1.0 n=100	A3	mean_ttft_ms	—	878.5597	new	6
1/6 · r=1.0 n=100	A3	request_throughput	—	0.963	new	6
1/6 · r=1.0 n=100	H100	mean_e2el_ms	1006.1663	992.3741	-1.37%	7
1/6 · r=1.0 n=100	H100	mean_tpot_ms	—	6.1976	new	7
1/6 · r=1.0 n=100	H100	mean_ttft_ms	—	378.4494	new	7
1/6 · r=1.0 n=100	H100	request_throughput	—	0.9903	new	7

test_qwen3_omni_no_async_chunk.json

T1/B1: test_qwen3_omni / random
T1/B2: test_qwen3_omni / random-mm
T1/B3: test_qwen3_omni / random-mm
T1/B4: test_qwen3_omni / random-mm

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=4	A3	mean_audio_rtf	—	1.1436	new	6
1/1 · c=1 n=4	A3	mean_audio_ttfp_ms	—	140275.5251	new	6
1/1 · c=1 n=4	A3	mean_e2el_ms	—	140296.3104	new	6
1/1 · c=1 n=4	A3	mean_tpot_ms	—	28.0779	new	6
1/1 · c=1 n=4	A3	mean_ttft_ms	—	209.3508	new	6
1/1 · c=1 n=4	A3	request_throughput	—	0.0078	new	6
1/1 · c=1 n=4	H100	mean_audio_rtf	—	0.1527	new	5
1/1 · c=1 n=4	H100	mean_audio_ttfp_ms	—	18729.2352	new	5
1/1 · c=1 n=4	H100	mean_e2el_ms	—	18752.4068	new	5
1/1 · c=1 n=4	H100	mean_tpot_ms	—	6.6888	new	5
1/1 · c=1 n=4	H100	mean_ttft_ms	—	116.3593	new	5
1/1 · c=1 n=4	H100	request_throughput	—	0.0533	new	5
1/1 · c=4 n=16	A3	mean_audio_rtf	—	1.2789	new	6
1/1 · c=4 n=16	A3	mean_audio_ttfp_ms	—	156867.7576	new	6
1/1 · c=4 n=16	A3	mean_e2el_ms	—	156889.7434	new	6
1/1 · c=4 n=16	A3	mean_tpot_ms	—	33.0101	new	6
1/1 · c=4 n=16	A3	mean_ttft_ms	—	364.3165	new	6
1/1 · c=4 n=16	A3	request_throughput	—	0.0274	new	6
1/1 · c=4 n=16	H100	mean_audio_rtf	—	0.2042	new	5
1/1 · c=4 n=16	H100	mean_audio_ttfp_ms	—	25052.649	new	5
1/1 · c=4 n=16	H100	mean_e2el_ms	—	25074.0092	new	5
1/1 · c=4 n=16	H100	mean_tpot_ms	—	10.9254	new	5
1/1 · c=4 n=16	H100	mean_ttft_ms	—	215.2391	new	5
1/1 · c=4 n=16	H100	request_throughput	—	0.158	new	5
1/1 · c=8 n=32	A3	mean_audio_rtf	—	1.4004	new	6
1/1 · c=8 n=32	A3	mean_audio_ttfp_ms	—	171714.097	new	6
1/1 · c=8 n=32	A3	mean_e2el_ms	—	171817.2825	new	6
1/1 · c=8 n=32	A3	mean_tpot_ms	—	38.8505	new	6
1/1 · c=8 n=32	A3	mean_ttft_ms	—	458.4411	new	6
1/1 · c=8 n=32	A3	request_throughput	—	0.0491	new	6
1/1 · c=8 n=32	H100	mean_audio_rtf	—	0.2449	new	5
1/1 · c=8 n=32	H100	mean_audio_ttfp_ms	—	30040.0737	new	5
1/1 · c=8 n=32	H100	mean_e2el_ms	—	30065.1658	new	5
1/1 · c=8 n=32	H100	mean_tpot_ms	—	13.9474	new	5
1/1 · c=8 n=32	H100	mean_ttft_ms	—	296.4664	new	5
1/1 · c=8 n=32	H100	request_throughput	—	0.2585	new	5
1/1 · c=16 n=64	A3	mean_audio_rtf	—	1.6888	new	6
1/1 · c=16 n=64	A3	mean_audio_ttfp_ms	—	206912.0565	new	6
1/1 · c=16 n=64	A3	mean_e2el_ms	—	207225.9885	new	6
1/1 · c=16 n=64	A3	mean_tpot_ms	—	50.6796	new	6
1/1 · c=16 n=64	A3	mean_ttft_ms	—	723.1123	new	6
1/1 · c=16 n=64	A3	request_throughput	—	0.0795	new	6
1/1 · c=16 n=64	H100	mean_audio_rtf	—	0.3064	new	5
1/1 · c=16 n=64	H100	mean_audio_ttfp_ms	—	37565.8908	new	5
1/1 · c=16 n=64	H100	mean_e2el_ms	—	37608.3122	new	5
1/1 · c=16 n=64	H100	mean_tpot_ms	—	18.1473	new	5
1/1 · c=16 n=64	H100	mean_ttft_ms	—	430.4905	new	5
1/1 · c=16 n=64	H100	request_throughput	—	0.4017	new	5
1/1 · c=32 n=128	A3	mean_audio_rtf	—	2.6025	new	6
1/1 · c=32 n=128	A3	mean_audio_ttfp_ms	—	318934.5767	new	6
1/1 · c=32 n=128	A3	mean_e2el_ms	—	319344.5635	new	6
1/1 · c=32 n=128	A3	mean_tpot_ms	—	120.7271	new	6
1/1 · c=32 n=128	A3	mean_ttft_ms	—	1171.6259	new	6
1/1 · c=32 n=128	A3	request_throughput	—	0.0994	new	6
1/1 · c=32 n=128	H100	mean_audio_rtf	—	0.4115	new	5
1/1 · c=32 n=128	H100	mean_audio_ttfp_ms	—	50449.3316	new	5
1/1 · c=32 n=128	H100	mean_e2el_ms	—	50491.0797	new	5
1/1 · c=32 n=128	H100	mean_tpot_ms	—	23.9778	new	5
1/1 · c=32 n=128	H100	mean_ttft_ms	—	812.2658	new	5
1/1 · c=32 n=128	H100	request_throughput	—	0.5873	new	5
1/2 · r=0.1 n=10	A3	mean_audio_ttfp_ms	—	31414.5578	new	6
1/2 · r=0.1 n=10	A3	mean_e2el_ms	—	31418.3177	new	6
1/2 · r=0.1 n=10	A3	mean_tpot_ms	—	26.7059	new	6
1/2 · r=0.1 n=10	A3	mean_ttft_ms	—	452.723	new	6
1/2 · r=0.1 n=10	A3	request_throughput	—	0.0769	new	6
1/2 · r=0.1 n=10	H100	mean_audio_ttfp_ms	—	3754.1397	new	5
1/2 · r=0.1 n=10	H100	mean_e2el_ms	—	3757.4875	new	5
1/2 · r=0.1 n=10	H100	mean_tpot_ms	—	6.4382	new	5
1/2 · r=0.1 n=10	H100	mean_ttft_ms	—	316.1369	new	5
1/2 · r=0.1 n=10	H100	request_throughput	—	0.0965	new	5
1/3 · r=0.5 n=40	A3	mean_audio_ttfp_ms	—	42196.627	new	6
1/3 · r=0.5 n=40	A3	mean_e2el_ms	—	42200.2865	new	6
1/3 · r=0.5 n=40	A3	mean_tpot_ms	—	30.9399	new	6
1/3 · r=0.5 n=40	A3	mean_ttft_ms	—	455.5273	new	6
1/3 · r=0.5 n=40	A3	request_throughput	—	0.3298	new	6
1/3 · r=0.5 n=40	H100	mean_audio_rtf	0.1609	0.1543	-4.10%	5
1/3 · r=0.5 n=40	H100	mean_audio_ttfp_ms	—	4082.0392	new	5
1/3 · r=0.5 n=40	H100	mean_e2el_ms	—	4085.4559	new	5
1/3 · r=0.5 n=40	H100	mean_tpot_ms	—	7.0756	new	5
1/3 · r=0.5 n=40	H100	mean_ttft_ms	—	272.1915	new	5
1/3 · r=0.5 n=40	H100	request_throughput	—	0.4781	new	5
1/4 · r=1.0 n=100	A3	mean_audio_ttfp_ms	—	69675.1373	new	6
1/4 · r=1.0 n=100	A3	mean_e2el_ms	—	69678.8378	new	6
1/4 · r=1.0 n=100	A3	mean_tpot_ms	—	47.28	new	6
1/4 · r=1.0 n=100	A3	mean_ttft_ms	—	1236.0965	new	6
1/4 · r=1.0 n=100	A3	request_throughput	—	0.5375	new	6
1/4 · r=1.0 n=100	H100	mean_audio_rtf	0.2048	0.1939	-5.32%	5
1/4 · r=1.0 n=100	H100	mean_audio_ttfp_ms	—	5243.423	new	5
1/4 · r=1.0 n=100	H100	mean_e2el_ms	—	5246.8779	new	5
1/4 · r=1.0 n=100	H100	mean_tpot_ms	—	9.1347	new	5
1/4 · r=1.0 n=100	H100	mean_ttft_ms	—	464.283	new	5
1/4 · r=1.0 n=100	H100	request_throughput	—	0.9521	new	5

test_qwen_image_vllm_omni.json

T2/B1: test_qwen_image_single_device_a3 / 512x512_steps20 / random
T3/B3: test_qwen_image_single_device_step_execution / 512x512_steps20_high_concurrency / random
T4/B1: test_qwen_image_single_device_step_execution_a3 / 512x512_steps20 / random
T4/B2: test_qwen_image_single_device_step_execution_a3 / 512x512_steps20_high_concurrency / random
T5/B1: test_qwen_image_ulysses2_cfg2_vae_patch4 / 1536x1536_steps35 / random
T6/B1: test_qwen_image_ulysses2_cfg2_cache_dit / 512x512_steps20 / random
T7/B1: test_qwen_image_ulysses2_cfg2_cache_dit_a3 / 512x512_steps20 / random

T/B · slot	Hardware	Metric	Old	New	Δ	Days
2/1 · c=1 n=10	A3	latency_mean	—	7.1679	new	6
2/1 · c=1 n=10	A3	peak_memory_mb_mean	—	56318.0	new	6
2/1 · c=1 n=10	A3	throughput_qps	—	0.1395	new	6
3/3 · c=1 n=20	H100	peak_memory_mb_mean	55149.0	55033.0	-0.21%	6
3/3 · c=2 n=40	H100	peak_memory_mb_mean	55150.0	55033.0	-0.21%	6
3/3 · c=4 n=80	H100	latency_mean	6.3948	6.3884	-0.10%	6
3/3 · c=4 n=80	H100	peak_memory_mb_mean	55153.1458	55033.6667	-0.22%	6
3/3 · c=4 n=80	H100	throughput_qps	0.6188	0.6189	+0.02%	6
3/3 · c=8 n=160	H100	latency_mean	10.1944	9.9601	-2.30%	6
3/3 · c=8 n=160	H100	peak_memory_mb_mean	55156.325	55035.7521	-0.22%	6
3/3 · c=8 n=160	H100	throughput_qps	0.7752	0.7931	+2.31%	6
4/1 · c=1 n=10	A3	latency_mean	—	7.1803	new	6
4/1 · c=1 n=10	A3	peak_memory_mb_mean	—	56318.0	new	6
4/1 · c=1 n=10	A3	throughput_qps	—	0.1393	new	6
4/2 · c=1 n=20	A3	latency_mean	—	7.2208	new	6
4/2 · c=1 n=20	A3	peak_memory_mb_mean	—	56321.0	new	6
4/2 · c=1 n=20	A3	throughput_qps	—	0.1385	new	6
4/2 · c=2 n=40	A3	latency_mean	—	11.3784	new	6
4/2 · c=2 n=40	A3	peak_memory_mb_mean	—	56321.3333	new	6
4/2 · c=2 n=40	A3	throughput_qps	—	0.1756	new	6
4/2 · c=4 n=80	A3	latency_mean	—	19.208	new	6
4/2 · c=4 n=80	A3	peak_memory_mb_mean	—	56321.3333	new	6
4/2 · c=4 n=80	A3	throughput_qps	—	0.2076	new	6
4/2 · c=8 n=160	A3	latency_mean	—	36.7984	new	6
4/2 · c=8 n=160	A3	peak_memory_mb_mean	—	56323.8562	new	6
4/2 · c=8 n=160	A3	throughput_qps	—	0.2154	new	6
5/1 · c=1 n=10	A3	latency_mean	—	30.1144	new	6
5/1 · c=1 n=10	A3	peak_memory_mb_mean	—	55994.0	new	6
5/1 · c=1 n=10	A3	throughput_qps	—	0.0332	new	6
6/1 · c=1 n=10	H100	peak_memory_mb_mean	55174.0	55154.0	-0.04%	7
7/1 · c=1 n=10	A3	latency_mean	—	3.5091	new	6
7/1 · c=1 n=10	A3	peak_memory_mb_mean	—	56418.0	new	6
7/1 · c=1 n=10	A3	throughput_qps	—	0.2854	new	6

test_tts.json

T1/B1: test_qwen3_tts_base / seed-tts / latency
T1/B2: test_qwen3_tts_base / seed-tts / throughput
T2/B1: test_qwen3_tts_customvoice / seed-tts-text / latency
T2/B2: test_qwen3_tts_customvoice / seed-tts-text / throughput
T2/B4: test_qwen3_tts_customvoice / seed-tts-text / stress
T3/B1: test_qwen3_tts_voicedesign / seed-tts-design / latency
T3/B2: test_qwen3_tts_voicedesign / seed-tts-design / throughput
T3/B3: test_qwen3_tts_voicedesign / seed-tts-design / stress

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=20	A3	audio_throughput	—	1.744	new	6
1/1 · c=1 n=20	A3	mean_audio_rtf	—	0.6243	new	6
1/1 · c=1 n=20	A3	mean_audio_ttfp_ms	—	238.3013	new	6
1/1 · c=1 n=20	A3	mean_e2el_ms	—	2892.7748	new	6
1/1 · c=1 n=20	A3	request_throughput	—	0.3743	new	6
1/1 · c=1 n=20	H100	audio_throughput	—	9.571	new	7
1/1 · c=1 n=20	H100	mean_audio_rtf	—	0.107	new	7
1/1 · c=1 n=20	H100	mean_audio_ttfp_ms	—	114.4824	new	7
1/1 · c=1 n=20	H100	mean_e2el_ms	—	483.1199	new	7
1/1 · c=1 n=20	H100	median_audio_rtf	0.1591	0.1045	-34.32%	7
1/1 · c=1 n=20	H100	median_audio_ttfp_ms	141.2791	113.263	-19.83%	7
1/1 · c=1 n=20	H100	request_throughput	—	2.0722	new	7
1/2 · c=8 n=80	A3	mean_audio_rtf	—	0.8344	new	6
1/2 · c=8 n=80	A3	mean_audio_ttfp_ms	—	366.4724	new	6
1/2 · c=8 n=80	A3	mean_e2el_ms	—	3567.9805	new	6
1/2 · c=8 n=80	A3	request_throughput	—	2.2676	new	6
1/2 · c=8 n=80	H100	audio_throughput	24.2296	40.7791	+68.30%	7
1/2 · c=8 n=80	H100	mean_audio_rtf	—	0.1946	new	7
1/2 · c=8 n=80	H100	mean_audio_ttfp_ms	—	140.3161	new	7
1/2 · c=8 n=80	H100	mean_e2el_ms	—	828.2049	new	7
1/2 · c=8 n=80	H100	median_audio_rtf	0.3313	0.2057	-37.91%	7
1/2 · c=8 n=80	H100	median_audio_ttfp_ms	226.747	130.2628	-42.55%	7
1/2 · c=8 n=80	H100	request_throughput	—	9.4583	new	7
1/2 · c=16 n=128	A3	mean_audio_rtf	—	0.9881	new	6
1/2 · c=16 n=128	A3	mean_audio_ttfp_ms	—	451.8776	new	6
1/2 · c=16 n=128	A3	mean_e2el_ms	—	4214.6533	new	6
1/2 · c=16 n=128	A3	median_audio_ttfp_ms	451.6758	424.197	-6.08%	6
1/2 · c=16 n=128	A3	request_throughput	—	3.6742	new	6
1/2 · c=16 n=128	H100	audio_throughput	40.3552	98.0287	+142.91%	7
1/2 · c=16 n=128	H100	mean_audio_rtf	—	0.161	new	7
1/2 · c=16 n=128	H100	mean_audio_ttfp_ms	—	102.533	new	7
1/2 · c=16 n=128	H100	mean_e2el_ms	—	667.4215	new	7
1/2 · c=16 n=128	H100	median_audio_rtf	0.3642	0.1344	-63.10%	7
1/2 · c=16 n=128	H100	median_audio_ttfp_ms	647.1272	74.8594	-88.43%	7
1/2 · c=16 n=128	H100	request_throughput	—	22.9114	new	7
1/2 · c=64 n=128	A3	mean_audio_rtf	—	1.7718	new	6
1/2 · c=64 n=128	A3	mean_audio_ttfp_ms	—	1336.6221	new	6
1/2 · c=64 n=128	A3	mean_e2el_ms	—	7559.9614	new	6
1/2 · c=64 n=128	A3	median_audio_ttfp_ms	1844.1087	1140.1605	-38.17%	6
1/2 · c=64 n=128	A3	request_throughput	—	7.0458	new	6
1/2 · c=64 n=128	H100	audio_throughput	43.2572	159.1806	+267.99%	7
1/2 · c=64 n=128	H100	mean_audio_rtf	—	0.3847	new	7
1/2 · c=64 n=128	H100	mean_audio_ttfp_ms	—	629.0015	new	7
1/2 · c=64 n=128	H100	mean_e2el_ms	—	1579.9485	new	7
1/2 · c=64 n=128	H100	median_audio_rtf	1.2282	0.3753	-69.44%	7
1/2 · c=64 n=128	H100	median_audio_ttfp_ms	4777.2714	596.889	-87.51%	7
1/2 · c=64 n=128	H100	request_throughput	—	37.1243	new	7
2/1 · c=1 n=20	H100	audio_throughput	—	11.3761	new	7
2/1 · c=1 n=20	H100	mean_audio_rtf	—	0.0882	new	7
2/1 · c=1 n=20	H100	mean_audio_ttfp_ms	—	23.1222	new	7
2/1 · c=1 n=20	H100	mean_e2el_ms	—	496.5752	new	7
2/1 · c=1 n=20	H100	median_audio_rtf	0.1367	0.0881	-35.55%	7
2/1 · c=1 n=20	H100	median_audio_ttfp_ms	46.8434	22.4748	-52.02%	7
2/1 · c=1 n=20	H100	request_throughput	—	2.0143	new	7
2/2 · c=8 n=80	H100	audio_throughput	38.3858	73.8802	+92.47%	7
2/2 · c=8 n=80	H100	mean_audio_rtf	—	0.1033	new	7
2/2 · c=8 n=80	H100	mean_audio_ttfp_ms	—	32.2869	new	7
2/2 · c=8 n=80	H100	mean_e2el_ms	—	581.5505	new	7
2/2 · c=8 n=80	H100	median_audio_rtf	0.1859	0.1034	-44.38%	7
2/2 · c=8 n=80	H100	median_audio_ttfp_ms	75.3341	29.6018	-60.71%	7
2/2 · c=8 n=80	H100	request_throughput	—	13.0839	new	7
2/2 · c=16 n=128	H100	audio_throughput	55.3523	126.089	+127.79%	7
2/2 · c=16 n=128	H100	mean_audio_rtf	—	0.1213	new	7
2/2 · c=16 n=128	H100	mean_audio_ttfp_ms	—	39.5704	new	7
2/2 · c=16 n=128	H100	mean_e2el_ms	—	678.9494	new	7
2/2 · c=16 n=128	H100	median_audio_rtf	0.3148	0.1212	-61.50%	7
2/2 · c=16 n=128	H100	median_audio_ttfp_ms	713.7727	33.5364	-95.30%	7
2/2 · c=16 n=128	H100	request_throughput	—	22.4624	new	7
2/2 · c=64 n=128	H100	audio_throughput	68.7032	225.7789	+228.63%	7
2/2 · c=64 n=128	H100	mean_audio_rtf	—	0.2688	new	7
2/2 · c=64 n=128	H100	mean_audio_ttfp_ms	—	140.4855	new	7
2/2 · c=64 n=128	H100	mean_e2el_ms	—	1480.2301	new	7
2/2 · c=64 n=128	H100	median_audio_rtf	1.1662	0.2687	-76.96%	7
2/2 · c=64 n=128	H100	median_audio_ttfp_ms	5941.7179	129.2319	-97.83%	7
2/2 · c=64 n=128	H100	request_throughput	—	40.2506	new	7
2/4 · r=2.0 n=100	H100	audio_throughput	11.1226	11.1367	+0.13%	7
2/4 · r=2.0 n=100	H100	mean_audio_rtf	—	0.0922	new	7
2/4 · r=2.0 n=100	H100	mean_audio_ttfp_ms	—	26.5259	new	7
2/4 · r=2.0 n=100	H100	mean_e2el_ms	—	517.4759	new	7
2/4 · r=2.0 n=100	H100	median_audio_rtf	0.1571	0.0921	-41.37%	7
2/4 · r=2.0 n=100	H100	median_audio_ttfp_ms	56.746	25.7631	-54.60%	7
2/4 · r=2.0 n=100	H100	request_throughput	—	1.9764	new	7
3/1 · c=1 n=20	H100	audio_throughput	—	11.2089	new	7
3/1 · c=1 n=20	H100	mean_audio_rtf	—	0.0895	new	7
3/1 · c=1 n=20	H100	mean_audio_ttfp_ms	—	22.7988	new	7
3/1 · c=1 n=20	H100	mean_e2el_ms	—	400.8743	new	7
3/1 · c=1 n=20	H100	median_audio_rtf	0.1352	0.0891	-34.10%	7
3/1 · c=1 n=20	H100	median_audio_ttfp_ms	46.3479	22.1761	-52.15%	7
3/1 · c=1 n=20	H100	request_throughput	—	2.4938	new	7
3/2 · c=8 n=80	H100	audio_throughput	38.9378	68.8888	+76.92%	7
3/2 · c=8 n=80	H100	mean_audio_rtf	—	0.1119	new	7
3/2 · c=8 n=80	H100	mean_audio_ttfp_ms	—	37.131	new	7
3/2 · c=8 n=80	H100	mean_e2el_ms	—	505.5975	new	7
3/2 · c=8 n=80	H100	median_audio_rtf	0.1871	0.1122	-40.03%	7
3/2 · c=8 n=80	H100	median_audio_ttfp_ms	75.5967	33.4549	-55.75%	7
3/2 · c=8 n=80	H100	request_throughput	—	15.1764	new	7
3/2 · c=16 n=128	H100	audio_throughput	55.1846	114.687	+107.82%	7
3/2 · c=16 n=128	H100	mean_audio_rtf	—	0.1344	new	7
3/2 · c=16 n=128	H100	mean_audio_ttfp_ms	—	43.8621	new	7
3/2 · c=16 n=128	H100	mean_e2el_ms	—	591.5199	new	7
3/2 · c=16 n=128	H100	median_audio_rtf	0.3143	0.134	-57.37%	7
3/2 · c=16 n=128	H100	median_audio_ttfp_ms	587.5877	36.392	-93.81%	7
3/2 · c=16 n=128	H100	request_throughput	—	25.9533	new	7
3/2 · c=64 n=128	H100	audio_throughput	71.5708	214.7879	+200.11%	7
3/2 · c=64 n=128	H100	mean_audio_rtf	—	0.2698	new	7
3/2 · c=64 n=128	H100	mean_audio_ttfp_ms	—	143.9237	new	7
3/2 · c=64 n=128	H100	mean_e2el_ms	—	1183.4333	new	7
3/2 · c=64 n=128	H100	median_audio_rtf	1.1703	0.2614	-77.66%	7
3/2 · c=64 n=128	H100	median_audio_ttfp_ms	4960.3726	114.5276	-97.69%	7
3/2 · c=64 n=128	H100	request_throughput	—	48.6876	new	7
3/3 · r=2.0 n=100	H100	mean_audio_rtf	—	0.0948	new	7
3/3 · r=2.0 n=100	H100	mean_audio_ttfp_ms	—	27.4077	new	7
3/3 · r=2.0 n=100	H100	mean_e2el_ms	—	418.9156	new	7
3/3 · r=2.0 n=100	H100	median_audio_rtf	0.1508	0.0945	-37.33%	7
3/3 · r=2.0 n=100	H100	median_audio_ttfp_ms	55.5029	27.028	-51.30%	7
3/3 · r=2.0 n=100	H100	request_throughput	—	1.9812	new	7

test_wan22_i2v_vllm_omni.json

T1/B1: test_wan22_i2v_single_device / 832x480_frames81_steps4 / random-mm
T2/B1: test_wan22_i2v_usp2_vae_patch2_hsdp_slicing / 832x480_frames81_steps4 / random-mm
T2/B2: test_wan22_i2v_usp2_vae_patch2_hsdp_slicing / 1280x720_frames121_steps4 / random-mm

T/B · slot	Hardware	Metric	Old	New	Δ	Days
1/1 · c=1 n=10	H100	mean_e2el_ms	27620.6	24166.28	-12.51%	7
1/1 · c=1 n=10	H100	mean_peak_memory_mb	80548.0	74032.6571	-8.09%	7
1/1 · c=1 n=10	H100	request_throughput	0.0361	0.0414	+14.68%	7
2/1 · c=1 n=10	A3	mean_peak_memory_mb	59932.5714	59763.64	-0.28%	5
2/1 · c=1 n=10	H100	mean_e2el_ms	18941.5	18296.7849	-3.40%	7
2/1 · c=1 n=10	H100	mean_peak_memory_mb	50053.7	47041.8	-6.02%	7
2/1 · c=1 n=10	H100	request_throughput	0.05263	0.0547	+3.93%	7
2/2 · c=1 n=10	A3	mean_e2el_ms	469604.3521	467818.7769	-0.38%	6
2/2 · c=1 n=10	A3	mean_peak_memory_mb	59889.7714	59792.0	-0.16%	6
2/2 · c=1 n=10	H100	mean_e2el_ms	104105.8	93497.9734	-10.19%	7
2/2 · c=1 n=10	H100	mean_peak_memory_mb	59718.0	53625.1143	-10.20%	7
2/2 · c=1 n=10	H100	request_throughput	0.009595	0.0107	+11.52%	7

Source runs

H100/A3 Buildkite runs and H200 HunyuanImage local source revisions

Date	Hardware	Build	Source revision
20261008	A3	8630	`1e0c4f3d54a91c30de97aa332e772dfe7338f52c`
20261008	H100	16885	`4e675f8b6d742c0b9f097594ab98414485bd8692`
20261009	A3	8715	`b65107d61331327424910c72e7a2da137dcfb3df`
20261009	H100	16971	`3b666e9196528f00edb6ad9f558bfafde5d6cddf`
20261010	A3	8754	`639abb2a9fca94f2f2e0be4df184d3f161e46080`
20261010	H100	17021	`131a7731a44e2a64feac165580ce99d36beb94ae`
20261011	A3	8790	`198c2105930bd1bac0b0966ffd0997cdd84d4790`
20261011	H100	17048	`bdc01f866e12c5b4d7e0523b161b4dd87f77bf8f`
20261012	H100	17083	`c10115052ef691e4d0270f1ddf5bce5bc4d140f8`
20261013	A3	8893	`549965b94a1aeb9ace43b4713ccc4871f5bbc0bc`
20261013	H100	17137	`2ac2cb3ba9816933442cab73b671a9409857a838`
20261014	A3	8965	`c4871a09b006507ed38e3f4eb2e5db81d18f1191`
20261014	H100	17214	`c2ba8e715518d8d8c32f6dfb8f047524fd4681c1`

Hunyuan local date	Hardware	Archive	Source revision
20261010	H200	results	`b65107d61331327424910c72e7a2da137dcfb3df`
20261011	H200	results	`194b936637f4e7356cc998c2796c67d14d52d854`
20261012	H200	results	`198c2105930bd1bac0b0966ffd0997cdd84d4790`
20261013	H200	results	`76f52f1ae59078cecf47759a56d73e8bcd2ba311`
20261014	H200	results	`549965b94a1aeb9ace43b4713ccc4871f5bbc0bc`
20261015	H200	results	`c4871a09b006507ed38e3f4eb2e5db81d18f1191`
20261016	H200	results	`730632e64282f1d04d628116a00efd1066e2e904`

Unchanged references and remaining measurement gaps

Within the selected calibration windows, 52 existing values had only one comparable day and 76 had no comparable data; all retain their previous references. Seven eligible existing values are unchanged after rounding. B200 configurations without measurements and MiniCPM chat TPOT without measured samples remain uncalibrated.

Existing references left unchanged where the window mean regressed by more than 10%.

Test	Hardware	Values	Range
`test_minicpmo_4_5`	A3	9	10.73%–31.71%
`test_minimax_h3_t2v_4gpu`	H100	1	27.93%–27.93%
`test_minimax_h3_t2v_dlo_4gpu`	H100	1	20.69%–20.69%
`test_minimax_h3_ti2v_4gpu`	H100	1	28.87%–28.87%
`test_qwen3_omni`	A3	3	29.51%–72.25%
`test_qwen3_omni_chunk`	A3	6	18.74%–83.47%
`test_qwen3_tts_base`	A3	6	10.55%–55.07%
`test_qwen_image_single_device`	H100	1	11.19%–11.19%
`test_qwen_image_ulysses2_cfg2_cache_dit`	H100	2	12.59%–32.13%
`test_qwen_image_ulysses2_cfg2_vae_patch4`	H100	2	11.53%–32.26%

Percentages compare the archived window means with the historical references. Their causes have not been established by this calibration.

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

…resh Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

congw729 · 2026-10-16T02:01:30Z

Hi @yenuo26, could you help review this performance baseline update when you have a chance? It refreshes existing references and fills missing metric coverage, including local MiniCPM-o OmniInteract. Thanks!

yenuo26 · 2026-10-16T02:28:15Z

@@ -48,7 +48,8 @@
            "throughput_qps": 0.1855,


The actual value from huanyuan already looks significantly better than the baseline, so I suggest optimizing them all.

Thanks for catching this! The local H200 results were missing from my first pass. Updated all four HunyuanImage cases in c497217 using the October 10–16 daily means (seven comparable days, four-decimal rounding): 16 existing values tightened and two missing mean-latency values added. All 882 measured requests completed without errors. The PR description now includes the source revisions and full per-case numbers.

Thanks for the LGTM! A merge-readiness follow-up: main has now merged #9033 (OmniInteract basic sliding window) and #9066 (VAE tiling for single-device HunyuanVideo I2V). Their workload settings differ from the archived calibration runs, so d778567 defers the 18 window-off OmniInteract values and the three untiled HunyuanVideo values. The HunyuanImage H200 updates remain included.

This removes the merge conflict; I verified the merged tree preserves main's settings and all remaining 381 baseline values. The PR is now ready for review, with the description updated. Could you please recheck this small scope reduction on d778567 before merging? Thank you!

yenuo26 · 2026-10-16T02:33:22Z

@@ -47,12 +47,42 @@
        "baseline": {


After the tests\dfx\perf\tests\test_qwen3_omni_multi_replicas.json were adjusted, the baseline was deleted, so it needs to be re-added based on the latest results.

Thanks, agreed that these need recalibration. I checked the latest kanban archive and Buildkite main builds: the newest archived 3×H100 performance sweep is October 10, before #8826 and #8833 merged on October 13. The 90 main builds created after #8833 through October 16 09:00 UTC contain no matching multi-replica performance job.

I've documented the gap in the PR and retained the existing TPOT references. Restoring the 14 audio TTFP/RTF values needs repeated post-#8833, fixed-Ethan 3×H100 results covering c=8/16/24/32 and all three multimodal configurations. Do you have a newer run or artifact for this setup that I can use?

Quick update: I found the new October 17 H100 sweep in build #17479. It runs b164fea797531c1fa10f8b87829632cad006b9fc, includes #8826/#8833, and matches the fixed-Ethan configuration here. All seven scenarios completed all 662 requests, and the archived JSONs match the build artifacts exactly.

I've added the measurements and provenance to the PR description. We now have one comparable day; this refresh requires at least five, so the 14 audio baseline values are still pending. P99 audio TTFP is 6.13 s at c=24 and 8.68 s at c=32, which also warrants checking variation over additional runs. The four existing TPOT references remain unchanged.

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

yenuo26

LGTM

chatgpt-codex-connector · 2026-10-20T08:02:09Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-20T08:02:06.941169Z	`c497217`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-10-20T08:02:12Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

vllm-omni-review-bot · 2026-10-20T09:05:49Z

This PR appears to be related to model: minicpm.

Model owners: @y-null @yenuo26 @NickCao

Routing: @y-null via model owner; @yenuo26 via CI owner, CODEOWNERS; @NickCao via CODEOWNERS

@congw729, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

…m-project#8957) Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com> Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>

congw729 added 2 commits October 15, 2026 11:27

[CI/Build] Refresh non-regressing GPU and A3 performance baselines

1eff69b

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

[CI/Build] Complete confirmed baselines and allow MiniCPM A3 TTFT ref…

20b4123

…resh Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

congw729 changed the title ~~[CI/Build] Refresh non-regressing GPU and A3 performance baselines~~ [CI/Build] Refresh and complete GPU and A3 performance baselines Oct 15, 2026

congw729 changed the title ~~[CI/Build] Refresh and complete GPU and A3 performance baselines~~ [CI/Build] Update performance baselines and fill missing metrics Oct 15, 2026

hsliuustc0106 added the CI/CD codes related to changes to CI/CD label Oct 16, 2026

yenuo26 reviewed Oct 16, 2026

View reviewed changes

[CI/Build] Refresh HunyuanImage H200 performance baselines

c497217

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

This was referenced Oct 16, 2026

[CI/Build][MiniCPM-o] Run the OmniInteract nightly with the basic sliding window #9033

Merged

[Model][MiniCPM-o] Default the duplex Stage-0 window to basic #9035

Merged

yenuo26 approved these changes Oct 20, 2026

View reviewed changes

congw729 marked this pull request as ready for review October 20, 2026 08:02

congw729 requested a review from NickCao as a code owner October 20, 2026 08:02

[CI/Build] Defer baselines for changed upstream workloads

d778567

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>

Merge branch 'main' into codex/perf-baseline-20261015

2c8712d

yenuo26 added the ready label to trigger buildkite CI label Oct 20, 2026

yenuo26 enabled auto-merge (squash) October 20, 2026 09:10

yenuo26 merged commit ef9eded into vllm-project:main Oct 20, 2026
8 of 11 checks passed

Dong1017 mentioned this pull request Oct 23, 2026

[CI/Build] Enable validated MiniMax-H3 A3 CI coverage #9030

Open

Navigation Menu

Repository navigation

[CI/Build] Update performance baselines and fill missing metrics - #8957

[CI/Build] Update performance baselines and fill missing metrics#8957
yenuo26 merged 5 commits into
vllm-project:mainfrom
congw729:codex/perf-baseline-20261015

congw729 commented Oct 15, 2026 •

edited

Loading

Uh oh!

congw729 commented Oct 16, 2026

Uh oh!

yenuo26 Oct 16, 2026

Uh oh!

congw729 Oct 16, 2026

Uh oh!

congw729 Oct 20, 2026

Uh oh!

yenuo26 Oct 16, 2026

Uh oh!

congw729 Oct 16, 2026

Uh oh!

congw729 Oct 20, 2026

Uh oh!

yenuo26 left a comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026

Uh oh!

vllm-omni-review-bot commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

congw729 commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Calibration policy

Deferred coverage

Test Plan

Test Result

Contributor intent (required human answers)

1. What outcome do you want to achieve?

2. What is the highest-leverage variable for that outcome?

3. How will this work move that variable?

4. Whose customer or user needs does this address, and what are those needs?

Changes by file

Baseline values

Source runs

Uh oh!

congw729 commented Oct 16, 2026

Uh oh!

yenuo26 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

congw729 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

congw729 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

yenuo26 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

congw729 Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

congw729 Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

yenuo26 left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Oct 20, 2026

Uh oh!

vllm-omni-review-bot commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

congw729 commented Oct 15, 2026 •

edited

Loading