Skip to content

Update perf baseline base on 7/5-7/11 7 days avg. - #5231

Merged
david6666666 merged 1 commit into
vllm-project:mainfrom
congw729:update-perf-baselines-jul
Jul 21, 2026
Merged

Update perf baseline base on 7/5-7/11 7 days avg.#5231
david6666666 merged 1 commit into
vllm-project:mainfrom
congw729:update-perf-baselines-jul

Conversation

@congw729

@congw729 congw729 commented Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Perf Baseline Update — July 2026

  • Date: 2026-07-21
  • Branch: update-perf-baselines-jul (worktree), commit 75943e72, based on origin/main @ 6cd1ff7e
  • Rule this cycle: new baseline = plain 7-day average of nightly results, 2026-07-05 .. 2026-07-11 (no ×0.95 improvement factor, unlike PR [CI] Update perf json baselines #4289).
    • Qwen3-Omni (all variants), Higgs Audio v3, and WAN2.2 I2V update even when the average regresses vs the current baseline. WAN2.2's regression is judged to be an artifact of last cycle's avg×0.95 tightening, not a real perf loss.
    • All other models update on a regression only if it's within 7% (treated as noise / prior over-tightening); regressions beyond 7% keep the existing baseline.
    • Changes >10% are flagged and summarized in a separate table per model.
  • Reference: previous cycle PR [CI] Update perf json baselines #4289
  • Data: vllm-omni-kanban data/buildkite_nightly_raw/, builds 12082, 12098, 12109, 12135, 12157, 12195, 12205
  • Result: 211 metrics updated across 13 files (see per-model breakdown below). No data in window for cosmos3, ltx2, hunyuan_image3_it2i, hunyuan_image_tp2_cfgp2, hunyuan_image_tp2_sp2, hunyuan_image_tp4 (hunyuan is measured manually on H200; cosmos3/ltx2 configs are new).
  • Follow-up: qwen_image_vllm_omni still keeps 22 metrics that regressed beyond 7%, and the qwen_image_edit family / layered / tts keep a smaller number each — worth investigating as a possible real regression rather than noise, independent of the baseline-update decision.
  • History: commit 0480a0e9 (plain update-everywhere) → a1a027a7 (rebuilt after a main rebase) → 7af3ae53 (allow-list rule: qwen3_omni + higgs only) → 75943e72 (added WAN2.2 override + 7% regression tolerance for all other models). Earlier commits superseded.

Per-model update summary

Model file Updated Kept (regression) No data
test_bagel_vllm_omni.json 16 0 0
test_cosmos3_vllm_omni.json 0 0 12
test_higgs_audio_v3.json 5 0 0
test_hunyuan_image3_it2i.json 0 0 5
test_hunyuan_image_tp2_cfgp2.json 0 0 4
test_hunyuan_image_tp2_sp2.json 0 0 4
test_hunyuan_image_tp4.json 0 0 5
test_qwen3_omni_async_chunk.json 40 0 0
test_qwen3_omni_multi_replicas.json 25 0 0
test_qwen3_omni_no_async_chunk.json 24 0 0
test_qwen3_omni_vllm_text.json 16 0 0
test_qwen_image_edit_2511_vllm_omni.json 10 10 0
test_qwen_image_edit_vllm_omni.json 10 10 0
test_qwen_image_layered_vllm_omni.json 4 4 0
test_qwen_image_vllm_omni.json 11 22 0
test_tts.json 30 17 2
test_voxcpm2.json 11 2 2
test_wan22_i2v_vllm_omni.json 9 0 0

Qwen3-Omni (all variants) and Higgs Audio v3 update even when the average regresses vs the current baseline. All other models only update where the average is as good or better; regressed metrics keep the old baseline.

test_bagel_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
512x512_t2i_steps20 / t2i / random throughput_qps 0.3678 0.4950 0.495 7 updated (better +34.6%)
512x512_t2i_steps20 / t2i / random latency_mean 2.691 2.0238 2.0238 7 updated (better -24.8%)
512x512_t2i_steps20 / t2i / random peak_memory_mb_max 33083.6 29393.4286 29393.4286 7 updated (better -11.2%)
512x512_t2i_steps20 / t2i / random peak_memory_mb_mean 33082.5 29392.7143 29392.7143 7 updated (better -11.2%)
512x512_i2i_steps20 / i2i / random throughput_qps 0.3124 0.3451 0.3451 7 updated (better +10.5%)
512x512_i2i_steps20 / i2i / random latency_mean 3.1686 2.9031 2.9031 7 updated
512x512_i2i_steps20 / i2i / random peak_memory_mb_max 33448.8 30480.2857 30480.2857 7 updated
512x512_i2i_steps20 / i2i / random peak_memory_mb_mean 33447.7 30479.2857 30479.2857 7 updated
512x512_t2i_steps20 / t2i / random throughput_qps 0.3746 0.5048 0.5048 7 updated (better +34.8%)
512x512_t2i_steps20 / t2i / random latency_mean 2.6425 1.9870 1.987 7 updated (better -24.8%)
512x512_t2i_steps20 / t2i / random peak_memory_mb_max 33077.0 29380.0000 29380.0 7 updated (better -11.2%)
512x512_t2i_steps20 / t2i / random peak_memory_mb_mean 33077.0 29380.0000 29380.0 7 updated (better -11.2%)
512x512_i2i_steps20 / i2i / random throughput_qps 0.1349 0.1490 0.149 7 updated (better +10.4%)
512x512_i2i_steps20 / i2i / random latency_mean 7.339 6.7157 6.7157 7 updated
512x512_i2i_steps20 / i2i / random peak_memory_mb_max 33321.2 30574.0000 30574.0 7 updated
512x512_i2i_steps20 / i2i / random peak_memory_mb_mean 33321.2 30574.0000 30574.0 7 updated
test_cosmos3_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
1024x1024_steps4 / t2i / random throughput_qps 1.107 - 1.107 0 no data
1024x1024_steps4 / t2i / random latency_mean 0.8929 - 0.8929 0 no data
1024x1024_steps4 / t2i / random peak_memory_mb_mean 80000.0 - 80000.0 0 no data
1280x720_frames189_steps4 / t2v / random throughput_qps 0.027 - 0.027 0 no data
1280x720_frames189_steps4 / t2v / random latency_mean 36.6824 - 36.6824 0 no data
1280x720_frames189_steps4 / t2v / random peak_memory_mb_mean 30155.4 - 30155.4 0 no data
1280x720_frames189_steps4 / i2v / random throughput_qps 0.027 - 0.027 0 no data
1280x720_frames189_steps4 / i2v / random latency_mean 36.968 - 36.968 0 no data
1280x720_frames189_steps4 / i2v / random peak_memory_mb_mean 30443.6 - 30443.6 0 no data
1280x720_frames189_steps4 / v2v / random throughput_qps 0.027 - 0.027 0 no data
1280x720_frames189_steps4 / v2v / random latency_mean 37.2117 - 37.2117 0 no data
1280x720_frames189_steps4 / v2v / random peak_memory_mb_mean 30135.6 - 30135.6 0 no data
test_higgs_audio_v3.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 3000 223.3584 223.3584 7 updated (better -92.6%)
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.5 0.4393 0.4393 7 updated (better -12.1%)
default_voice / seed-tts-text / c=16 / p=500 median_audio_ttfp_ms[0] 5000 276.5750 276.575 7 updated (better -94.5%)
default_voice / seed-tts-text / c=16 / p=500 median_audio_rtf[0] 1.2 0.5073 0.5073 7 updated (better -57.7%)
default_voice / seed-tts-text / c=16 / p=500 audio_throughput[0] 30 27.3728 27.3728 7 updated (WORSE -8.8%)
test_hunyuan_image3_it2i.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
inline_dataset_5_requests / ti2i / custom throughput_qps 0.1874 - 0.1874 0 no data
inline_dataset_5_requests / ti2i / custom latency_p99 101.852 - 101.852 0 no data
inline_dataset_5_requests / ti2i / custom latency_mean 79.0385 - 79.0385 0 no data
inline_dataset_5_requests / ti2i / custom peak_memory_mb_max 70000 - 70000 0 no data
inline_dataset_5_requests / ti2i / custom peak_memory_mb_mean 70000 - 70000 0 no data
test_hunyuan_image_tp2_cfgp2.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
1024x1024_steps8 / t2i / random throughput_qps 0.21 - 0.21 0 no data
1024x1024_steps8 / t2i / random latency_p99 4.7469 - 4.7469 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_max 101100 - 101100 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_mean 101100 - 101100 0 no data
test_hunyuan_image_tp2_sp2.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
1024x1024_steps8 / t2i / random throughput_qps 0.2 - 0.2 0 no data
1024x1024_steps8 / t2i / random latency_p99 5.1025 - 5.1025 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_max 97402 - 97402 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_mean 97402 - 97402 0 no data
test_hunyuan_image_tp4.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
1024x1024_steps8 / t2i / random throughput_qps 0.213 - 0.213 0 no data
1024x1024_steps8 / t2i / random latency_mean 4.6953 - 4.6953 0 no data
1024x1024_steps8 / t2i / random latency_p99 4.8601 - 4.8601 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_max 56912.0 - 56912.0 0 no data
1024x1024_steps8 / t2i / random peak_memory_mb_mean 56912.0 - 56912.0 0 no data
test_qwen3_omni_async_chunk.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
random / c=1 / p=4 mean_ttft_ms[0] 135.4379 96.4061 96.4061 7 updated (better -28.8%)
random / c=1 / p=4 mean_e2el_ms[0] 17619.9851 18507.4582 18507.4582 7 updated (WORSE +5.0%)
random / c=1 / p=4 mean_audio_rtf[0] 0.1279 0.1175 0.1175 7 updated
random / c=4 / p=16 mean_ttft_ms[1] 187.0896 140.8147 140.8147 7 updated (better -24.7%)
random / c=4 / p=16 mean_e2el_ms[1] 34811.8335 28365.4956 28365.4956 7 updated (better -18.5%)
random / c=4 / p=16 mean_audio_rtf[1] 0.2238 0.1800 0.18 7 updated (better -19.6%)
random / c=8 / p=32 mean_ttft_ms[2] 400 271.9025 271.9025 7 updated (better -32.0%)
random / c=8 / p=32 mean_e2el_ms[2] 43656.9501 31907.9613 31907.9613 7 updated (better -26.9%)
random / c=8 / p=32 mean_audio_rtf[2] 0.2735 0.2359 0.2359 7 updated (better -13.8%)
random / c=16 / p=64 mean_ttft_ms[3] 435.675 362.2916 362.2916 7 updated (better -16.8%)
random / c=16 / p=64 mean_e2el_ms[3] 56657.1548 48161.5829 48161.5829 7 updated (better -15.0%)
random / c=16 / p=64 mean_audio_rtf[3] 0.3712 0.3278 0.3278 7 updated (better -11.7%)
random / c=32 / p=128 mean_ttft_ms[4] 724.6836 507.7760 507.776 7 updated (better -29.9%)
random / c=32 / p=128 mean_e2el_ms[4] 85097.6016 72630.8601 72630.8601 7 updated (better -14.6%)
random / c=32 / p=128 mean_audio_rtf[4] 0.5634 0.6296 0.6296 7 updated (WORSE +11.7%)
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 340.329 354.0951 354.0951 7 updated (WORSE +4.0%)
random-mm / qps=0.1 / p=10 mean_e2el_ms[0] 4305.477 3952.5615 3952.5615 7 updated
random-mm / qps=0.1 / p=10 mean_audio_rtf[0] 0.1305 0.1170 0.117 7 updated (better -10.4%)
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 220.0909 259.6317 259.6317 7 updated (WORSE +18.0%)
random-mm / qps=0.5 / p=40 mean_e2el_ms[0] 5475.9565 4253.3009 4253.3009 7 updated (better -22.3%)
random-mm / qps=0.5 / p=40 mean_audio_rtf[0] 0.1818 0.1448 0.1448 7 updated (better -20.4%)
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 450.0155 457.4453 457.4453 7 updated (WORSE +1.7%)
random-mm / qps=1.0 / p=100 mean_e2el_ms[0] 8516.8316 5912.5623 5912.5623 7 updated (better -30.6%)
random-mm / qps=1.0 / p=100 mean_audio_rtf[0] 0.2717 0.1967 0.1967 7 updated (better -27.6%)
random / c=1 / p=4 mean_ttft_ms[0] 78.4278 90.8033 90.8033 7 updated (WORSE +15.8%)
random / c=1 / p=4 mean_e2el_ms[0] 4288.3029 4517.3411 4517.3411 7 updated (WORSE +5.3%)
random / c=4 / p=16 mean_ttft_ms[1] 195.8037 202.1779 202.1779 7 updated (WORSE +3.3%)
random / c=4 / p=16 mean_e2el_ms[1] 7429.9429 7857.9711 7857.9711 7 updated (WORSE +5.8%)
random / c=8 / p=32 mean_ttft_ms[2] 333.7318 326.5864 326.5864 7 updated
random / c=8 / p=32 mean_e2el_ms[2] 9451.6184 9861.1687 9861.1687 7 updated (WORSE +4.3%)
random / c=16 / p=64 mean_ttft_ms[3] 561.805 544.8914 544.8914 7 updated
random / c=16 / p=64 mean_e2el_ms[3] 12607.5292 13283.3363 13283.3363 7 updated (WORSE +5.4%)
random / c=32 / p=128 mean_ttft_ms[4] 1038.043 961.3929 961.3929 7 updated
random / c=32 / p=128 mean_e2el_ms[4] 18153.2547 19541.0736 19541.0736 7 updated (WORSE +7.6%)
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 275.3875 318.0480 318.048 7 updated (WORSE +15.5%)
random-mm / qps=0.1 / p=10 mean_e2el_ms[0] 730.6995 807.6114 807.6114 7 updated (WORSE +10.5%)
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 166.2201 186.1680 186.168 7 updated (WORSE +12.0%)
random-mm / qps=0.5 / p=40 mean_e2el_ms[0] 673.5789 726.1137 726.1137 7 updated (WORSE +7.8%)
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 357.7781 375.5282 375.5282 7 updated (WORSE +5.0%)
random-mm / qps=1.0 / p=100 mean_e2el_ms[0] 953.3493 1006.1663 1006.1663 7 updated (WORSE +5.5%)
test_qwen3_omni_multi_replicas.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
random / c=8 / p=128 mean_ttft_ms[0] 203 130.5751 130.5751 6 updated (better -35.7%)
random / c=8 / p=128 mean_tpot_ms[0] 28 7.9854 7.9854 6 updated (better -71.5%)
random / c=8 / p=128 mean_audio_ttfp_ms[0] 861 486.4860 486.486 6 updated (better -43.5%)
random / c=8 / p=128 mean_audio_rtf[0] 0.24 0.1896 0.1896 6 updated (better -21.0%)
random / c=16 / p=128 mean_ttft_ms[1] 283 278.6469 278.6469 6 updated
random / c=16 / p=128 mean_tpot_ms[1] 36 11.1209 11.1209 6 updated (better -69.1%)
random / c=16 / p=128 mean_audio_ttfp_ms[1] 1200 954.1791 954.1791 6 updated (better -20.5%)
random / c=16 / p=128 mean_audio_rtf[1] 0.3 0.3962 0.3962 6 updated (WORSE +32.1%)
random / c=24 / p=128 mean_ttft_ms[2] 470 324.7249 324.7249 6 updated (better -30.9%)
random / c=24 / p=128 mean_tpot_ms[2] 41 16.1507 16.1507 6 updated (better -60.6%)
random / c=24 / p=128 mean_audio_ttfp_ms[2] 1528 1289.6514 1289.6514 6 updated (better -15.6%)
random / c=24 / p=128 mean_audio_rtf[2] 0.35 0.4098 0.4098 6 updated (WORSE +17.1%)
random / c=32 / p=128 mean_ttft_ms[3] 869 911.6085 911.6085 6 updated (WORSE +4.9%)
random / c=32 / p=128 mean_tpot_ms[3] 45 20.7485 20.7485 6 updated (better -53.9%)
random / c=32 / p=128 mean_audio_ttfp_ms[3] 2100 2196.8275 2196.8275 6 updated (WORSE +4.6%)
random / c=32 / p=128 mean_audio_rtf[3] 0.39 0.7322 0.7322 6 updated (WORSE +87.7%)
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 377 340.5122 340.5122 6 updated
random-mm / qps=0.1 / p=10 mean_audio_ttfp_ms[0] 684 463.0752 463.0752 6 updated (better -32.3%)
random-mm / qps=0.1 / p=10 mean_audio_rtf[0] 0.14 0.1153 0.1153 6 updated (better -17.6%)
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 228 236.9260 236.926 6 updated (WORSE +3.9%)
random-mm / qps=0.5 / p=40 mean_audio_ttfp_ms[0] 686 395.2182 395.2182 6 updated (better -42.4%)
random-mm / qps=0.5 / p=40 mean_audio_rtf[0] 0.16 0.1259 0.1259 6 updated (better -21.3%)
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 522 470.6126 470.6126 6 updated
random-mm / qps=1.0 / p=100 mean_audio_ttfp_ms[0] 1119 665.5474 665.5474 6 updated (better -40.5%)
random-mm / qps=1.0 / p=100 mean_audio_rtf[0] 0.21 0.1660 0.166 6 updated (better -21.0%)
test_qwen3_omni_no_async_chunk.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
random / c=1 / p=4 mean_ttft_ms[0] 109.1218 115.8348 115.8348 7 updated (WORSE +6.2%)
random / c=1 / p=4 mean_e2el_ms[0] 22413.667 25070.9523 25070.9523 7 updated (WORSE +11.9%)
random / c=1 / p=4 mean_audio_rtf[0] 0.1695 0.1690 0.169 7 updated
random / c=4 / p=16 mean_ttft_ms[1] 247.4411 173.1823 173.1823 7 updated (better -30.0%)
random / c=4 / p=16 mean_e2el_ms[1] 30808.9715 31992.0646 31992.0646 7 updated (WORSE +3.8%)
random / c=4 / p=16 mean_audio_rtf[1] 0.2033 0.2111 0.2111 7 updated (WORSE +3.8%)
random / c=8 / p=32 mean_ttft_ms[2] 354.9384 446.6720 446.672 7 updated (WORSE +25.8%)
random / c=8 / p=32 mean_e2el_ms[2] 35677.844 37619.7669 37619.7669 7 updated (WORSE +5.4%)
random / c=8 / p=32 mean_audio_rtf[2] 0.2467 0.2475 0.2475 7 updated (WORSE +0.3%)
random / c=16 / p=64 mean_ttft_ms[3] 447.7885 464.5985 464.5985 7 updated (WORSE +3.8%)
random / c=16 / p=64 mean_e2el_ms[3] 46345.8978 50393.8257 50393.8257 7 updated (WORSE +8.7%)
random / c=16 / p=64 mean_audio_rtf[3] 0.3234 0.3551 0.3551 7 updated (WORSE +9.8%)
random / c=32 / p=128 mean_ttft_ms[4] 694.7487 3420.0634 3420.0634 7 updated (WORSE +392.3%)
random / c=32 / p=128 mean_e2el_ms[4] 68350.4853 136619.2743 136619.2743 7 updated (WORSE +99.9%)
random / c=32 / p=128 mean_audio_rtf[4] 0.4438 0.9266 0.9266 7 updated (WORSE +108.8%)
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 323.3573 343.1377 343.1377 7 updated (WORSE +6.1%)
random-mm / qps=0.1 / p=10 mean_e2el_ms[0] 4623.4173 4811.1123 4811.1123 7 updated (WORSE +4.1%)
random-mm / qps=0.1 / p=10 mean_audio_rtf[0] 0.1406 0.1463 0.1463 7 updated (WORSE +4.1%)
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 215.4502 245.5446 245.5446 7 updated (WORSE +14.0%)
random-mm / qps=0.5 / p=40 mean_e2el_ms[0] 4924.149 5019.1256 5019.1256 7 updated (WORSE +1.9%)
random-mm / qps=0.5 / p=40 mean_audio_rtf[0] 0.1637 0.1713 0.1713 7 updated (WORSE +4.6%)
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 442.5815 506.1407 506.1407 7 updated (WORSE +14.4%)
random-mm / qps=1.0 / p=100 mean_e2el_ms[0] 6441.0986 6814.9656 6814.9656 7 updated (WORSE +5.8%)
random-mm / qps=1.0 / p=100 mean_audio_rtf[0] 0.2091 0.2213 0.2213 7 updated (WORSE +5.8%)
test_qwen3_omni_vllm_text.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
random / c=1 / p=4 mean_ttft_ms[0] 69.1276 69.8081 69.8081 7 updated (WORSE +1.0%)
random / c=1 / p=4 mean_e2el_ms[0] 4382.7211 4720.8225 4720.8225 7 updated (WORSE +7.7%)
random / c=4 / p=16 mean_ttft_ms[1] 255.1882 173.0651 173.0651 7 updated (better -32.2%)
random / c=4 / p=16 mean_e2el_ms[1] 7510.3815 7937.2532 7937.2532 7 updated (WORSE +5.7%)
random / c=8 / p=32 mean_ttft_ms[2] 310.639 271.8108 271.8108 7 updated (better -12.5%)
random / c=8 / p=32 mean_e2el_ms[2] 9440.4854 9962.3566 9962.3566 7 updated (WORSE +5.5%)
random / c=16 / p=64 mean_ttft_ms[3] 422.0337 413.4079 413.4079 7 updated
random / c=16 / p=64 mean_e2el_ms[3] 12553.0186 13724.5479 13724.5479 7 updated (WORSE +9.3%)
random / c=32 / p=128 mean_ttft_ms[4] 532.0582 497.2546 497.2546 7 updated
random / c=32 / p=128 mean_e2el_ms[4] 16932.6065 18324.0910 18324.091 7 updated (WORSE +8.2%)
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 26.3381 25.9477 25.9477 7 updated
random-mm / qps=0.1 / p=10 mean_e2el_ms[0] 485.2436 517.3155 517.3155 7 updated (WORSE +6.6%)
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 26.4517 26.7056 26.7056 7 updated (WORSE +1.0%)
random-mm / qps=0.5 / p=40 mean_e2el_ms[0] 517.2017 551.3080 551.308 7 updated (WORSE +6.6%)
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 27.6972 27.6906 27.6906 7 updated
random-mm / qps=1.0 / p=100 mean_e2el_ms[0] 561.3684 596.5284 596.5284 7 updated (WORSE +6.3%)
test_qwen_image_edit_2511_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
512x512_steps20_i2i_2img / i2i / random throughput_qps 0.0721 0.0631 0.0721 7 kept (regression -12.5%)
512x512_steps20_i2i_2img / i2i / random latency_mean 13.8398 15.8953 13.8398 7 kept (regression +14.9%)
512x512_steps20_i2i_2img / i2i / random peak_memory_mb_max 54731.4 57610.0000 57610.0 7 updated (within 7% tolerance, WORSE +5.3%)
512x512_steps20_i2i_2img / i2i / random peak_memory_mb_mean 54731.4 57610.0000 57610.0 7 updated (within 7% tolerance, WORSE +5.3%)
1536x1536_steps35_i2i_2img / i2i / random throughput_qps 0.0186 0.0166 0.0186 7 kept (regression -10.9%)
1536x1536_steps35_i2i_2img / i2i / random latency_mean 53.6942 60.4930 53.6942 7 kept (regression +12.7%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_max 64402.4 67790.0000 67790.0 7 updated (within 7% tolerance, WORSE +5.3%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_mean 64402.4 67790.0000 67790.0 7 updated (within 7% tolerance, WORSE +5.3%)
1536x1536_steps35_i2i_2img / i2i / random throughput_qps 0.0538 0.0487 0.0538 7 kept (regression -9.6%)
1536x1536_steps35_i2i_2img / i2i / random latency_mean 18.5591 20.5823 18.5591 7 kept (regression +10.9%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_max 53485.0 56449.4286 56449.4286 7 updated (within 7% tolerance, WORSE +5.5%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_mean 53485.0 56449.4286 56449.4286 7 updated (within 7% tolerance, WORSE +5.5%)
512x512_steps20_i2i_2img / i2i / random throughput_qps 0.2776 0.2496 0.2776 7 kept (regression -10.1%)
512x512_steps20_i2i_2img / i2i / random latency_mean 3.5938 4.0114 3.5938 7 kept (regression +11.6%)
512x512_steps20_i2i_2img / i2i / random peak_memory_mb_max 55214.0 58113.7143 58113.7143 7 updated (within 7% tolerance, WORSE +5.3%)
512x512_steps20_i2i_2img / i2i / random peak_memory_mb_mean 55214.0 58075.1429 58075.1429 7 updated (within 7% tolerance, WORSE +5.2%)
1536x1536_steps35_i2i_2img / i2i / random throughput_qps 0.1051 0.0963 0.1051 7 kept (regression -8.4%)
1536x1536_steps35_i2i_2img / i2i / random latency_mean 9.4918 10.3935 9.4918 7 kept (regression +9.5%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_max 64265.6 68107.4286 68107.4286 7 updated (within 7% tolerance, WORSE +6.0%)
1536x1536_steps35_i2i_2img / i2i / random peak_memory_mb_mean 64265.6 68107.4286 68107.4286 7 updated (within 7% tolerance, WORSE +6.0%)
test_qwen_image_edit_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
512x512_steps20_i2i / i2i / random throughput_qps 0.1129 0.0980 0.1129 7 kept (regression -13.2%)
512x512_steps20_i2i / i2i / random latency_mean 8.8352 10.2447 8.8352 7 kept (regression +16.0%)
512x512_steps20_i2i / i2i / random peak_memory_mb_max 55356.5 58572.0000 58572.0 7 updated (within 7% tolerance, WORSE +5.8%)
512x512_steps20_i2i / i2i / random peak_memory_mb_mean 55356.5 58572.0000 58572.0 7 updated (within 7% tolerance, WORSE +5.8%)
1536x1536_steps35_i2i / i2i / random throughput_qps 0.024 0.0213 0.024 7 kept (regression -11.4%)
1536x1536_steps35_i2i / i2i / random latency_mean 41.5686 47.1148 41.5686 7 kept (regression +13.3%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_max 62762.7 66368.0000 66368.0 7 updated (within 7% tolerance, WORSE +5.7%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_mean 62762.7 66368.0000 66368.0 7 updated (within 7% tolerance, WORSE +5.7%)
1536x1536_steps35_i2i / i2i / random throughput_qps 0.0672 0.0600 0.0672 7 kept (regression -10.7%)
1536x1536_steps35_i2i / i2i / random latency_mean 14.8517 16.6924 14.8517 7 kept (regression +12.4%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_max 53931.5 57076.0000 57076.0 7 updated (within 7% tolerance, WORSE +5.8%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_mean 53931.5 57076.0000 57076.0 7 updated (within 7% tolerance, WORSE +5.8%)
512x512_steps20_i2i / i2i / random throughput_qps 0.2879 0.2603 0.2879 7 kept (regression -9.6%)
512x512_steps20_i2i / i2i / random latency_mean 3.4701 3.8532 3.4701 7 kept (regression +11.0%)
512x512_steps20_i2i / i2i / random peak_memory_mb_max 55525.6 58758.0000 58758.0 7 updated (within 7% tolerance, WORSE +5.8%)
512x512_steps20_i2i / i2i / random peak_memory_mb_mean 55525.6 58758.0000 58758.0 7 updated (within 7% tolerance, WORSE +5.8%)
1536x1536_steps35_i2i / i2i / random throughput_qps 0.1113 0.1005 0.1113 7 kept (regression -9.7%)
1536x1536_steps35_i2i / i2i / random latency_mean 8.9656 9.9633 8.9656 7 kept (regression +11.1%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_max 63758.3 67424.0000 67424.0 7 updated (within 7% tolerance, WORSE +5.7%)
1536x1536_steps35_i2i / i2i / random peak_memory_mb_mean 63758.3 67424.0000 67424.0 7 updated (within 7% tolerance, WORSE +5.7%)
test_qwen_image_layered_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
640x640_steps20_i2i / i2i / random throughput_qps 0.0703 0.0622 0.0703 7 kept (regression -11.6%)
640x640_steps20_i2i / i2i / random latency_mean 14.208 16.1354 14.208 7 kept (regression +13.6%)
640x640_steps20_i2i / i2i / random peak_memory_mb_max 60543.5 63225.1429 63225.1429 7 updated (within 7% tolerance, WORSE +4.4%)
640x640_steps20_i2i / i2i / random peak_memory_mb_mean 60543.5 63225.1429 63225.1429 7 updated (within 7% tolerance, WORSE +4.4%)
1024x1024_steps35_i2i / i2i / random throughput_qps 0.0417 0.0365 0.0417 7 kept (regression -12.5%)
1024x1024_steps35_i2i / i2i / random latency_mean 23.914 27.5044 23.914 7 kept (regression +15.0%)
1024x1024_steps35_i2i / i2i / random peak_memory_mb_max 60543.5 63225.1429 63225.1429 7 updated (within 7% tolerance, WORSE +4.4%)
1024x1024_steps35_i2i / i2i / random peak_memory_mb_mean 60543.5 63225.1429 63225.1429 7 updated (within 7% tolerance, WORSE +4.4%)
test_qwen_image_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
512x512_steps20 / t2i / random throughput_qps 0.4153 0.3095 0.4153 7 kept (regression -25.5%)
512x512_steps20 / t2i / random latency_mean 2.4065 3.3027 2.4065 7 kept (regression +37.2%)
512x512_steps20 / t2i / random peak_memory_mb_mean 52289.9 55121.4286 55121.4286 7 updated (within 7% tolerance, WORSE +5.4%)
1536x1536_steps35 / t2i / random throughput_qps 0.0423 0.0372 0.0423 7 kept (regression -11.9%)
1536x1536_steps35 / t2i / random latency_mean 23.5991 26.9167 23.5991 7 kept (regression +14.1%)
1536x1536_steps35 / t2i / random peak_memory_mb_mean 61360.5 65118.0000 65118.0 7 updated (within 7% tolerance, WORSE +6.1%)
512x512_steps20 / t2i / random throughput_qps 0.4175 0.2834 0.4175 7 kept (regression -32.1%)
512x512_steps20 / t2i / random latency_mean 2.3953 3.5984 2.3953 7 kept (regression +50.2%)
512x512_steps20 / t2i / random peak_memory_mb_mean 56000 55088.5714 55088.5714 7 updated
1536x1536_steps35 / t2i / random throughput_qps 0.0423 0.0372 0.0423 7 kept (regression -12.1%)
1536x1536_steps35 / t2i / random latency_mean 23.5768 26.9599 23.5768 7 kept (regression +14.3%)
1536x1536_steps35 / t2i / random peak_memory_mb_mean 65000 66012.2857 66012.2857 7 updated (within 7% tolerance, WORSE +1.6%)
1536x1536_steps35 / t2i / random throughput_qps 0.1214 0.1081 0.1214 7 kept (regression -10.9%)
1536x1536_steps35 / t2i / random latency_mean 8.2211 9.2649 8.2211 7 kept (regression +12.7%)
1536x1536_steps35 / t2i / random peak_memory_mb_mean 51704.7 54501.1429 54501.1429 7 updated (within 7% tolerance, WORSE +5.4%)
512x512_steps20 / t2i / random throughput_qps 0.559 0.4656 0.559 7 kept (regression -16.7%)
512x512_steps20 / t2i / random latency_mean 1.794 2.1822 1.794 7 kept (regression +21.6%)
512x512_steps20 / t2i / random peak_memory_mb_mean 52397.6775 55290.5714 55290.5714 7 updated (within 7% tolerance, WORSE +5.5%)
1536x1536_steps35 / t2i / random throughput_qps 0.1918 0.1724 0.1918 7 kept (regression -10.1%)
1536x1536_steps35 / t2i / random latency_mean 5.2022 5.8129 5.2022 7 kept (regression +11.7%)
1536x1536_steps35 / t2i / random peak_memory_mb_mean 61468.8 64838.8571 64838.8571 7 updated (within 7% tolerance, WORSE +5.5%)
512x512_steps20_high_concurrency / t2i / random throughput_qps[0] 0.4125 0.3045 0.4125 7 kept (regression -26.2%)
512x512_steps20_high_concurrency / t2i / random latency_mean[0] 2.4215 3.3944 2.4215 7 kept (regression +40.2%)
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[0] 67000 55147.4286 55147.4286 7 updated (better -17.7%)
512x512_steps20_high_concurrency / t2i / random throughput_qps[1] 0.6714 0.5595 0.6714 7 kept (regression -16.7%)
512x512_steps20_high_concurrency / t2i / random latency_mean[1] 2.992 3.6149 2.992 7 kept (regression +20.8%)
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[1] 67000 55148.0000 55148.0 7 updated (better -17.7%)
512x512_steps20_high_concurrency / t2i / random throughput_qps[2] 0.6674 0.5648 0.6674 7 kept (regression -15.4%)
512x512_steps20_high_concurrency / t2i / random latency_mean[2] 5.9293 7.0315 5.9293 7 kept (regression +18.6%)
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[2] 67000 55150.4643 55150.4643 7 updated (better -17.7%)
512x512_steps20_high_concurrency / t2i / random throughput_qps[3] 0.8159 0.6966 0.8159 7 kept (regression -14.6%)
512x512_steps20_high_concurrency / t2i / random latency_mean[3] 9.656 11.3963 9.656 7 kept (regression +18.0%)
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[3] 67000 55152.3821 55152.3821 7 updated (better -17.7%)
test_tts.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
voice_clone / seed-tts / c=1 / p=20 median_audio_ttfp_ms[0] 139.9434 141.2791 141.2791 7 updated (within 7% tolerance, WORSE +1.0%)
voice_clone / seed-tts / c=1 / p=20 median_audio_rtf[0] 0.1683 0.1591 0.1591 7 updated
voice_clone / seed-tts / c=8 / p=80 median_audio_ttfp_ms[0] 227.5653 226.7470 226.747 7 updated
voice_clone / seed-tts / c=8 / p=80 median_audio_rtf[0] 0.3313 0.3597 0.3313 7 kept (regression +8.6%)
voice_clone / seed-tts / c=8 / p=80 audio_throughput[0] 23.501 24.2296 24.2296 7 updated
voice_clone / seed-tts / c=16 / p=128 median_audio_ttfp_ms[1] 647.1272 799.3196 647.1272 7 kept (regression +23.5%)
voice_clone / seed-tts / c=16 / p=128 median_audio_rtf[1] 0.3642 0.4694 0.3642 7 kept (regression +28.9%)
voice_clone / seed-tts / c=16 / p=128 audio_throughput[1] 40.634 40.3552 40.3552 7 updated (within 7% tolerance, WORSE -0.7%)
voice_clone / seed-tts / c=64 / p=128 median_audio_ttfp_ms[2] 4777.2714 5726.9927 4777.2714 7 kept (regression +19.9%)
voice_clone / seed-tts / c=64 / p=128 median_audio_rtf[2] 1.2282 1.5709 1.2282 7 kept (regression +27.9%)
voice_clone / seed-tts / c=64 / p=128 audio_throughput[2] 43.0758 43.2572 43.2572 7 updated
voice_clone / seed-tts / c=4 / p=200 mean_audio_rtf[0] 0.45 - 0.45 0 no data
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 49.7908 46.8434 46.8434 7 updated
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.1425 0.1367 0.1367 7 updated
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 75.3341 87.9854 75.3341 7 kept (regression +16.8%)
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.1859 0.2277 0.1859 7 kept (regression +22.5%)
default_voice / seed-tts-text / c=8 / p=80 audio_throughput[0] 36.465 38.3858 38.3858 7 updated
default_voice / seed-tts-text / c=16 / p=128 median_audio_ttfp_ms[1] 713.7727 831.8829 713.7727 7 kept (regression +16.5%)
default_voice / seed-tts-text / c=16 / p=128 median_audio_rtf[1] 0.3148 0.3958 0.3148 7 kept (regression +25.7%)
default_voice / seed-tts-text / c=16 / p=128 audio_throughput[1] 49.6268 55.3523 55.3523 7 updated (better +11.5%)
default_voice / seed-tts-text / c=64 / p=128 median_audio_ttfp_ms[2] 5941.7179 6691.4531 5941.7179 7 kept (regression +12.6%)
default_voice / seed-tts-text / c=64 / p=128 median_audio_rtf[2] 1.1662 1.3965 1.1662 7 kept (regression +19.7%)
default_voice / seed-tts-text / c=64 / p=128 audio_throughput[2] 48.3091 68.7032 68.7032 7 updated (better +42.2%)
default_voice / seed-tts-text / c=4 / p=200 mean_audio_rtf[0] 0.35 - 0.35 0 no data
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_ttfp_ms[0] 56.6977 56.7460 56.746 7 updated (within 7% tolerance, WORSE +0.1%)
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_rtf[0] 0.1555 0.1571 0.1571 7 updated (within 7% tolerance, WORSE +1.0%)
default_voice / seed-tts-text / qps=2.0 / p=100 audio_throughput[0] 11.5292 11.1226 11.1226 7 updated (within 7% tolerance, WORSE -3.5%)
voice_design / seed-tts-design / c=1 / p=20 median_audio_ttfp_ms[0] 52.0447 46.3479 46.3479 7 updated (better -10.9%)
voice_design / seed-tts-design / c=1 / p=20 median_audio_rtf[0] 0.1452 0.1352 0.1352 7 updated
voice_design / seed-tts-design / c=8 / p=80 median_audio_ttfp_ms[0] 75.5967 85.4666 75.5967 7 kept (regression +13.1%)
voice_design / seed-tts-design / c=8 / p=80 median_audio_rtf[0] 0.1871 0.2255 0.1871 7 kept (regression +20.5%)
voice_design / seed-tts-design / c=8 / p=80 audio_throughput[0] 40.2198 38.9378 38.9378 7 updated (within 7% tolerance, WORSE -3.2%)
voice_design / seed-tts-design / c=16 / p=128 median_audio_ttfp_ms[1] 587.5877 712.2530 587.5877 7 kept (regression +21.2%)
voice_design / seed-tts-design / c=16 / p=128 median_audio_rtf[1] 0.3143 0.3914 0.3143 7 kept (regression +24.5%)
voice_design / seed-tts-design / c=16 / p=128 audio_throughput[1] 48.9046 55.1846 55.1846 7 updated (better +12.8%)
voice_design / seed-tts-design / c=64 / p=128 median_audio_ttfp_ms[2] 4960.3726 6193.6563 4960.3726 7 kept (regression +24.9%)
voice_design / seed-tts-design / c=64 / p=128 median_audio_rtf[2] 1.1703 1.4849 1.1703 7 kept (regression +26.9%)
voice_design / seed-tts-design / c=64 / p=128 audio_throughput[2] 48.0505 71.5708 71.5708 7 updated (better +48.9%)
voice_design / seed-tts-design / qps=2.0 / p=100 median_audio_ttfp_ms[0] 56.3961 55.5029 55.5029 7 updated
voice_design / seed-tts-design / qps=2.0 / p=100 median_audio_rtf[0] 0.1542 0.1508 0.1508 7 updated
voice_design / seed-tts-design / qps=2.0 / p=100 audio_throughput[0] 9.6672 9.4885 9.4885 7 updated (within 7% tolerance, WORSE -1.8%)
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 150 82.1886 82.1886 7 updated (better -45.2%)
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.15 0.1241 0.1241 7 updated (better -17.3%)
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 1500 160.9168 160.9168 7 updated (better -89.3%)
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.3 0.1907 0.1907 7 updated (better -36.4%)
default_voice / seed-tts-text / c=8 / p=80 audio_throughput[0] 30.0 39.2332 39.2332 7 updated (better +30.8%)
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_ttfp_ms[0] 3000 100.1265 100.1265 7 updated (better -96.7%)
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_rtf[0] 0.25 0.1437 0.1437 7 updated (better -42.5%)
default_voice / seed-tts-text / qps=2.0 / p=100 audio_throughput[0] 4.0 11.7605 11.7605 7 updated (better +194.0%)
test_voxcpm2.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
voice_clone / seed-tts / c=1 / p=20 median_audio_ttfp_ms[0] 130.2115 137.6450 137.645 5 updated (within 7% tolerance, WORSE +5.7%)
voice_clone / seed-tts / c=1 / p=20 median_audio_rtf[0] 0.0488 0.0497 0.0497 5 updated (within 7% tolerance, WORSE +1.8%)
voice_clone / seed-tts / c=8 / p=80 median_audio_ttfp_ms[0] 475.3357 656.7823 475.3357 5 kept (regression +38.2%)
voice_clone / seed-tts / c=8 / p=80 median_audio_rtf[0] 0.1671 0.1243 0.1243 5 updated (better -25.6%)
voice_clone / seed-tts / c=8 / p=80 audio_throughput[0] 22.8717 36.6358 36.6358 5 updated (better +60.2%)
voice_clone / seed-tts / c=4 / p=200 mean_audio_rtf[0] 0.45 - 0.45 0 no data
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 75.7427 79.6927 79.6927 5 updated (within 7% tolerance, WORSE +5.2%)
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.0419 0.0423 0.0423 5 updated (within 7% tolerance, WORSE +0.9%)
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 351.5855 262.6212 262.6212 5 updated (better -25.3%)
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.1207 0.0753 0.0753 5 updated (better -37.6%)
default_voice / seed-tts-text / c=8 / p=80 audio_throughput[0] 73.7632 103.5843 103.5843 5 updated (better +40.4%)
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_ttfp_ms[0] 80.1095 109.0520 80.1095 5 kept (regression +36.1%)
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_rtf[0] 0.0485 0.0479 0.0479 5 updated
default_voice / seed-tts-text / qps=2.0 / p=100 audio_throughput[0] 18.3715 17.2918 17.2918 5 updated (within 7% tolerance, WORSE -5.9%)
default_voice / seed-tts-text / c=4 / p=200 mean_audio_rtf[0] 0.35 - 0.35 0 no data
test_wan22_i2v_vllm_omni.json
Config Metric Old baseline Jul 5-11 avg New baseline n Status
832x480_frames81_steps4 / i2v / random throughput_qps 0.0465 0.0434 0.0434 7 updated (WORSE -6.6%)
832x480_frames81_steps4 / i2v / random latency_mean 21.4354 23.0654 23.0654 7 updated (WORSE +7.6%)
832x480_frames81_steps4 / i2v / random peak_memory_mb_mean 72557.58 76360.0000 76360.0 7 updated (WORSE +5.2%)
832x480_frames81_steps4 / i2v / random throughput_qps 0.0642 0.0606 0.0606 7 updated (WORSE -5.6%)
832x480_frames81_steps4 / i2v / random latency_mean 15.5589 16.4928 16.4928 7 updated (WORSE +6.0%)
832x480_frames81_steps4 / i2v / random peak_memory_mb_mean 45742.88 47164.8000 47164.8 7 updated (WORSE +3.1%)
1280x720_frames121_steps4 / i2v / random throughput_qps 0.0128 0.0124 0.0124 7 updated (WORSE -3.2%)
1280x720_frames121_steps4 / i2v / random latency_mean 77.8565 80.7784 80.7784 7 updated (WORSE +3.8%)
1280x720_frames121_steps4 / i2v / random peak_memory_mb_mean 53947.84 55374.3429 55374.3429 7 updated (WORSE +2.6%)

Large changes (>10%) by model

test_bagel_vllm_omni.json (10 changes >10%)
Config Metric Old New Diff
512x512_t2i_steps20 / t2i / random throughput_qps 0.3678 0.495 better +34.6%
512x512_t2i_steps20 / t2i / random latency_mean 2.691 2.0238 better -24.8%
512x512_t2i_steps20 / t2i / random peak_memory_mb_max 33083.6 29393.4286 better -11.2%
512x512_t2i_steps20 / t2i / random peak_memory_mb_mean 33082.5 29392.7143 better -11.2%
512x512_i2i_steps20 / i2i / random throughput_qps 0.3124 0.3451 better +10.5%
512x512_t2i_steps20 / t2i / random throughput_qps 0.3746 0.5048 better +34.8%
512x512_t2i_steps20 / t2i / random latency_mean 2.6425 1.987 better -24.8%
512x512_t2i_steps20 / t2i / random peak_memory_mb_max 33077.0 29380.0 better -11.2%
512x512_t2i_steps20 / t2i / random peak_memory_mb_mean 33077.0 29380.0 better -11.2%
512x512_i2i_steps20 / i2i / random throughput_qps 0.1349 0.149 better +10.4%
test_higgs_audio_v3.json (4 changes >10%)
Config Metric Old New Diff
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 3000 223.3584 better -92.6%
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.5 0.4393 better -12.1%
default_voice / seed-tts-text / c=16 / p=500 median_audio_ttfp_ms[0] 5000 276.575 better -94.5%
default_voice / seed-tts-text / c=16 / p=500 median_audio_rtf[0] 1.2 0.5073 better -57.7%
test_qwen3_omni_async_chunk.json (23 changes >10%)
Config Metric Old New Diff
random / c=1 / p=4 mean_ttft_ms[0] 135.4379 96.4061 better -28.8%
random / c=4 / p=16 mean_ttft_ms[1] 187.0896 140.8147 better -24.7%
random / c=4 / p=16 mean_e2el_ms[1] 34811.8335 28365.4956 better -18.5%
random / c=4 / p=16 mean_audio_rtf[1] 0.2238 0.18 better -19.6%
random / c=8 / p=32 mean_ttft_ms[2] 400 271.9025 better -32.0%
random / c=8 / p=32 mean_e2el_ms[2] 43656.9501 31907.9613 better -26.9%
random / c=8 / p=32 mean_audio_rtf[2] 0.2735 0.2359 better -13.8%
random / c=16 / p=64 mean_ttft_ms[3] 435.675 362.2916 better -16.8%
random / c=16 / p=64 mean_e2el_ms[3] 56657.1548 48161.5829 better -15.0%
random / c=16 / p=64 mean_audio_rtf[3] 0.3712 0.3278 better -11.7%
random / c=32 / p=128 mean_ttft_ms[4] 724.6836 507.776 better -29.9%
random / c=32 / p=128 mean_e2el_ms[4] 85097.6016 72630.8601 better -14.6%
random / c=32 / p=128 mean_audio_rtf[4] 0.5634 0.6296 WORSE +11.7%
random-mm / qps=0.1 / p=10 mean_audio_rtf[0] 0.1305 0.117 better -10.4%
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 220.0909 259.6317 WORSE +18.0%
random-mm / qps=0.5 / p=40 mean_e2el_ms[0] 5475.9565 4253.3009 better -22.3%
random-mm / qps=0.5 / p=40 mean_audio_rtf[0] 0.1818 0.1448 better -20.4%
random-mm / qps=1.0 / p=100 mean_e2el_ms[0] 8516.8316 5912.5623 better -30.6%
random-mm / qps=1.0 / p=100 mean_audio_rtf[0] 0.2717 0.1967 better -27.6%
random / c=1 / p=4 mean_ttft_ms[0] 78.4278 90.8033 WORSE +15.8%
random-mm / qps=0.1 / p=10 mean_ttft_ms[0] 275.3875 318.048 WORSE +15.5%
random-mm / qps=0.1 / p=10 mean_e2el_ms[0] 730.6995 807.6114 WORSE +10.5%
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 166.2201 186.168 WORSE +12.0%
test_qwen3_omni_multi_replicas.json (19 changes >10%)
Config Metric Old New Diff
random / c=8 / p=128 mean_ttft_ms[0] 203 130.5751 better -35.7%
random / c=8 / p=128 mean_tpot_ms[0] 28 7.9854 better -71.5%
random / c=8 / p=128 mean_audio_ttfp_ms[0] 861 486.486 better -43.5%
random / c=8 / p=128 mean_audio_rtf[0] 0.24 0.1896 better -21.0%
random / c=16 / p=128 mean_tpot_ms[1] 36 11.1209 better -69.1%
random / c=16 / p=128 mean_audio_ttfp_ms[1] 1200 954.1791 better -20.5%
random / c=16 / p=128 mean_audio_rtf[1] 0.3 0.3962 WORSE +32.1%
random / c=24 / p=128 mean_ttft_ms[2] 470 324.7249 better -30.9%
random / c=24 / p=128 mean_tpot_ms[2] 41 16.1507 better -60.6%
random / c=24 / p=128 mean_audio_ttfp_ms[2] 1528 1289.6514 better -15.6%
random / c=24 / p=128 mean_audio_rtf[2] 0.35 0.4098 WORSE +17.1%
random / c=32 / p=128 mean_tpot_ms[3] 45 20.7485 better -53.9%
random / c=32 / p=128 mean_audio_rtf[3] 0.39 0.7322 WORSE +87.7%
random-mm / qps=0.1 / p=10 mean_audio_ttfp_ms[0] 684 463.0752 better -32.3%
random-mm / qps=0.1 / p=10 mean_audio_rtf[0] 0.14 0.1153 better -17.6%
random-mm / qps=0.5 / p=40 mean_audio_ttfp_ms[0] 686 395.2182 better -42.4%
random-mm / qps=0.5 / p=40 mean_audio_rtf[0] 0.16 0.1259 better -21.3%
random-mm / qps=1.0 / p=100 mean_audio_ttfp_ms[0] 1119 665.5474 better -40.5%
random-mm / qps=1.0 / p=100 mean_audio_rtf[0] 0.21 0.166 better -21.0%
test_qwen3_omni_no_async_chunk.json (8 changes >10%)
Config Metric Old New Diff
random / c=1 / p=4 mean_e2el_ms[0] 22413.667 25070.9523 WORSE +11.9%
random / c=4 / p=16 mean_ttft_ms[1] 247.4411 173.1823 better -30.0%
random / c=8 / p=32 mean_ttft_ms[2] 354.9384 446.672 WORSE +25.8%
random / c=32 / p=128 mean_ttft_ms[4] 694.7487 3420.0634 WORSE +392.3%
random / c=32 / p=128 mean_e2el_ms[4] 68350.4853 136619.2743 WORSE +99.9%
random / c=32 / p=128 mean_audio_rtf[4] 0.4438 0.9266 WORSE +108.8%
random-mm / qps=0.5 / p=40 mean_ttft_ms[0] 215.4502 245.5446 WORSE +14.0%
random-mm / qps=1.0 / p=100 mean_ttft_ms[0] 442.5815 506.1407 WORSE +14.4%
test_qwen3_omni_vllm_text.json (2 changes >10%)
Config Metric Old New Diff
random / c=4 / p=16 mean_ttft_ms[1] 255.1882 173.0651 better -32.2%
random / c=8 / p=32 mean_ttft_ms[2] 310.639 271.8108 better -12.5%
test_qwen_image_edit_2511_vllm_omni.json (7 changes >10%)
Config Metric Old New Diff
512x512_steps20_i2i_2img / i2i / random throughput_qps 0.0721 0.0721 regression -12.5%
512x512_steps20_i2i_2img / i2i / random latency_mean 13.8398 13.8398 regression +14.9%
1536x1536_steps35_i2i_2img / i2i / random throughput_qps 0.0186 0.0186 regression -10.9%
1536x1536_steps35_i2i_2img / i2i / random latency_mean 53.6942 53.6942 regression +12.7%
1536x1536_steps35_i2i_2img / i2i / random latency_mean 18.5591 18.5591 regression +10.9%
512x512_steps20_i2i_2img / i2i / random throughput_qps 0.2776 0.2776 regression -10.1%
512x512_steps20_i2i_2img / i2i / random latency_mean 3.5938 3.5938 regression +11.6%
test_qwen_image_edit_vllm_omni.json (8 changes >10%)
Config Metric Old New Diff
512x512_steps20_i2i / i2i / random throughput_qps 0.1129 0.1129 regression -13.2%
512x512_steps20_i2i / i2i / random latency_mean 8.8352 8.8352 regression +16.0%
1536x1536_steps35_i2i / i2i / random throughput_qps 0.024 0.024 regression -11.4%
1536x1536_steps35_i2i / i2i / random latency_mean 41.5686 41.5686 regression +13.3%
1536x1536_steps35_i2i / i2i / random throughput_qps 0.0672 0.0672 regression -10.7%
1536x1536_steps35_i2i / i2i / random latency_mean 14.8517 14.8517 regression +12.4%
512x512_steps20_i2i / i2i / random latency_mean 3.4701 3.4701 regression +11.0%
1536x1536_steps35_i2i / i2i / random latency_mean 8.9656 8.9656 regression +11.1%
test_qwen_image_layered_vllm_omni.json (4 changes >10%)
Config Metric Old New Diff
640x640_steps20_i2i / i2i / random throughput_qps 0.0703 0.0703 regression -11.6%
640x640_steps20_i2i / i2i / random latency_mean 14.208 14.208 regression +13.6%
1024x1024_steps35_i2i / i2i / random throughput_qps 0.0417 0.0417 regression -12.5%
1024x1024_steps35_i2i / i2i / random latency_mean 23.914 23.914 regression +15.0%
test_qwen_image_vllm_omni.json (26 changes >10%)
Config Metric Old New Diff
512x512_steps20 / t2i / random throughput_qps 0.4153 0.4153 regression -25.5%
512x512_steps20 / t2i / random latency_mean 2.4065 2.4065 regression +37.2%
1536x1536_steps35 / t2i / random throughput_qps 0.0423 0.0423 regression -11.9%
1536x1536_steps35 / t2i / random latency_mean 23.5991 23.5991 regression +14.1%
512x512_steps20 / t2i / random throughput_qps 0.4175 0.4175 regression -32.1%
512x512_steps20 / t2i / random latency_mean 2.3953 2.3953 regression +50.2%
1536x1536_steps35 / t2i / random throughput_qps 0.0423 0.0423 regression -12.1%
1536x1536_steps35 / t2i / random latency_mean 23.5768 23.5768 regression +14.3%
1536x1536_steps35 / t2i / random throughput_qps 0.1214 0.1214 regression -10.9%
1536x1536_steps35 / t2i / random latency_mean 8.2211 8.2211 regression +12.7%
512x512_steps20 / t2i / random throughput_qps 0.559 0.559 regression -16.7%
512x512_steps20 / t2i / random latency_mean 1.794 1.794 regression +21.6%
1536x1536_steps35 / t2i / random throughput_qps 0.1918 0.1918 regression -10.1%
1536x1536_steps35 / t2i / random latency_mean 5.2022 5.2022 regression +11.7%
512x512_steps20_high_concurrency / t2i / random throughput_qps[0] 0.4125 0.4125 regression -26.2%
512x512_steps20_high_concurrency / t2i / random latency_mean[0] 2.4215 2.4215 regression +40.2%
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[0] 67000 55147.4286 better -17.7%
512x512_steps20_high_concurrency / t2i / random throughput_qps[1] 0.6714 0.6714 regression -16.7%
512x512_steps20_high_concurrency / t2i / random latency_mean[1] 2.992 2.992 regression +20.8%
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[1] 67000 55148.0 better -17.7%
512x512_steps20_high_concurrency / t2i / random throughput_qps[2] 0.6674 0.6674 regression -15.4%
512x512_steps20_high_concurrency / t2i / random latency_mean[2] 5.9293 5.9293 regression +18.6%
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[2] 67000 55150.4643 better -17.7%
512x512_steps20_high_concurrency / t2i / random throughput_qps[3] 0.8159 0.8159 regression -14.6%
512x512_steps20_high_concurrency / t2i / random latency_mean[3] 9.656 9.656 regression +18.0%
512x512_steps20_high_concurrency / t2i / random peak_memory_mb_mean[3] 67000 55152.3821 better -17.7%
test_tts.json (29 changes >10%)
Config Metric Old New Diff
voice_clone / seed-tts / c=16 / p=128 median_audio_ttfp_ms[1] 647.1272 647.1272 regression +23.5%
voice_clone / seed-tts / c=16 / p=128 median_audio_rtf[1] 0.3642 0.3642 regression +28.9%
voice_clone / seed-tts / c=64 / p=128 median_audio_ttfp_ms[2] 4777.2714 4777.2714 regression +19.9%
voice_clone / seed-tts / c=64 / p=128 median_audio_rtf[2] 1.2282 1.2282 regression +27.9%
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 75.3341 75.3341 regression +16.8%
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.1859 0.1859 regression +22.5%
default_voice / seed-tts-text / c=16 / p=128 median_audio_ttfp_ms[1] 713.7727 713.7727 regression +16.5%
default_voice / seed-tts-text / c=16 / p=128 median_audio_rtf[1] 0.3148 0.3148 regression +25.7%
default_voice / seed-tts-text / c=16 / p=128 audio_throughput[1] 49.6268 55.3523 better +11.5%
default_voice / seed-tts-text / c=64 / p=128 median_audio_ttfp_ms[2] 5941.7179 5941.7179 regression +12.6%
default_voice / seed-tts-text / c=64 / p=128 median_audio_rtf[2] 1.1662 1.1662 regression +19.7%
default_voice / seed-tts-text / c=64 / p=128 audio_throughput[2] 48.3091 68.7032 better +42.2%
voice_design / seed-tts-design / c=1 / p=20 median_audio_ttfp_ms[0] 52.0447 46.3479 better -10.9%
voice_design / seed-tts-design / c=8 / p=80 median_audio_ttfp_ms[0] 75.5967 75.5967 regression +13.1%
voice_design / seed-tts-design / c=8 / p=80 median_audio_rtf[0] 0.1871 0.1871 regression +20.5%
voice_design / seed-tts-design / c=16 / p=128 median_audio_ttfp_ms[1] 587.5877 587.5877 regression +21.2%
voice_design / seed-tts-design / c=16 / p=128 median_audio_rtf[1] 0.3143 0.3143 regression +24.5%
voice_design / seed-tts-design / c=16 / p=128 audio_throughput[1] 48.9046 55.1846 better +12.8%
voice_design / seed-tts-design / c=64 / p=128 median_audio_ttfp_ms[2] 4960.3726 4960.3726 regression +24.9%
voice_design / seed-tts-design / c=64 / p=128 median_audio_rtf[2] 1.1703 1.1703 regression +26.9%
voice_design / seed-tts-design / c=64 / p=128 audio_throughput[2] 48.0505 71.5708 better +48.9%
default_voice / seed-tts-text / c=1 / p=20 median_audio_ttfp_ms[0] 150 82.1886 better -45.2%
default_voice / seed-tts-text / c=1 / p=20 median_audio_rtf[0] 0.15 0.1241 better -17.3%
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 1500 160.9168 better -89.3%
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.3 0.1907 better -36.4%
default_voice / seed-tts-text / c=8 / p=80 audio_throughput[0] 30.0 39.2332 better +30.8%
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_ttfp_ms[0] 3000 100.1265 better -96.7%
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_rtf[0] 0.25 0.1437 better -42.5%
default_voice / seed-tts-text / qps=2.0 / p=100 audio_throughput[0] 4.0 11.7605 better +194.0%
test_voxcpm2.json (7 changes >10%)
Config Metric Old New Diff
voice_clone / seed-tts / c=8 / p=80 median_audio_ttfp_ms[0] 475.3357 475.3357 regression +38.2%
voice_clone / seed-tts / c=8 / p=80 median_audio_rtf[0] 0.1671 0.1243 better -25.6%
voice_clone / seed-tts / c=8 / p=80 audio_throughput[0] 22.8717 36.6358 better +60.2%
default_voice / seed-tts-text / c=8 / p=80 median_audio_ttfp_ms[0] 351.5855 262.6212 better -25.3%
default_voice / seed-tts-text / c=8 / p=80 median_audio_rtf[0] 0.1207 0.0753 better -37.6%
default_voice / seed-tts-text / c=8 / p=80 audio_throughput[0] 73.7632 103.5843 better +40.4%
default_voice / seed-tts-text / qps=2.0 / p=100 median_audio_ttfp_ms[0] 80.1095 80.1095 regression +36.1%

@congw729
congw729 requested a review from yenuo26 as a code owner July 20, 2026 09:44
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@congw729
congw729 force-pushed the update-perf-baselines-jul branch from 0480a0e to a1a027a Compare July 20, 2026 09:47
@congw729 congw729 mentioned this pull request Jul 20, 2026
Closed
1 task done
@congw729 congw729 changed the title Update perf baseline base on 7/5-7/11 7 days avg. [WIP] Update perf baseline base on 7/5-7/11 7 days avg. Jul 20, 2026
@congw729 congw729 mentioned this pull request Jul 21, 2026
Closed
1 task done
@congw729
congw729 force-pushed the update-perf-baselines-jul branch from a1a027a to 7af3ae5 Compare July 21, 2026 02:57
@congw729 congw729 changed the title [WIP] Update perf baseline base on 7/5-7/11 7 days avg. Update perf baseline base on 7/5-7/11 7 days avg. Jul 21, 2026
@R2-Y

R2-Y commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

TTS part LGTM, Thanks!

@congw729
congw729 force-pushed the update-perf-baselines-jul branch from 7af3ae5 to 75943e7 Compare July 21, 2026 03:26
@amy-why-3459

amy-why-3459 commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

please fix DCO.

@congw729
update baseine wq
9e4e1ff 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
congw729 force-pushed the update-perf-baselines-jul branch from 75943e7 to 9e4e1ff Compare July 21, 2026 03:41
david6666666
david6666666 approved these changes Jul 21, 2026
View reviewed changes

@david6666666 david6666666 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@david6666666 david6666666 added the ready label to trigger buildkite CI label Jul 21, 2026
@david6666666
david6666666 enabled auto-merge (squash) July 21, 2026 11:56
@david6666666
david6666666 merged commit 0034eaa into vllm-project:main Jul 21, 2026
7 of 9 checks passed
y-null pushed a commit to y-null/vllm-omni that referenced this pull request Jul 21, 2026
@congw729 @y-null
Update perf baseline base on 7/5-7/11 7 days avg. (vllm-project#5231)
fa026bf 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Signed-off-by: y_null <y_null@qq.com>
bowieshi pushed a commit to bowieshi/vllm-omni that referenced this pull request Jul 22, 2026
@congw729 @bowieshi
Update perf baseline base on 7/5-7/11 7 days avg. (vllm-project#5231)
fe92d3c 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
nagisa-kunhah pushed a commit to nagisa-kunhah/vllm-omni that referenced this pull request Jul 30, 2026
@congw729 @nagisa-kunhah
Update perf baseline base on 7/5-7/11 7 days avg. (vllm-project#5231)
bbede16 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@EchoHayate EchoHayate mentioned this pull request Aug 4, 2026
Closed
@linyueqian linyueqian mentioned this pull request Aug 11, 2026
Merged
@congw729 congw729 mentioned this pull request Aug 14, 2026
Merged
3 of 4 tasks
@psv666 psv666 mentioned this pull request Sep 22, 2026
Closed
1 task done
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
@congw729
Update perf baseline base on 7/5-7/11 7 days avg. (vllm-project#5231)
fdf88c9 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@david6666666 david6666666 david6666666 approved these changes
@yenuo26 yenuo26 Awaiting requested review from yenuo26 yenuo26 is a code owner

Assignees

No one assigned

Labels

ready label to trigger buildkite CI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@congw729 @R2-Y @amy-why-3459 @david6666666