Skip to content

[CI] Update perf json baselines - #4289

Merged
hsliuustc0106 merged 28 commits into
vllm-project:mainfrom
congw729:monthly-update-baseline
Jun 17, 2026
Merged

[CI] Update perf json baselines#4289
hsliuustc0106 merged 28 commits into
vllm-project:mainfrom
congw729:monthly-update-baseline

Conversation

@congw729

@congw729 congw729 commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Before submitting: run the precheck-pr skill with code agent for a self-check against project conventions — catches dead code, missing benchmarks, and title format issues.

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Monthly update baseline results.

Update old baseline to new - 6.1-6.8 avg by 5% improvements.
Keep the baseline unchanged unless the new results are worse than the old ones.

History table: https://hsliuustc0106.github.io/vllm-omni-kanban/
History raw data: https://github.com/hsliuustc0106/vllm-omni-kanban/tree/main/data/buildkite_nightly_raw

Last time updates: #3695

🆗 Qwen3 Omni @amy-why-3459
🆗 Qwen3 TTS @R2-Y
🆗 Qwen Image @NumberWan
🆗 Qwen Image Layered @spencerr221
🆗 Qwen Image Edit @fhfuih
🆗 Qwen Image Edit 2511 @fhfuih
🆗 WAN 2.2 @bjf-frz
🆗 Hunyuan Image 3 @Bounty-hunter (avg 3 times results for H200 machine )
NA BAGEL @natureofnature (skip this updatation due to performance degression)
🆗 VoxCPM2 @Shirley125

Updation

Qwen3Omni no async:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
qwen3_omni_no_async / random / c=1 / prompts=4 mean_ttft_ms[0] 400.0000 114.8651 109.1218
qwen3_omni_no_async / random / c=4 / prompts=16 mean_ttft_ms[1] 300.0000 260.4643 247.4411
qwen3_omni_no_async / random / c=8 / prompts=32 mean_ttft_ms[2] 400.0000 373.6194 354.9384
qwen3_omni_no_async / random / c=16 / prompts=64 mean_ttft_ms[3] 600.0000 471.3563 447.7885
qwen3_omni_no_async / random / c=32 / prompts=128 mean_ttft_ms[4] 900.0000 731.3144 694.7487
qwen3_omni_no_async / random / c=1 / prompts=4 mean_e2el_ms[0] 32000.0000 23593.3337 22413.6670
qwen3_omni_no_async / random / c=4 / prompts=16 mean_e2el_ms[1] 34000.0000 32430.4963 30808.9715
qwen3_omni_no_async / random / c=8 / prompts=32 mean_e2el_ms[2] 45000.0000 37555.6253 35677.8440
qwen3_omni_no_async / random / c=16 / prompts=64 mean_e2el_ms[3] 61000.0000 48785.1556 46345.8978
qwen3_omni_no_async / random / c=32 / prompts=128 mean_e2el_ms[4] 90000.0000 71947.8793 68350.4853
qwen3_omni_no_async / random / c=1 / prompts=4 mean_audio_rtf[0] 0.2000 0.1784 0.1695
qwen3_omni_no_async / random / c=4 / prompts=16 mean_audio_rtf[1] 0.2500 0.2140 0.2033
qwen3_omni_no_async / random / c=8 / prompts=32 mean_audio_rtf[2] 0.3000 0.2597 0.2467
qwen3_omni_no_async / random / c=16 / prompts=64 mean_audio_rtf[3] 0.4500 0.3404 0.3234
qwen3_omni_no_async / random / c=32 / prompts=128 mean_audio_rtf[4] 0.6000 0.4671 0.4438
qwen3_omni_no_async / random-mm / qps=0.1 / prompts=10 mean_ttft_ms[0] 500.0000 340.3761 323.3573
qwen3_omni_no_async / random-mm / qps=0.1 / prompts=10 mean_e2el_ms[0] 6000.0000 4866.7550 4623.4173
qwen3_omni_no_async / random-mm / qps=0.1 / prompts=10 mean_audio_rtf[0] 0.2000 0.1480 0.1406
qwen3_omni_no_async / random-mm / qps=0.5 / prompts=40 mean_ttft_ms[0] 300.0000 226.7897 215.4502
qwen3_omni_no_async / random-mm / qps=0.5 / prompts=40 mean_e2el_ms[0] 7000.0000 5183.3148 4924.1490
qwen3_omni_no_async / random-mm / qps=0.5 / prompts=40 mean_audio_rtf[0] 0.2000 0.1723 0.1637
qwen3_omni_no_async / random-mm / qps=1.0 / prompts=100 mean_ttft_ms[0] 700.0000 465.8753 442.5815
qwen3_omni_no_async / random-mm / qps=1.0 / prompts=100 mean_e2el_ms[0] 9000.0000 6780.1038 6441.0986
qwen3_omni_no_async / random-mm / qps=1.0 / prompts=100 mean_audio_rtf[0] 0.3000 0.2201 0.2091
Qwen 3 Omni Async Chunk:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
qwen3_omni_async / audio/omni / random / c=1 / prompts=4 mean_ttft_ms[0] 400.0000 142.5662 135.4379
qwen3_omni_async / audio/omni / random / c=4 / prompts=16 mean_ttft_ms[1] 300.0000 196.9364 187.0896
qwen3_omni_async / audio/omni / random / c=8 / prompts=32 mean_ttft_ms[2] 400.0000 443.5706 400.0000
qwen3_omni_async / audio/omni / random / c=16 / prompts=64 mean_ttft_ms[3] 600.0000 458.6052 435.6750
qwen3_omni_async / audio/omni / random / c=32 / prompts=128 mean_ttft_ms[4] 900.0000 762.8248 724.6836
qwen3_omni_async / audio/omni / random / c=1 / prompts=4 mean_e2el_ms[0] 26000.0000 18547.3528 17619.9851
qwen3_omni_async / audio/omni / random / c=4 / prompts=16 mean_e2el_ms[1] 43000.0000 36644.0352 34811.8335
qwen3_omni_async / audio/omni / random / c=8 / prompts=32 mean_e2el_ms[2] 52000.0000 45954.6844 43656.9501
qwen3_omni_async / audio/omni / random / c=16 / prompts=64 mean_e2el_ms[3] 72000.0000 59639.1103 56657.1548
qwen3_omni_async / audio/omni / random / c=32 / prompts=128 mean_e2el_ms[4] 109000.0000 89576.4228 85097.6016
qwen3_omni_async / audio/omni / random / c=1 / prompts=4 mean_audio_rtf[0] 0.1500 0.1346 0.1279
qwen3_omni_async / audio/omni / random / c=4 / prompts=16 mean_audio_rtf[1] 0.3000 0.2355 0.2238
qwen3_omni_async / audio/omni / random / c=8 / prompts=32 mean_audio_rtf[2] 0.3500 0.2879 0.2735
qwen3_omni_async / audio/omni / random / c=16 / prompts=64 mean_audio_rtf[3] 0.5000 0.3908 0.3712
qwen3_omni_async / audio/omni / random / c=32 / prompts=128 mean_audio_rtf[4] 0.7500 0.5931 0.5634
qwen3_omni_async / audio/omni / random-mm / qps=0.1 / prompts=10 mean_ttft_ms[0] 600.0000 358.2411 340.3290
qwen3_omni_async / audio/omni / random-mm / qps=0.1 / prompts=10 mean_e2el_ms[0] 6000.0000 4532.0810 4305.4770
qwen3_omni_async / audio/omni / random-mm / qps=0.1 / prompts=10 mean_audio_rtf[0] 0.2000 0.1374 0.1305
qwen3_omni_async / audio/omni / random-mm / qps=0.5 / prompts=40 mean_ttft_ms[0] 300.0000 231.6746 220.0909
qwen3_omni_async / audio/omni / random-mm / qps=0.5 / prompts=40 mean_e2el_ms[0] 7000.0000 5764.1647 5475.9565
qwen3_omni_async / audio/omni / random-mm / qps=0.5 / prompts=40 mean_audio_rtf[0] 0.2500 0.1914 0.1818
qwen3_omni_async / audio/omni / random-mm / qps=1.0 / prompts=100 mean_ttft_ms[0] 700.0000 473.7006 450.0155
qwen3_omni_async / audio/omni / random-mm / qps=1.0 / prompts=100 mean_e2el_ms[0] 11000.0000 8965.0859 8516.8316
qwen3_omni_async / audio/omni / random-mm / qps=1.0 / prompts=100 mean_audio_rtf[0] 0.3500 0.2860 0.2717
qwen3_omni_async / text-only / random / c=1 / prompts=4 mean_ttft_ms[0] 100.0000 82.5555 78.4278
qwen3_omni_async / text-only / random / c=4 / prompts=16 mean_ttft_ms[1] 300.0000 206.1091 195.8037
qwen3_omni_async / text-only / random / c=8 / prompts=32 mean_ttft_ms[2] 500.0000 351.2966 333.7318
qwen3_omni_async / text-only / random / c=16 / prompts=64 mean_ttft_ms[3] 800.0000 591.3737 561.8050
qwen3_omni_async / text-only / random / c=32 / prompts=128 mean_ttft_ms[4] 1500.0000 1092.6769 1038.0430
qwen3_omni_async / text-only / random / c=1 / prompts=4 mean_e2el_ms[0] 5000.0000 4514.0030 4288.3029
qwen3_omni_async / text-only / random / c=4 / prompts=16 mean_e2el_ms[1] 8000.0000 7820.9925 7429.9429
qwen3_omni_async / text-only / random / c=8 / prompts=32 mean_e2el_ms[2] 10000.0000 9949.0720 9451.6184
qwen3_omni_async / text-only / random / c=16 / prompts=64 mean_e2el_ms[3] 14000.0000 13271.0834 12607.5292
qwen3_omni_async / text-only / random / c=32 / prompts=128 mean_e2el_ms[4] 20000.0000 19108.6891 18153.2547
qwen3_omni_async / text-only / random-mm / qps=0.1 / prompts=10 mean_ttft_ms[0] 500.0000 289.8816 275.3875
qwen3_omni_async / text-only / random-mm / qps=0.1 / prompts=10 mean_e2el_ms[0] 1000.0000 769.1574 730.6995
qwen3_omni_async / text-only / random-mm / qps=0.5 / prompts=40 mean_ttft_ms[0] 200.0000 174.9686 166.2201
qwen3_omni_async / text-only / random-mm / qps=0.5 / prompts=40 mean_e2el_ms[0] 800.0000 709.0304 673.5789
qwen3_omni_async / text-only / random-mm / qps=1.0 / prompts=100 mean_ttft_ms[0] 600.0000 376.6085 357.7781
qwen3_omni_async / text-only / random-mm / qps=1.0 / prompts=100 mean_e2el_ms[0] 1300.0000 1003.5255 953.3493
Qwen 3 Omni vllm text (lack 2 days data):
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
qwen3_omni_vllm_text / random / c=1 / prompts=4 mean_ttft_ms[0] 300.0000 72.7659 69.1276
qwen3_omni_vllm_text / random / c=4 / prompts=16 mean_ttft_ms[1] 300.0000 268.6192 255.1882
qwen3_omni_vllm_text / random / c=8 / prompts=32 mean_ttft_ms[2] 1000.0000 326.9884 310.6390
qwen3_omni_vllm_text / random / c=16 / prompts=64 mean_ttft_ms[3] 600.0000 444.2460 422.0337
qwen3_omni_vllm_text / random / c=32 / prompts=128 mean_ttft_ms[4] 800.0000 560.0612 532.0582
qwen3_omni_vllm_text / random / c=1 / prompts=4 mean_e2el_ms[0] 6000.0000 4613.3907 4382.7211
qwen3_omni_vllm_text / random / c=4 / prompts=16 mean_e2el_ms[1] 8000.0000 7905.6647 7510.3815
qwen3_omni_vllm_text / random / c=8 / prompts=32 mean_e2el_ms[2] 11000.0000 9937.3531 9440.4854
qwen3_omni_vllm_text / random / c=16 / prompts=64 mean_e2el_ms[3] 13000.0000 13213.7038 12553.0186
qwen3_omni_vllm_text / random / c=32 / prompts=128 mean_e2el_ms[4] 17000.0000 17823.7964 16932.6065
qwen3_omni_vllm_text / random-mm / qps=0.1 / prompts=10 mean_ttft_ms[0] 30.0000 27.7243 26.3381
qwen3_omni_vllm_text / random-mm / qps=0.1 / prompts=10 mean_e2el_ms[0] 540.0000 510.7827 485.2436
qwen3_omni_vllm_text / random-mm / qps=0.5 / prompts=40 mean_ttft_ms[0] 30.0000 27.8439 26.4517
qwen3_omni_vllm_text / random-mm / qps=0.5 / prompts=40 mean_e2el_ms[0] 600.0000 544.4229 517.2017
qwen3_omni_vllm_text / random-mm / qps=1.0 / prompts=100 mean_ttft_ms[0] 40.0000 29.1549 27.6972
qwen3_omni_vllm_text / random-mm / qps=1.0 / prompts=100 mean_e2el_ms[0] 700.0000 590.9141 561.3684
Qwen Image Edit:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
single 512 throughput_qps 0.1000 0.1075 0.1129
single 512 latency_mean 10.0000 9.3002 8.8352
single 512 peak_memory_mb_max 57000.0000 58270.0000 55356.5000
single 512 peak_memory_mb_mean 57000.0000 58270.0000 55356.5000
single 1536 throughput_qps 0.0200 0.0229 0.0240
single 1536 latency_mean 50.0000 43.7564 41.5686
single 1536 peak_memory_mb_max 65000.0000 66066.0000 62762.7000
single 1536 peak_memory_mb_mean 65000.0000 66066.0000 62762.7000
ulysses2 cfg2 vae patch4 1536 throughput_qps 0.0600 0.0640 0.0672
ulysses2 cfg2 vae patch4 1536 latency_mean 20.0000 15.6333 14.8517
ulysses2 cfg2 vae patch4 1536 peak_memory_mb_max 56000.0000 56770.0000 53931.5000
ulysses2 cfg2 vae patch4 1536 peak_memory_mb_mean 56000.0000 56770.0000 53931.5000
cache_dit 512 throughput_qps 0.2600 0.2742 0.2879
cache_dit 512 latency_mean 10.0000 3.6527 3.4701
cache_dit 512 peak_memory_mb_max 58000.0000 58448.0000 55525.6000
cache_dit 512 peak_memory_mb_mean 58000.0000 58448.0000 55525.6000
cache_dit 1536 throughput_qps 0.1000 0.1060 0.1113
cache_dit 1536 latency_mean 10.0000 9.4375 8.9656
cache_dit 1536 peak_memory_mb_max 67000.0000 67114.0000 63758.3000
cache_dit 1536 peak_memory_mb_mean 67000.0000 67114.0000 63758.3000
Qwen Image Edit 2511:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
single 512 2img throughput_qps 0.0600 0.0686 0.0721
single 512 2img latency_mean 20.0000 14.5682 13.8398
single 512 2img peak_memory_mb_max 57000.0000 57612.0000 54731.4000
single 512 2img peak_memory_mb_mean 57000.0000 57612.0000 54731.4000
single 1536 2img throughput_qps 0.0100 0.0177 0.0186
single 1536 2img latency_mean 60.0000 56.5202 53.6942
single 1536 2img peak_memory_mb_max 67000.0000 67792.0000 64402.4000
single 1536 2img peak_memory_mb_mean 67000.0000 67792.0000 64402.4000
ulysses2 cfg2 vae patch4 1536 2img throughput_qps 0.0500 0.0512 0.0538
ulysses2 cfg2 vae patch4 1536 2img latency_mean 20.0000 19.5359 18.5591
ulysses2 cfg2 vae patch4 1536 2img peak_memory_mb_max 55000.0000 56300.0000 53485.0000
ulysses2 cfg2 vae patch4 1536 2img peak_memory_mb_mean 55000.0000 56300.0000 53485.0000
cache_dit 512 2img throughput_qps 0.2100 0.2644 0.2776
cache_dit 512 2img latency_mean 10.0000 3.7830 3.5938
cache_dit 512 2img peak_memory_mb_max 57000.0000 58120.0000 55214.0000
cache_dit 512 2img peak_memory_mb_mean 57000.0000 58120.0000 55214.0000
cache_dit 1536 2img throughput_qps 0.0800 0.1001 0.1051
cache_dit 1536 2img latency_mean 20.0000 9.9913 9.4918
cache_dit 1536 2img peak_memory_mb_max 66000.0000 67648.0000 64265.6000
cache_dit 1536 2img peak_memory_mb_mean 66000.0000 67648.0000 64265.6000
Qwen 3 TTS (contains degression baselines): Lower the baseline for below:
File Config Metric n Baseline before PR Average Candidate/Updated Baseline Current baseline
test_tts.json customvoice / seed-tts-text / c=64 / prompts=128 median_audio_ttfp_ms[2] 7 5000.0000 6254.4399 5941.7179 5000.0000
test_tts.json customvoice / seed-tts-text / c=64 / prompts=128 median_audio_rtf[2] 7 1.1000 1.2276 1.1662 1.1000

The historical data shows:

TTS CustomVoice seed-tts-text c=64 median_audio_ttfp_ms (current baseline = 5000)

Week n Weekly avg 5% Improve Candidate Baseline vs avg
2026-W20 3 5921.4966 5625.4218 -15.6%
2026-W21 7 5889.5131 5595.0374 -15.1%
2026-W22 7 6087.7233 5783.3372 -17.9%
2026-W23 6 6267.9937 5954.5940 -20.2%
2026-W24 7 5984.1785 5684.9696 -16.4%
2026-W25 1 6191.3079 5881.7425 -19.2%

TTS CustomVoice seed-tts-text c=64 median_audio_rtf (current baseline=1.1):

Week n Weekly avg 5% Improve Candidate Baseline vs avg
2026-W20 3 1.1729 1.1142 -6.2%
2026-W21 7 1.1607 1.1027 -5.2%
2026-W22 7 1.1915 1.1319 -7.7%
2026-W23 6 1.2364 1.1746 -11.0%
2026-W24 7 1.1640 1.1058 -5.5%
2026-W25 1 1.1943 1.1346 -7.9%

Others are improvements:

Config Metric Baseline before this PR Jun 1-8 average Updated baseline
qwen3_tts_base / voice_clone / latency / seed-tts / c=1 / prompts=20 median_audio_ttfp_ms[0] 3000 128.8952 122.4504
qwen3_tts_base / voice_clone / latency / seed-tts / c=1 / prompts=20 median_audio_rtf[0] 0.6 0.1550 0.1472
qwen3_tts_base / voice_clone / throughput / seed-tts / c=8 / prompts=80 median_audio_ttfp_ms[0] 1000 209.5996 199.1197
qwen3_tts_base / voice_clone / throughput / seed-tts / c=16 / prompts=128 median_audio_ttfp_ms[1] 3000 681.1866 647.1272
qwen3_tts_base / voice_clone / throughput / seed-tts / c=64 / prompts=128 median_audio_ttfp_ms[2] 14000 5028.7068 4777.2714
qwen3_tts_base / voice_clone / throughput / seed-tts / c=8 / prompts=80 median_audio_rtf[0] 0.7 0.3051 0.2899
qwen3_tts_base / voice_clone / throughput / seed-tts / c=16 / prompts=128 median_audio_rtf[1] 1.15 0.3834 0.3642
qwen3_tts_base / voice_clone / throughput / seed-tts / c=64 / prompts=128 median_audio_rtf[2] 3.9 1.2929 1.2282
qwen3_tts_base / voice_clone / throughput / seed-tts / c=8 / prompts=80 audio_throughput[0] 12 19.5842 20.5634
qwen3_tts_base / voice_clone / throughput / seed-tts / c=16 / prompts=128 audio_throughput[1] 14 38.6991 40.634
qwen3_tts_base / voice_clone / throughput / seed-tts / c=64 / prompts=128 audio_throughput[2] 14 41.0246 43.0758
qwen3_tts_customvoice / default_voice / latency / seed-tts-text / c=1 / prompts=20 median_audio_ttfp_ms[0] 2000 45.8599 43.5669
qwen3_tts_customvoice / default_voice / latency / seed-tts-text / c=1 / prompts=20 median_audio_rtf[0] 0.4 0.1313 0.1247
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=8 / prompts=80 median_audio_ttfp_ms[0] 300 69.3867 65.9174
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=16 / prompts=128 median_audio_ttfp_ms[1] 1000 751.3397 713.7727
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=64 / prompts=128 median_audio_ttfp_ms[2] 5000 6254.4399 5000
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=8 / prompts=80 median_audio_rtf[0] 0.25 0.1712 0.1626
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=16 / prompts=128 median_audio_rtf[1] 0.35 0.3314 0.3148
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=64 / prompts=128 median_audio_rtf[2] 1.1 1.2276 1.1
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=8 / prompts=80 audio_throughput[0] 30 30.3875 31.9068
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=16 / prompts=128 audio_throughput[1] 40 47.2637 49.6268
qwen3_tts_customvoice / default_voice / throughput / seed-tts-text / c=64 / prompts=128 audio_throughput[2] 40 46.0087 48.3091
qwen3_tts_customvoice / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 median_audio_ttfp_ms[0] 100 52.2216 49.6105
qwen3_tts_customvoice / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 median_audio_rtf[0] 0.2 0.1432 0.1361
qwen3_tts_customvoice / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 audio_throughput[0] 10 9.6077 10.0881
qwen3_tts_customvoice / voice_design / latency / seed-tts-design / c=1 / prompts=20 median_audio_ttfp_ms[0] 100 47.9359 45.5391
qwen3_tts_customvoice / voice_design / latency / seed-tts-design / c=1 / prompts=20 median_audio_rtf[0] 0.15 0.1337 0.1271
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=8 / prompts=80 median_audio_ttfp_ms[0] 100 69.6285 66.1471
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=16 / prompts=128 median_audio_ttfp_ms[1] 1000 618.5133 587.5877
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=64 / prompts=128 median_audio_ttfp_ms[2] 5000 5221.4448 4960.3726
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=8 / prompts=80 median_audio_rtf[0] 0.25 0.1724 0.1637
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=16 / prompts=128 median_audio_rtf[1] 0.4 0.3308 0.3143
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=64 / prompts=128 median_audio_rtf[2] 1.2 1.2319 1.1703
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=8 / prompts=80 audio_throughput[0] 30 33.5165 35.1923
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=16 / prompts=128 audio_throughput[1] 40 46.5758 48.9046
qwen3_tts_customvoice / voice_design / throughput / seed-tts-design / c=64 / prompts=128 audio_throughput[2] 40 45.7624 48.0505
qwen3_tts_customvoice / voice_design / stress / seed-tts-design / qps=2.0 / prompts=100 median_audio_ttfp_ms[0] 100 51.9438 49.3466
qwen3_tts_customvoice / voice_design / stress / seed-tts-design / qps=2.0 / prompts=100 median_audio_rtf[0] 0.2 0.1420 0.1349
Qwen Image: Config | Metric | Baseline before this PR | Jun 1-8 average | Updated baseline -- | -- | -- | -- | -- qwen_image / single_device / 512x512_steps20 / c=1 / prompts=10 | throughput_qps[0] | 0.3500 | 0.3955 | 0.4153 qwen_image / single_device / 512x512_steps20 / c=1 / prompts=10 | latency_mean[0] | 3 | 2.5332 | 2.4065 qwen_image / single_device / 512x512_steps20 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 56000 | 55042.0000 | 52289.9000 qwen_image / single_device / 1536x1536_steps35 / c=1 / prompts=10 | throughput_qps[0] | 0.0300 | 0.0403 | 0.0423 qwen_image / single_device / 1536x1536_steps35 / c=1 / prompts=10 | latency_mean[0] | 26 | 24.8411 | 23.5991 qwen_image / single_device / 1536x1536_steps35 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 65000 | 64590.0000 | 61360.5000 qwen_image / single_device_step_execution / 512x512_steps20 / c=1 / prompts=10 | throughput_qps[0] | 0.3500 | 0.3976 | 0.4175 qwen_image / single_device_step_execution / 512x512_steps20 / c=1 / prompts=10 | latency_mean[0] | 3 | 2.5213 | 2.3953 qwen_image / single_device_step_execution / 512x512_steps20 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 56000 | 0.0000 | 56000 qwen_image / single_device_step_execution / 1536x1536_steps35 / c=1 / prompts=10 | throughput_qps[0] | 0.0300 | 0.0403 | 0.0423 qwen_image / single_device_step_execution / 1536x1536_steps35 / c=1 / prompts=10 | latency_mean[0] | 26 | 24.8177 | 23.5768 qwen_image / single_device_step_execution / 1536x1536_steps35 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 65000 | 0.0000 | 65000 qwen_image / ulysses2_cfg2_vae_patch4 / 1536x1536_steps35 / c=1 / prompts=10 | throughput_qps[0] | 0.1100 | 0.1156 | 0.1214 qwen_image / ulysses2_cfg2_vae_patch4 / 1536x1536_steps35 / c=1 / prompts=10 | latency_mean[0] | 9 | 8.6538 | 8.2211 qwen_image / ulysses2_cfg2_vae_patch4 / 1536x1536_steps35 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 55000 | 54426.0000 | 51704.7000 qwen_image / ulysses2_cfg2_cache_dit / 512x512_steps20 / c=1 / prompts=10 | throughput_qps[0] | 0.4600 | 0.5323 | 0.5590 qwen_image / ulysses2_cfg2_cache_dit / 512x512_steps20 / c=1 / prompts=10 | latency_mean[0] | 3 | 1.8884 | 1.7940 qwen_image / ulysses2_cfg2_cache_dit / 512x512_steps20 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 56000 | 55155.4500 | 52397.6775 qwen_image / ulysses2_cfg2_cache_dit / 1536x1536_steps35 / c=1 / prompts=10 | throughput_qps[0] | 0.1800 | 0.1826 | 0.1918 qwen_image / ulysses2_cfg2_cache_dit / 1536x1536_steps35 / c=1 / prompts=10 | latency_mean[0] | 6 | 5.4760 | 5.2022 qwen_image / ulysses2_cfg2_cache_dit / 1536x1536_steps35 / c=1 / prompts=10 | peak_memory_mb_mean[0] | 67000 | 64704.0000 | 61468.8000 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=1 / prompts=20 | throughput_qps[0] | 0.3000 | 0.3928 | 0.4125 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=2 / prompts=40 | throughput_qps[1] | 0.5000 | 0.6394 | 0.6714 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=4 / prompts=80 | throughput_qps[2] | 0.6000 | 0.6356 | 0.6674 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=8 / prompts=160 | throughput_qps[3] | 0.7000 | 0.7770 | 0.8159 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=1 / prompts=20 | latency_mean[0] | 3.5000 | 2.5490 | 2.4215 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=2 / prompts=40 | latency_mean[1] | 4.0000 | 3.1495 | 2.9920 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=4 / prompts=80 | latency_mean[2] | 7.0000 | 6.2414 | 5.9293 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=8 / prompts=160 | latency_mean[3] | 12.0000 | 10.1642 | 9.6560 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=1 / prompts=20 | peak_memory_mb_mean[0] | 67000 | 0.0000 | 67000 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=2 / prompts=40 | peak_memory_mb_mean[1] | 67000 | 0.0000 | 67000 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=4 / prompts=80 | peak_memory_mb_mean[2] | 67000 | 0.0000 | 67000 qwen_image / single_device_step_execution_high_concurrency / 512x512_steps20_high_concurrency / c=8 / prompts=160 | peak_memory_mb_mean[3] | 67000 | 0.0000 | 67000
VoxCPM2:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
voxcpm2 / voice_clone / latency / seed-tts / c=1 / prompts=20 median_audio_ttfp_ms[0] 600.0000 119.9316 113.9351
voxcpm2 / voice_clone / latency / seed-tts / c=1 / prompts=20 median_audio_rtf[0] 0.2000 0.0449 0.0427
voxcpm2 / voice_clone / throughput / seed-tts / c=8 / prompts=80 median_audio_ttfp_ms[0] 3500.0000 437.8092 415.9187
voxcpm2 / voice_clone / throughput / seed-tts / c=8 / prompts=80 median_audio_rtf[0] 0.8500 0.1539 0.1462
voxcpm2 / voice_clone / throughput / seed-tts / c=8 / prompts=80 audio_throughput[0] 8.0000 19.0598 20.0128
voxcpm2 / default_voice / latency / seed-tts-text / c=1 / prompts=20 median_audio_ttfp_ms[0] 350.0000 69.7630 66.2748
voxcpm2 / default_voice / latency / seed-tts-text / c=1 / prompts=20 median_audio_rtf[0] 0.1500 0.0385 0.0366
voxcpm2 / default_voice / throughput / seed-tts-text / c=8 / prompts=80 median_audio_ttfp_ms[0] 3000.0000 323.8287 307.6373
voxcpm2 / default_voice / throughput / seed-tts-text / c=8 / prompts=80 median_audio_rtf[0] 0.5000 0.1112 0.1056
voxcpm2 / default_voice / throughput / seed-tts-text / c=8 / prompts=80 audio_throughput[0] 10.0000 61.4693 64.5428
voxcpm2 / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 median_audio_ttfp_ms[0] 500.0000 73.7850 70.0958
voxcpm2 / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 median_audio_rtf[0] 0.3000 0.0446 0.0424
voxcpm2 / default_voice / stress / seed-tts-text / qps=2.0 / prompts=100 audio_throughput[0] 5.0000 15.3095 16.0750
Qwen Image Layered:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
qwen_image_layered / layered_single_device / 640x640_steps20_i2i / c=1 / prompts=10 throughput_qps[0] 0.0500 0.0669 0.0703
qwen_image_layered / layered_single_device / 640x640_steps20_i2i / c=1 / prompts=10 latency_mean[0] 20 14.9558 14.2080
qwen_image_layered / layered_single_device / 640x640_steps20_i2i / c=1 / prompts=10 peak_memory_mb_max[0] 63000 63730.0000 60543.5000
qwen_image_layered / layered_single_device / 640x640_steps20_i2i / c=1 / prompts=10 peak_memory_mb_mean[0] 63000 63730.0000 60543.5000
qwen_image_layered / layered_single_device / 1024x1024_steps35_i2i / c=1 / prompts=10 throughput_qps[0] 0.0300 0.0397 0.0417
qwen_image_layered / layered_single_device / 1024x1024_steps35_i2i / c=1 / prompts=10 latency_mean[0] 30 25.1726 23.9140
qwen_image_layered / layered_single_device / 1024x1024_steps35_i2i / c=1 / prompts=10 peak_memory_mb_max[0] 63000 63730.0000 60543.5000
qwen_image_layered / layered_single_device / 1024x1024_steps35_i2i / c=1 / prompts=10 peak_memory_mb_mean[0] 63000 63730.0000 60543.5000
Wan2.2:
Config Metric Baseline before this PR Jun 1-8 average Updated baseline
wan2.2 / single_device / 832x480_frames81_steps4 / c=1 / prompts=10 throughput_qps[0] 0.0400 0.0443 0.0465
wan2.2 / single_device / 832x480_frames81_steps4 / c=1 / prompts=10 latency_mean[0] 30 22.5636 21.4354
wan2.2 / single_device / 832x480_frames81_steps4 / c=1 / prompts=10 peak_memory_mb_mean[0] 75000 76376.4000 72557.5800
wan2.2 / usp2_vae_patch2_hsdp_slicing / 832x480_frames81_steps4 / c=1 / prompts=10 throughput_qps[0] 0.0600 0.0611 0.0642
wan2.2 / usp2_vae_patch2_hsdp_slicing / 832x480_frames81_steps4 / c=1 / prompts=10 latency_mean[0] 20 16.3778 15.5589
wan2.2 / usp2_vae_patch2_hsdp_slicing / 832x480_frames81_steps4 / c=1 / prompts=10 peak_memory_mb_mean[0] 48000 48150.4000 45742.8800
wan2.2 / usp2_vae_patch2_hsdp_slicing / 1280x720_frames121_steps4 / c=1 / prompts=10 throughput_qps[0] 0.0100 0.0122 0.0128
wan2.2 / usp2_vae_patch2_hsdp_slicing / 1280x720_frames121_steps4 / c=1 / prompts=10 latency_mean[0] 90 81.9542 77.8565
wan2.2 / usp2_vae_patch2_hsdp_slicing / 1280x720_frames121_steps4 / c=1 / prompts=10 peak_memory_mb_mean[0] 56000 56787.2000 53947.8400
Hunyuan Image3: Fix some metric based on historical data, and these decreases are not regression.
Config Metric Baseline before this PR Updated baseline
test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / c=16 / prompts=96 throughput_qps[0] 0.1000 0.1874
test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / c=16 / prompts=96 latency_p99[0] 44.4600 101.8520
test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / c=16 / prompts=96 latency_mean[0] 35.5700 79.0385
test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / c=16 / prompts=96 peak_memory_mb_max[0] 70000 70000
test_hunyuan_image3_ti2i_inline / inline_dataset_5_requests / c=16 / prompts=96 peak_memory_mb_mean[0] 70000 70000
test_hunyuan_image_tp4 / 1024x1024_steps8 / c=1 / prompts=10 throughput_qps[0] 0.2100 0.2130
test_hunyuan_image_tp4 / 1024x1024_steps8 / c=1 / prompts=10 latency_mean[0] N/A 4.6953
test_hunyuan_image_tp4 / 1024x1024_steps8 / c=1 / prompts=10 latency_p99[0] 4.9235 4.8601
test_hunyuan_image_tp4 / 1024x1024_steps8 / c=1 / prompts=10 peak_memory_mb_max[0] 57576 56912.0000
test_hunyuan_image_tp4 / 1024x1024_steps8 / c=1 / prompts=10 peak_memory_mb_mean[0] 57576 56912.0000

Several baselines didn't update due to regression:

File Config Metric n Baseline before PR Average Candidate Current baseline
test_tts.json customvoice / seed-tts-design / qps=2.0 / prompts=100 audio_throughput[0] 8 9.0000 8.0560 8.4588 9.0000
test_qwen3_omni_async_chunk.json audio/omni random c=8 prompts=32 mean_ttft_ms[2] 8 400.0000 443.5706 421.3921 400.0000

History data for those baselines:

TTS CustomVoice seed-tts-design qps=2.0 audio_throughput (current baseline = 9):
Week n Weekly avg 5% Improve Candidate Baseline vs avg
2026-W18 1 9.1691 9.6276 -1.8%
2026-W19 7 9.1159 9.5717 -1.3%
2026-W20 6 9.1867 9.6460 -2.0%
2026-W21 7 9.2693 9.7328 -2.9%
2026-W22 7 9.2966 9.7614 -3.2%
2026-W23 7 7.9049 8.3002 +13.9%
2026-W24 7 9.2932 9.7578 -3.2%
2026-W25 1 9.2939 9.7585 -3.2%
Qwen3 Omni Async random c=8 mean_ttft_ms (current baseline = 400)
Week n Weekly avg 5% Improve Candidate Baseline vs avg
2026-W17 3 4908.4158 4662.9950 -91.9%
2026-W18 6 3427.4583 3256.0854 -88.3%
2026-W19 7 4219.1289 4008.1725 -90.5%
2026-W20 7 904.6862 859.4519 -55.8%
2026-W21 6 430.1859 408.6766 -7.0%
2026-W22 7 420.7177 399.6818 -4.9%
2026-W23 7 441.6766 419.5928 -9.4%
2026-W24 7 668.4498 635.0273 -40.2%
2026-W25 1 426.3967 405.0769 -6.2%

Prompts you might need:

Baseline Update Rule

  1. Data source: local nightly raw data under vllm-omni-kanban/data/buildkite_nightly_raw.
  2. Date range: June 1 to June 8, corresponding to Buildkite builds 10852 through 11204.
    • If fewer than 8 nightly runs are available, the metric may still be updated if the missing data is not more than 3 days. In that case, compute the average from the available runs only.
    • If fewer than 5 valid runs are available, do not update that metric.
    • Please adjust the denominator correctly if historical data is missing.
  3. Calculation: for each test case and each metric, compute the average value across the 8 nightly runs.
  4. Metric direction:
    • Higher is better: e.g. throughput_qps, qps, audio_throughput.
    • Lower is better: e.g. latency_mean, latency_p99, peak_memory_mb_max, peak_memory_mb_mean, memory, latency, RTF, TTFP.
  5. Candidate baseline:
    • For higher-is-better metrics: candidate_baseline = average * 1.05
    • For lower-is-better metrics: candidate_baseline = average * 0.95
  6. Compare the candidate baseline with the baseline before this PR:
    • For higher-is-better metrics: update only if candidate_baseline > previous_baseline.
    • For lower-is-better metrics: update only if candidate_baseline < previous_baseline.
  7. If the previous baseline is already better than the candidate baseline, keep the previous baseline unchanged.
  8. Round calculated baseline values to four decimal places.
  9. Only update the explicitly requested file/model. Do not automatically expand the change to unrelated files or models.
  10. If the measured memory value is 0, treat it as missing or invalid memory data and do not use it to update the baseline. Keep the previous baseline for that memory metric.

Table Format

Config Metric Baseline before this PR Jun 1-8 average Updated baseline
single 512 throughput_qps[0] 0.1020 0.1075 0.1129
single 512 latency_mean[0] 9.7810 9.3002 8.8352

Column Meanings

  • Config: readable test case name, such as single 512 or cache_dit 1536.
  • Metric: metric name from the baseline block.
  • Baseline before this PR: value before the update.
  • Jun 1-8 average: average value computed from historical nightly data.
  • Updated baseline: final value written to the JSON file after applying the 5% improvement rule.

Test Result

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please paste the results comparison before and after, or the e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, please read https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md
(anything written below this line will be removed by GitHub Actions)

@congw729
congw729 requested a review from yenuo26 as a code owner June 9, 2026 09:06
hsliuustc0106
hsliuustc0106 reviewed Jun 9, 2026
View reviewed changes

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance baseline updated.

@hsliuustc0106

hsliuustc0106 commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

update it asap

@congw729 congw729 mentioned this pull request Jun 11, 2026
Closed
1 task done
@fhfuih

fhfuih commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Updated qwen image edit series, directly pushed to this PR 🦄 💅 🕺

@congw729
congw729 force-pushed the monthly-update-baseline branch 2 times, most recently from 45d2718 to 48d709e Compare June 11, 2026 09:16
congw729 and others added 7 commits June 12, 2026 15:07
@congw729
Update baseline
5eafd32 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@amy-why-3459 @congw729
update baseline (#8)
9926116 
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
@herotai214 @congw729
Updated Qwen-image and Qwen-image-layered baseline
683da45 
Signed-off-by: herotai214 <herotai214@gmail.com>
@bjf-frz @congw729
Update Wan2.2 perf baselines
8ca992f 
Signed-off-by: bjf-frz <frz123db@gmail.com>
@Bounty-hunter @congw729
hunyuan baseline
2edd501 
Signed-off-by: dengyunyang <584797741@qq.com>
@fhfuih @congw729
adjust qwen image edit perf baseline
b539266 
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
@Shirley125 @congw729
voxcpm2 baseline update
e3f9e15 
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
(cherry picked from commit 5ec51d8)
@congw729
congw729 force-pushed the monthly-update-baseline branch from 9b6319a to c8aac4f Compare June 12, 2026 07:07
R2-Y and others added 2 commits June 12, 2026 15:20
@R2-Y @congw729
update tts baseline (#13)
ec46d87 
Signed-off-by: rein yang <73573651+R2-Y@users.noreply.github.com>

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Update bagel baseline
486a901 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
congw729 force-pushed the monthly-update-baseline branch from c8aac4f to 486a901 Compare June 12, 2026 07:21
@congw729

congw729 commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

@hsliuustc0106 @Gaohan123 PR ready.

hsliuustc0106
hsliuustc0106 reviewed Jun 15, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_bagel_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_bagel_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_bagel_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_hunyuan_image3_it2i.json Outdated
yenuo26
yenuo26 reviewed Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_bagel_vllm_omni.json Outdated
Comment thread tests/dfx/perf/tests/test_bagel_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_qwen_image_vllm_omni.json Outdated
@congw729

congw729 commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator Author

@herotai214 @NumberWan Please double-check the stats for Qwen Image. We need the baseline with 5% optimization from the 7-day history.

congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_qwen_image_layered_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_qwen_image_layered_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json Outdated
congw729
congw729 commented Jun 16, 2026
View reviewed changes
Comment thread tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json Outdated
@natureofnature natureofnature mentioned this pull request Jun 16, 2026
Merged
congw729 and others added 14 commits June 16, 2026 15:54
@congw729
Update qwen image edit
cdfdacc 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Revert bagel baseline due to regression. Remove hunyuan memory updation.
dbacac7 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Update qwen image 2511 baseline
71cf59d 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Hunyuan metirc round to 4 decimals.
37502ca 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Update qwen 3 tts baseline
7a0a6de 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@NumberWan
Update Qwen Image and Qwen Image Layered perf baselines
53b750a 
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
@congw729
Update voxcpm2 baseline
7fda3e8 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Merge remote-tracking branch 'origin/monthly-update-baseline' into mo…
9bda2d8 
…nthly-update-baseline

Co-authored-by: Cursor <cursoragent@cursor.com>

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@bjf-frz
Fix Wan2.2 baseline update
6d27b9b 
Signed-off-by: bjf-frz <frz123db@gmail.com>
@congw729
Merge branch 'main' into monthly-update-baseline
39a587a
@congw729
Update wan2.2
8b2b2b7 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Merge remote-tracking branch 'refs/remotes/origin/monthly-update-base…
81760de 
…line' into monthly-update-baseline
@congw729
Update qwen3omni and qwen3tts
8352a34 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Merge branch 'main' into monthly-update-baseline
80ba4c1
@congw729

congw729 commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

@hsliuustc0106 Ready for merge

@linyueqian

linyueqian commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

LGTM. Most baselines tighten toward measured values. Noting for the record that 4 latency values relax versus the prior baseline, which is expected here and does not loosen any enforced gate:

  • tests/dfx/perf/tests/test_hunyuan_image3_it2i.json: latency_mean (35.57 to 79.04) and latency_p99 (44.46 to 101.85). This block sets "skip-performance-assertion": true, so these are not asserted by the diffusion gate, and throughput_qps in the same cell improved (0.1 to 0.187).
  • tests/dfx/perf/tests/test_tts.json default_voice c=64: median_audio_ttfp_ms[2] (5000 to 5942) and median_audio_rtf[2] (1.10 to 1.166). The old values were round placeholders replaced with measured numbers, and the same cell audio_throughput improved (40 to 48.3).

Everything else moves in the tightening direction. Thanks for the update.

@congw729

congw729 commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

LGTM. Most baselines tighten toward measured values. Noting for the record that 4 latency values relax versus the prior baseline, which is expected here and does not loosen any enforced gate:

  • tests/dfx/perf/tests/test_hunyuan_image3_it2i.json: latency_mean (35.57 to 79.04) and latency_p99 (44.46 to 101.85). This block sets "skip-performance-assertion": true, so these are not asserted by the diffusion gate, and throughput_qps in the same cell improved (0.1 to 0.187).
  • tests/dfx/perf/tests/test_tts.json default_voice c=64: median_audio_ttfp_ms[2] (5000 to 5942) and median_audio_rtf[2] (1.10 to 1.166). The old values were round placeholders replaced with measured numbers, and the same cell audio_throughput improved (40 to 48.3).

Everything else moves in the tightening direction. Thanks for the update.

For hunyuan, it's also a fixing based on historical data.

@hsliuustc0106
hsliuustc0106 merged commit 1761253 into vllm-project:main Jun 17, 2026
5 checks passed
@congw729 congw729 mentioned this pull request Jun 18, 2026
Merged
Nughm3 pushed a commit to Nughm3/vllm-omni that referenced this pull request Jun 18, 2026
@congw729 @amy-why-3459
@herotai214 @bjf-frz @Bounty-hunter @fhfuih @Shirley125 @R2-Y @NumberWan @Nughm3
[CI] Update perf json baselines (vllm-project#4289)
6920e25 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: dengyunyang <584797741@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: dengyunyang <584797741@qq.com>
Co-authored-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: rein yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: NumberWan <wantszkin2003@gmail.com>
alex-jw-brooks pushed a commit to alex-jw-brooks/vllm-omni that referenced this pull request Jun 23, 2026
@congw729 @amy-why-3459
@herotai214 @bjf-frz @Bounty-hunter @fhfuih @Shirley125 @R2-Y @NumberWan @alex-jw-brooks
[CI] Update perf json baselines (vllm-project#4289)
9bad0be 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: dengyunyang <584797741@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: dengyunyang <584797741@qq.com>
Co-authored-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: rein yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: NumberWan <wantszkin2003@gmail.com>
@congw729 congw729 mentioned this pull request Jul 20, 2026
Merged
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
@congw729 @amy-why-3459
@herotai214 @bjf-frz @Bounty-hunter @fhfuih @Shirley125 @R2-Y @NumberWan
[CI] Update perf json baselines (vllm-project#4289)
34b2b1d 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Signed-off-by: dengyunyang <584797741@qq.com>
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>
Signed-off-by: NumberWan <wantszkin2003@gmail.com>
Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: dengyunyang <584797741@qq.com>
Co-authored-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
Co-authored-by: CHEN <116010019@link.cuhk.edu.cn>
Co-authored-by: rein yang <73573651+R2-Y@users.noreply.github.com>
Co-authored-by: NumberWan <wantszkin2003@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yenuo26 yenuo26 yenuo26 left review comments
@hsliuustc0106 hsliuustc0106 hsliuustc0106 left review comments
@linyueqian linyueqian linyueqian approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.