Repository navigation
[CI] optional --assert-baseline and update perf JSON baselines - #3695
Merged
hsliuustc0106 merged 7 commits intoMay 20, 2026
Merged
Conversation
This commit introduces a new command-line option, `--assert-baseline`, to the performance benchmarking scripts. When enabled, it allows the comparison of benchmark metrics against the baseline values defined in the JSON configuration files. The option is integrated into the `pytest` configuration and affects the behavior of the `assert_result` function in both `run_benchmark.py` and `run_diffusion_benchmark.py`. Additionally, several performance test JSON files have been updated to reflect new baseline values for various metrics. Signed-off-by: wangyu <410167048@qq.com>
yenuo26
changed the title
Collaborator
Author
hsliuustc0106
left a comment
hsliuustc0106
left a comment
Collaborator
There was a problem hiding this comment.
Good pattern for optional baseline assertions. 7-day mean baselines are comprehensive.
yenuo26
force-pushed
the
pr/perf-assert-baseline
branch
from
May 19, 2026 01:22
89368c2 to
3e89d4b
Compare
fhfuih
approved these changes
May 19, 2026
fhfuih
left a comment
fhfuih
left a comment
Contributor
There was a problem hiding this comment.
LGTM for the qwen image edit part
linyueqian
approved these changes
May 19, 2026
linyueqian
left a comment
linyueqian
left a comment
Collaborator
There was a problem hiding this comment.
LGTM for the TTS side. I checked the updated test_tts.json baselines against the configured sweeps: the baseline list lengths line up with num_prompts/max_concurrency/request_rate, JSON is valid, and the optional --assert-baseline behavior keeps baseline assertions disabled by default while preserving the recorded baseline blocks for explicit perf checks. No requested changes from me.
| "mean_audio_ttfp_ms": [1000, 3000, 5000, 7000, 9000], | ||
| "mean_audio_rtf": [0.2, 0.35, 0.6, 0.85, 0.9] | ||
| "mean_ttft_ms": [ | ||
| 300, |
Collaborator
There was a problem hiding this comment.
The baseline data does not quite match the actual data.
Signed-off-by: wangyu <410167048@qq.com>
Collaborator
|
Please resolve the conflict. |
… template This update introduces a fallback mechanism for benchmark results in the `run_benchmark` function. If the expected result file is not generated, the function now uses a predefined JSON template (`result_omni_template.json`) to ensure consistent output. Additionally, the function's docstring has been updated to clarify the behavior regarding baseline metrics and the fallback process. Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Collaborator
Author
fixed |
Collaborator
|
LGTM |
amy-why-3459
approved these changes
May 19, 2026
hsliuustc0106
approved these changes
May 19, 2026
hsliuustc0106
left a comment
hsliuustc0106
left a comment
Collaborator
There was a problem hiding this comment.
What I checked
- DCO: PASSED
- Mergeability: MERGEABLE
- Template file:
result_omni_template.jsonexists on main — the fallback path inconftest.pyresolves correctly - Deduplication:
_resolve_baseline_valueand_safe_filename_tokencorrectly removed fromrun_benchmark.py; the renamedresolve_baseline_valueis imported fromconftest.py - Flag default:
--assert-baselinedefaults toFalse— no breaking change to default CI - Baseline values: Spot-checked several entries against the nightly tables; thresholds are conservative and should catch real regressions without false positives
- PR description evidence: 7-day nightly benchmark tables are comprehensive; test results screenshots are provided
Verdict
No blocking issues. The flag is opt-in, the deduplication is clean, and baselines are data-driven from nightly runs. Domain experts have already approved their areas. LGTM once CI is green.
2 of 13 tasks
zengchuang-hw
pushed a commit
to zengchuang-hw/vllm-omni
that referenced
this pull request
Jun 1, 2026
…project#3695) Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
5 tasks
Nughm3
pushed a commit
to Nughm3/vllm-omni
that referenced
this pull request
Jun 18, 2026
…project#3695) Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
khairulkabir1661
pushed a commit
to khairulkabir1661/vllm-omni
that referenced
this pull request
Sep 25, 2026
…project#3695) Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.
Purpose
--assert-baseline(registered intests/dfx/conftest.py, default off) so omni serve benchmarks and diffusion benchmarks can optionally compare metrics to thebaselineblocks in perf JSON configs.tests/dfx/perf/scripts/run_benchmark.py: when the flag is set, restore baseline resolution /result["baseline"]writing and the previous print-only throughput/latency checks against baselines; when unset, only assert request completion counts.tests/dfx/perf/scripts/run_diffusion_benchmark.py: when the flag is set, restore strictassertcomparisons (and honorskip-performance-assertion); when unset, only assert completed request counts.test_qwen3_omni_*,test_qwen_image_*,test_wan22_i2v_vllm_omni.json,test_tts.json) per agreed rounding rules from benchmark tables.Nightly Baseline Reference(7 days mean)
Qwen-image
Qwen-image-layered
Qwen-Image-edit
Qwen-Image-edit-2509
WAN2.2
Qwen3-Omni
Qwen3-TTS
Test Plan
python -m json.toolon touchedtests/dfx/perf/tests/*.json(already valid JSON).--assert-baseline(default):--assert-baselineto confirm baseline checks run when explicitly requested:(Same pattern for
run_diffusion_benchmark.pywith an appropriate--test-config-file.)Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model. Please runmkdocs serveto sync the documentation editions to./docs.BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)