Skip to content

[Perf] Qwen-Image Nightly Performance CI Improvement#2111

Merged
Gaohan123 merged 8 commits into
vllm-project:mainfrom
wtomin:fix-test
Mar 24, 2026
Merged

[Perf] Qwen-Image Nightly Performance CI Improvement#2111
Gaohan123 merged 8 commits into
vllm-project:mainfrom
wtomin:fix-test

Conversation

@wtomin
Copy link
Copy Markdown
Collaborator

@wtomin wtomin commented Mar 24, 2026

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

  1. Inlined benchmark_params into test config files

Replaced all {"$ref": "benchmark_params.json"} references in test_qwen_image_sglang_diffusion.json and test_qwen_image_vllm_omni.json with the full inline parameter arrays (3 benchmark cases each: 512x512_steps20, 1536x1536_steps35, mixed_resolution).
Deleted tests/perf/tests/benchmark_params.json.

  1. Populated baselines with real measured results

Extracted throughput_qps, latency_p99, peak_memory_mb_max, peak_memory_mb_mean from results.json for all 9 combinations (3 test cases × 3 benchmark params) and wrote them into the baseline blocks of test_qwen_image_vllm_omni.json.

  1. Relocated tests/perf/ into tests/dfx/perf/

Used git mv to properly rename all three tracked files:
tests/perf/scripts/run_diffusion_benchmark.py → tests/dfx/perf/scripts/
tests/perf/tests/test_qwen_image_sglang_diffusion.json → tests/dfx/perf/tests/
tests/perf/tests/test_qwen_image_vllm_omni.json → tests/dfx/perf/tests/

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please paste the results comparison before and after, or the e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

wtomin added 6 commits March 24, 2026 10:49
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Made-with: Cursor

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Made-with: Cursor

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
@wtomin wtomin marked this pull request as ready for review March 24, 2026 03:27
@wtomin wtomin requested a review from hsliuustc0106 as a code owner March 24, 2026 03:27
@wtomin
Copy link
Copy Markdown
Collaborator Author

wtomin commented Mar 24, 2026

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 477702ab2c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread .buildkite/test-nightly.yml
Comment thread tests/dfx/perf/scripts/run_diffusion_benchmark.py Outdated
Comment thread tests/dfx/perf/scripts/run_diffusion_benchmark.py Outdated
Comment thread tests/dfx/perf/scripts/run_diffusion_benchmark.py Outdated
Made-with: Cursor

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Copy link
Copy Markdown
Collaborator

@congw729 congw729 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@congw729
Copy link
Copy Markdown
Collaborator

Please rebase to the latest origin/main branch and the Doc failed builds will be resolved.

@gcanlin gcanlin added the ready label to trigger buildkite CI label Mar 24, 2026
Copy link
Copy Markdown
Collaborator

@Gaohan123 Gaohan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Gaohan123 Gaohan123 merged commit c392257 into vllm-project:main Mar 24, 2026
8 checks passed
zhangj1an pushed a commit to zhangj1an/vllm-omni that referenced this pull request Mar 26, 2026
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
zhangj1an pushed a commit to zhangj1an/vllm-omni that referenced this pull request Mar 26, 2026
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
lengrongfu pushed a commit to lengrongfu/vllm-omni that referenced this pull request May 1, 2026
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
clodaghwalsh17 pushed a commit to clodaghwalsh17/nm-vllm-omni-ent that referenced this pull request May 12, 2026
Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants