[Perf] Add Performance Test for Qwen-Image Step-Level Execution

[wtomin]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

To monitor the step-level execution feature's performance on Qwen-Image, we need to include this test case in nightly perf test.

Test Plan

pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json

Running it on CI machines (H100) . Results in buildkite.

Test Result

Test	Resolution × steps	latency_mean (s)	latency_p50 (s)	throughput_qps	QwenImagePipeline.diffuse p50 (s)
test_qwen_image_single_device	512×512, 20	2.62	2.51	0.38	2.32
test_qwen_image_single_device	1536×1536, 35	24.45	24.43	0.04	23.19
test_qwen_image_single_device_step_execution	512×512, 20	2.48	2.48	0.40	—
test_qwen_image_single_device_step_execution	1536×1536, 35	24.46	24.42	0.04	—

Conclusion:
step_execution does not change end-to-end latency/throughput at the large resolution, and gives a small gain at the small resolution.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[gcanlin]

Could we cover this issue #845 in the tests? Although it's a functional issue actually.

[wtomin]

It depends on whether we want to monitor its performance in the long run. I think currently, 1328x1328 is not our priority.

[hsliuustc0106]

PR description incomplete. Test Plan and Test Result sections are empty.

Please update with:

• How baselines were derived (hardware, actual test run that produced these numbers)
• Simple test showing this config runs successfully

[wtomin]

PR description incomplete. Test Plan and Test Result sections are empty.

Please update with:

• How baselines were derived (hardware, actual test run that produced these numbers)
• Simple test showing this config runs successfully

It has been updated.

[wtomin]

@asukaqaq-s Do this performance test and results look good to you? The results shown in this PR may not be compared directly to the results in #1769, because they might be obtained on different machines.

[asukaqaq-s]

@asukaqaq-s Do this performance test and results look good to you? The results shown in this PR may not be compared directly to the results in #1769, because they might be obtained on different machines.

I think this conclusion is consistent with my tests: there is some performance improvement at low resolutions.

[wtomin]

I think nightly-gedit-bench-accuracy failed occasionly probably because the threshold is too tight. @david6666666 @yenuo26 Any comments?

[yenuo26]

I think nightly-gedit-bench-accuracy failed occasionly probably because the threshold is too tight. @david6666666 @yenuo26 Any comments?

#2644 this issue

[wtomin]

I think it's ready for merge. PTAL @hsliuustc0106 @Gaohan123 @fhfuih

[Gaohan123]

LGTM. Thanks