[CI] remove benchmark/testing comparison w/ other frameworks by fhfuih · Pull Request #2179 · vllm-project/vllm-omni

fhfuih · 2026-03-25T09:30:15Z

Signed-off-by: Huang, Zeyu 11222265+fhfuih@users.noreply.github.com

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

fhfuih · 2026-03-25T09:32:03Z

@wtomin @hsliuustc0106 PTAL thanks

Copilot

Pull request overview

Removes SGLang diffusion benchmark/testing comparison from the CI performance benchmarking flow, leaving vLLM-Omni as the only supported backend for these diffusion perf runs.

Changes:

Deleted the SGLang Qwen-Image diffusion perf test configuration JSON.
Removed the SGLang server runner + Cache-DiT version checks from the diffusion benchmark CI runner script.
Dropped SGLang backend support from the diffusion benchmark serving CLI and backend request mappings.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/dfx/perf/tests/test_qwen_image_sglang_diffusion.json	Removes the SGLang-specific diffusion perf test definitions.
tests/dfx/perf/scripts/run_diffusion_benchmark.py	Removes SGLang server support and enforces vLLM-Omni-only configs.
benchmarks/diffusion/diffusion_benchmark_serving.py	Removes `sglang` from `--backend` CLI choices and updates attribution comment.
benchmarks/diffusion/backends.py	Removes SGLang request implementations and mapping entries.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-25T09:37:03Z

 """
 Performance benchmark CI runner for diffusion models.

-Supports two server backends:
+Supports vLLM-Omni server backend:
  - vllm-omni (default): starts DiffusionServer via vllm_omni.entrypoints.cli.main,
    benchmarks with diffusion_benchmark_serving.py --backend vllm-omni
-  - sglang: starts SglangServer via `sglang serve`,
-    benchmarks with diffusion_benchmark_serving.py --backend sglang

 A config JSON file is REQUIRED via --config-file:
  pytest run_diffusion_benchmark.py --config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
-  pytest run_diffusion_benchmark.py --config-file tests/dfx/perf/tests/test_qwen_image_sglang_diffusion.json

-JSON config entries are distinguished by a "server_type" field ("vllm-omni" or "sglang").
-sglang entries support two additional fields under server_params:
-  - "env": dict of extra environment variables (e.g. SGLANG_CACHE_DIT_ENABLED)
-  - "cache_dit_config": dict written to a temp YAML and passed as
-    --cache-dit-config to sglang serve (requires cache-dit == 1.3.0)
+JSON config entries use a "server_type" field, and this runner executes
+the vllm-omni path.


The PR description is still the unfilled template (missing Purpose/Test Plan/Test Result), which doesn’t match the scope of this change (CI benchmark backend removal). Please update the PR description to include: (1) why SGLang benchmarking is being removed, (2) what CI/jobs are affected, and (3) what you ran (or why no tests are needed) and the results.

Copilot · 2026-03-25T09:37:04Z

        seen.add(test_name)
-        server_type = cfg.get("server_type", "vllm-omni")
+        if cfg.get("server_type", "vllm-omni") != "vllm-omni":
+            raise ValueError(f"Unsupported server_type in config: {cfg.get('server_type')}")


This error message is a bit terse for CI/debugging. Consider including the allowed value(s) and the failing test name to make failures actionable (e.g., mention that only "vllm-omni" is supported and include cfg.get("test_name")).

Suggested change

raise ValueError(f"Unsupported server_type in config: {cfg.get('server_type')}")

raise ValueError(

f"Unsupported server_type {cfg.get('server_type')!r} in config for test "

f"{cfg.get('test_name', '<unknown>')!r}. Only 'vllm-omni' is supported."

)

The suggestion makes it hard for future additions and flexibility

hsliuustc0106

lgtm

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Based on occassional fails such as https://buildkite.com/vllm/vllm-omni/builds/5232/steps/canvas Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Based on https://buildkite.com/vllm/vllm-omni/builds/5318/steps/canvas?sid=019d2cf6-1ec5-4d1c-a30f-e1fa841a1162&tab=output Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>

https://buildkite.com/vllm/vllm-omni/builds/5326/steps/canvas?sid=019d2d14-786e-47e0-a10a-274764d38354&tab=output Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Updates: remove mixed resolution and save server params into output json file

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Gaohan123

LGTM. Thanks

…oject#2179) Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com> Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk> Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com> Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>

[CI] remove SGLang benchmark/testing comparison

c645a6b

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

fhfuih requested a review from hsliuustc0106 as a code owner March 25, 2026 09:30

Copilot AI review requested due to automatic review settings March 25, 2026 09:30

Copilot AI reviewed Mar 25, 2026

View reviewed changes

Copilot started reviewing on behalf of fhfuih March 25, 2026 09:51 View session

hsliuustc0106 added the ready label to trigger buildkite CI label Mar 25, 2026

hsliuustc0106 approved these changes Mar 25, 2026

View reviewed changes

fhfuih changed the title ~~[CI] remove SGLang benchmark/testing comparison~~ [CI] remove benchmark/testing comparison w/ other frameworks Mar 26, 2026

Gaohan123 added nightly-test label to trigger buildkite nightly test CI and removed ready label to trigger buildkite CI labels Mar 26, 2026

fhfuih and others added 3 commits March 26, 2026 10:11

trigger nightly CI

9bf8d00

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

trigger nightly CI once more

034a61f

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Merge branch 'main' into remove-comparative-bench

7ca2787

fhfuih mentioned this pull request Mar 26, 2026

[CI] Qwen image edit performance benckmark #2216

Merged

5 tasks

congw729 and others added 3 commits March 26, 2026 14:32

Merge branch 'main' into remove-comparative-bench

d0f038f

[CI] lower Qwen Image perf test threshold

ecb756e

Based on occassional fails such as https://buildkite.com/vllm/vllm-omni/builds/5232/steps/canvas Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Merge branch 'main' into remove-comparative-bench

45f918c

wtomin added the ready label to trigger buildkite CI label Mar 27, 2026

wtomin approved these changes Mar 27, 2026

View reviewed changes

fhfuih and others added 8 commits March 27, 2026 10:07

Further lower threshold

a29fb89

Based on https://buildkite.com/vllm/vllm-omni/builds/5318/steps/canvas?sid=019d2cf6-1ec5-4d1c-a30f-e1fa841a1162&tab=output Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

remove mixed resolution

ccdbaab

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>

save server params in json

6eb6826

Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com>

Further adjust threshold

d7eb734

https://buildkite.com/vllm/vllm-omni/builds/5326/steps/canvas?sid=019d2d14-786e-47e0-a10a-274764d38354&tab=output Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Merge pull request #36 from wtomin/remove-ci

a73a253

Updates: remove mixed resolution and save server params into output json file

Replace max latency with mean; remove max memory

380d679

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Merge branch 'main' into remove-comparative-bench

b750b36

Further reduce threshold

7dd6268

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

update

ed64ffc

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Gaohan123 approved these changes Mar 27, 2026

View reviewed changes

Gaohan123 merged commit c139ee4 into vllm-project:main Mar 27, 2026
6 of 8 checks passed

fhfuih deleted the remove-comparative-bench branch March 27, 2026 09:30

BBuf mentioned this pull request Apr 28, 2026

SGLang Diffusion 外部影响力调研：kernel、feature 与平台采用情况 BBuf/how-to-optim-algorithm-in-cuda#14

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] remove benchmark/testing comparison w/ other frameworks#2179

[CI] remove benchmark/testing comparison w/ other frameworks#2179
Gaohan123 merged 16 commits into
vllm-project:mainfrom
fhfuih:remove-comparative-bench

fhfuih commented Mar 25, 2026

Uh oh!

fhfuih commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

Copilot AI Mar 25, 2026

Uh oh!

fhfuih Mar 25, 2026

Uh oh!

hsliuustc0106 left a comment

Uh oh!

Gaohan123 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

fhfuih commented Mar 25, 2026

Purpose

Test Plan

Test Result

Uh oh!

fhfuih commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

fhfuih Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Gaohan123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants