[Refactor][CI] Reorganize test tree, and move slow e2e to weekly by yenuo26 · Pull Request #4951 · vllm-project/vllm-omni

yenuo26 · 2026-07-08T02:18:15Z

PLEASE FILL IN THE PR DESCRIPTION HERE.

Purpose

Improve CI signal-to-noise and align the tests/ tree with component boundaries so new tests land in predictable places.

CI / marker routing

Move heavyweight TTS / robot e2e out of merge CI into weekly expansion (*_expansion.py + slow marker); weekly jobs collect under tests/e2e/ with markers like slow and hardware_test(...) and tts.
Remove deprecated GEBench / GEdit H100 smoke tests.

Test tree reorganization

Move root-level / misc unit tests into component dirs: tests/config/, tests/engine/, tests/diffusion/, tests/utils/, tests/tools/, tests/model_executor/models/, etc.
Reorganize e2e layout:
- tests/e2e/features/ for custom pipeline, RLHF (VeRL-Omni), fullduplex, ComfyUI
- keep model e2e under tests/e2e/online_serving/, tests/e2e/offline_inference/, tests/e2e/accuracy/
Consolidate accuracy assets under tests/assets/; helpers use shared get_asset_path().
Move tests/ar_diffusion/ to tests/diffusion/ar_diffusion/.
Move DreamZero unit tests to tests/diffusion/models/dreamzero/.
Move GGUF diffusion e2e to tests/diffusion/quantization/test_gguf_diffusion.py.
Move Step-Audio2 CI stage configs to vllm_omni/deploy/step_audio2_ci.yaml / step_audio2_thinker_ci.yaml; tests use get_deploy_config_path().
Consolidate OpenPI robot websocket helpers into tests/helpers/runtime.py; remove thin tests/gr00t/ / tests/dreamzero/ client wrappers.
Extract VeRL-Omni Ray actor to tests/e2e/features/helpers/verl_omni_server.py (fixes Ray unpickle ModuleNotFoundError: test_verl_omni_e2e).
Merge custom-pipeline test doubles into tests/e2e/features/helpers/custom_pipeline.py.
Document placement rules in docs/contributing/ci/test_system_overview.md / test_writing_guide.md.

Test Plan

vLLM Version: N/A (test layout / CI routing; no model runtime API change required for review)

vLLM-Omni Commit: 300b26d4

Ready CI

bash tools/run_ready_jobs.sh

Merge CI

bash tools/run_merge_jobs.sh

Nightly CI

bash tools/nightly/run_nightly_jobs.sh --test-type function

Test Result

Ready CI

Merge CI

Nightly CI

Note: screenshots above are from earlier validation runs on this branch. Re-run Ready / Merge / Nightly after the latest rebase onto main if reviewers need fresh green.

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

chatgpt-codex-connector · 2026-07-08T02:18:21Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

yenuo26 · 2026-07-27T08:16:54Z

@Gaohan123 @alex-jw-brooks @lishunyang12 please help to add ready and merge-test label

hsliuustc0106 · 2026-07-27T14:51:54Z

fix conflicts

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 · 2026-07-28T03:08:12Z

fix conflicts

fixed

- Updated MockScheduler in test_diffusion_engine.py to include methods for counting waiting and running requests, and adjusted request batch settings for improved latency handling. - Refactored test_paged_attention.py to ensure proper cleanup of the CompileCounter and maintain a clean state for subsequent tests. - Added compatibility checks in test_dynin_omni_token2audio.py to prevent permanent mutations of PreTrainedModel attributes during tests. - Introduced a CPU fallback for apply_top_k_top_p in test_higgs_audio_v3.py to ensure compatibility with CPU tensors. - Renamed decoding methods in test_codec_cudagraph.py for consistency and clarity in the interface. These changes aim to improve the robustness and reliability of the test suite. Signed-off-by: wangyu <410167048@qq.com>

yenuo26 · 2026-07-28T12:36:07Z

update branch

Ready CI

pytest -sv tests/diffusion tests/model_executor -m 'core_model and cpu'

pytest -sv tests/engine/ tests/model_executor/ -m 'core_model and cuda' --run-level 'core_model'

Merge CI

Nightly CI

Signed-off-by: wangyu <410167048@qq.com>

- Modified test commands in `test-merge.yml` and `test-ready.yml` to include the `tests/` directory for pytest. - Removed Kubernetes agent configuration from `test-merge.yml`. - Added new utility functions in `helpers.py` for resetting artifact directories and inferring model labels. Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

- Introduced a new conftest.py file to set up the test environment for ComfyUI plugin testing. - Added functionality to mock necessary modules (comfy_api and comfy_extras) and their components. - Updated test_comfyui_integration.py to remove redundant setup code, leveraging the new conftest.py for environment configuration. Signed-off-by: wangyu <410167048@qq.com>

- Updated test labels and commands in `test-merge.yml` and `test-ready.yml` to separate diffusion, model executor, and entrypoints tests for clarity. - Adjusted the ignore parameters in the "Simple · Other Test" section to exclude the newly added entrypoints tests. - Enhanced the `FakeDeadLLMStageClient` in `test_orchestrator_error_handling.py` to handle request states more accurately. Signed-off-by: wangyu <410167048@qq.com>

- Renamed the "Simple · Entrypoints Test" to "Simple · Engine&Entrypoints Test" for clarity. - Updated test commands to include the `tests/engine` directory in the entrypoints test. - Adjusted the ignore parameters in the "Simple · Other Test" section to exclude the `tests/engine` directory. Signed-off-by: wangyu <410167048@qq.com>

Gaohan123

LGTM. Thanks

…m-project#4951) Signed-off-by: wangyu <410167048@qq.com>

yenuo26 requested a review from congw729 as a code owner July 8, 2026 02:18

yenuo26 changed the title ~~[CI] Tier diffusion offloader tests and move slow TTS to weekly expansion~~ [WIP][CI] Tier diffusion offloader tests and move slow TTS to weekly expansion Jul 16, 2026

yenuo26 requested review from Gaohan123, Sy0307, david6666666, hsliuustc0106, tzhouam and ywang96 as code owners July 16, 2026 13:10

yenuo26 removed request for Gaohan123, Sy0307, congw729, david6666666, hsliuustc0106, tzhouam and ywang96 July 16, 2026 13:11

hsliuustc0106 added diffusion codes related to diffusion models tts code related to tts models labels Jul 17, 2026 — with ChatGPT Codex Connector

yenuo26 requested a review from lishunyang12 as a code owner July 18, 2026 03:59

yenuo26 changed the title ~~[WIP][CI] Tier diffusion offloader tests and move slow TTS to weekly expansion~~ [WIP][Refactor][CI] Reorganize test tree, tier diffusion offloader CI, and move slow e2e to weekly Jul 18, 2026

yenuo26 added CI/CD codes related to changes to CI/CD refactor refactoring for better code scalability and quality and removed diffusion codes related to diffusion models tts code related to tts models labels Jul 18, 2026

yenuo26 changed the title ~~[WIP][Refactor][CI] Reorganize test tree, tier diffusion offloader CI, and move slow e2e to weekly~~ [WIP][Refactor][CI] Reorganize test tree Jul 18, 2026

yenuo26 changed the title ~~[WIP][Refactor][CI] Reorganize test tree~~ [WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 18, 2026

yenuo26 changed the title ~~[WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly~~ [Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 23, 2026

yenuo26 changed the title ~~[Refactor][CI] Reorganize test tree, and move slow e2e to weekly~~ [WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 23, 2026

Gaohan123 added this to the v0.26.0 milestone Jul 23, 2026

yenuo26 requested a review from gcanlin as a code owner July 27, 2026 06:32

yenuo26 force-pushed the mv_testcase branch from 05e6157 to 300b26d Compare July 27, 2026 06:43

yenuo26 changed the title ~~[WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly~~ [Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 27, 2026

yenuo26 changed the title ~~[Refactor][CI] Reorganize test tree, and move slow e2e to weekly~~ [WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 27, 2026

yenuo26 changed the title ~~[WIP][Refactor][CI] Reorganize test tree, and move slow e2e to weekly~~ [Refactor][CI] Reorganize test tree, and move slow e2e to weekly Jul 27, 2026

Merge remote-tracking branch 'upstream/main' into mv_testcase

32403ed

Signed-off-by: wangyu <410167048@qq.com>

Merge branch 'main' into mv_testcase

9b87884

Gaohan123 added the ready label to trigger buildkite CI label Jul 28, 2026

yenuo26 removed the ready label to trigger buildkite CI label Jul 28, 2026

yenuo26 added 6 commits July 29, 2026 15:38

Merge remote-tracking branch 'upstream/main' into mv_testcase

2f56295

Signed-off-by: wangyu <410167048@qq.com>

Merge remote-tracking branch 'upstream/main' into mv_testcase

37db41c

Signed-off-by: wangyu <410167048@qq.com>

Gaohan123 added ready label to trigger buildkite CI merge-test label to trigger buildkite merge test CI labels Jul 29, 2026

Gaohan123 approved these changes Jul 29, 2026

View reviewed changes

Gaohan123 merged commit 7c17111 into vllm-project:main Jul 29, 2026
8 of 9 checks passed

nagisa-kunhah pushed a commit to nagisa-kunhah/vllm-omni that referenced this pull request Jul 30, 2026

[Refactor][CI] Reorganize test tree, and move slow e2e to weekly (vll…

4ad81b6

…m-project#4951) Signed-off-by: wangyu <410167048@qq.com>

yashkgp mentioned this pull request Aug 22, 2026

[Model] Add Vevo2 — unified AR + flow-matching TTS (#3391) #3422

Closed

9 of 11 tasks

linyueqian mentioned this pull request Aug 30, 2026

[Feature] Support Qwen3-TTS prefill-decode (PD) disaggregation #4934

Closed

This was referenced Sep 11, 2026

Real-time gen with minimax H3 on SM120 #7415

Open

[Model] Accelerate MiniMax-H3 end-to-end inference #7519

Draft

khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026

[Refactor][CI] Reorganize test tree, and move slow e2e to weekly (vll…

f7b41b2

…m-project#4951) Signed-off-by: wangyu <410167048@qq.com>

yenuo26 deleted the mv_testcase branch September 28, 2026 07:40

Navigation Menu

Repository navigation

[Refactor][CI] Reorganize test tree, and move slow e2e to weekly - #4951

[Refactor][CI] Reorganize test tree, and move slow e2e to weekly#4951
Gaohan123 merged 10 commits into
vllm-project:mainfrom
yenuo26:mv_testcase

yenuo26 commented Jul 8, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Jul 8, 2026

Uh oh!

yenuo26 commented Jul 27, 2026

Uh oh!

hsliuustc0106 commented Jul 27, 2026

Uh oh!

yenuo26 commented Jul 28, 2026

Uh oh!

yenuo26 commented Jul 28, 2026 •

edited

Loading

Uh oh!

Gaohan123 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

yenuo26 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

CI / marker routing

Test tree reorganization

Test Plan

Ready CI

Merge CI

Nightly CI

Test Result

Ready CI

Merge CI

Nightly CI

Uh oh!

chatgpt-codex-connector Bot commented Jul 8, 2026

Uh oh!

yenuo26 commented Jul 27, 2026

Uh oh!

hsliuustc0106 commented Jul 27, 2026

Uh oh!

yenuo26 commented Jul 28, 2026

Uh oh!

yenuo26 commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ready CI

Merge CI

Nightly CI

Uh oh!

Gaohan123 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yenuo26 commented Jul 8, 2026 •

edited

Loading

yenuo26 commented Jul 28, 2026 •

edited

Loading