[CI] Restructure vLLM-Omni Test Layout, Fixture Scope, and Support Modules by yenuo26 · Pull Request #2620 · vllm-project/vllm-omni

yenuo26 · 2026-04-09T02:57:13Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

due to #2299
Background and Goals
The original root directory tests/conftest.py centralized a large number of fixtures, assertions, media, and runtime logic, which could easily lead to:

Premature loading of heavy dependencies (vLLM / vllm_omni) during the pytest plugin loading phase, causing conflicts with session-level autouse fixtures (such as environment variables);
Difficulty in reusing test helper code and splitting responsibilities;
Mixing capabilities like hardware_test in tests/utils.py, blurring the boundary with code "for testing only."
Media is regenerated every time, which takes a lot of time.
In the Buildkite log, the test run output and the final result summary are not separated by a foldable group, which makes them hard to read.

This PR aims to thin the entry point, move implementations outward, modularize fixtures as plugins, and clarify import path instructions in the documentation.

Overview of Main Changes

Category	Description
Root `conftest.py`	1）Only responsible for: `pytest_plugins` registration, backward-compatible re-exports from `tests.helpers.*`, and lazy loading of runtime symbols via `__getattr__` to avoid immediately importing `tests.helpers.runtime` when `conftest` loads. 2）Add '--- Running Summary' before the pytest summary to create a foldable group in Buildkite.
`tests/helpers/`	1）Reusable helper implementations: assertions, env, media, mark, process, runtime, stage_config, etc.; `__init__.py` deliberately avoids star-import aggregation to prevent altering import order. 2）Refactor media helper functions to support caching of synthetic audio, video, and image generation.
`tests/helpers/fixtures/`	Loaded via `pytest_plugins`: env, log, run_args, runtime; runtime fixtures then import `OmniServer` / `OmniRunner` internally, delaying heavy dependency initialization.
`tests/utils.py`	`hardware_test` / `hardware_marks` forwarded to `tests.helpers.mark`
Subpackage `conftest`	Local `conftest` files retained as needed in `tests/e2e/accuracy`, `tests/examples`, etc., separating responsibilities from the root directory.
Numerous tests and scripts	Unified import adjustments (approx. 106 files) to align calling paths with the above structure.
CI documentation	Examples and instructions under `docs/contributing/ci/` updated to reference `tests.helpers.mark`, etc., consistent with the implementation.
`assertions` / `media`	Enhanced error handling and logging (corresponding commit: Enhance error handling and logging in assertion and media helper functions).
Pre-commit	`tools/pre_commit/check_pickle_imports.py` and others aligned with the new module layout.

tests/
├── conftest.py                    # Thin entry: pytest_plugins + backward-compat re-exports + lazy runtime imports
├── helpers/                       # Shared importable helpers package (not tests/helpers.py at repo root)
│   ├── __init__.py
│   ├── assertions.py              # assert_* helpers (split from legacy monolithic conftest)
│   ├── env.py                     # env vars, GPU cleanup, device helpers
│   ├── mark.py                    # hardware_test / hardware_marks (replaces deleted tests/utils.py)
│   ├── media.py
│   ├── process.py
│   ├── runtime.py                 # OmniRunner / OmniServer / clients (heavy imports)
│   ├── stage_config.py
│   └── fixtures/
│       ├── __init__.py
│       ├── env.py                 # default_env, GPU cleanup autouse, session fixtures
│       ├── log.py
│       ├── run_args.py
│       └── runtime.py             # omni_server, etc.; imports runtime inside fixtures to preserve init order
├── e2e/
│   └── accuracy/
│       ├── conftest.py            # Local pytest hooks / fixtures for accuracy tests
│       └── helpers.py             # Helpers paired with this package’s conftest
├── examples/
│   ├── conftest.py
│   └── helpers.py                 # Helpers paired with example tests
├── dfx/
│   ├── helpers.py                 # Shared helpers for dfx scripts / stability & perf
│   ├── stability/
│   │   └── conftest.py            # Local conftest; uses helpers from ../helpers.py
│   └── perf/
│       └── scripts/               # Benchmark scripts (no conftest here)
├── comfyui/
│   └── conftest.py                # Local conftest only (no comfyui/helpers.py in tree)
├── diffusion/
│   ├── …
│   └── lora/
│       └── helpers.py             # LoRA test utilities (no lora/conftest.py)
├── engine/
├── model_executor/
└── …                              # Test modules; imports updated from legacy conftest/tests.utils to tests.helpers.*

Impact on Contributors (Migration Notes)
New code: Prefer from tests.helpers.mark import hardware_test, hardware_marks; use tests.helpers.env for GPU/environment-related utilities. The re-exports in tests.conftest are only for transitional compatibility.
OmniRunner / OmniServer, etc.: Can still be lazily loaded from tests.conftest, but long-term recommendation is to switch to from tests.helpers.runtime import ..., aligning with the direction noted in the PR.

Test Plan

test ready CI in local
test merge CI in local
run in ci

Test Result

1.ready in local

Job name	result
Simple Unit Test	Passed
Voxtral TTS CUDA Unit Test	Passed
Diffusion Model Test	Passed
Diffusion Batching Test	Passed
Custom Pipeline Test	Passed
Diffusion Model CPU offloading Test	Passed
Audio Generation Model Test	Passed
Diffusion Cache Backend Test	Passed
Diffusion Sequence Parallelism Test	Passed
Diffusion GPU Worker Test	Passed
Engine Test	Passed
Omni Model Test	Passed
Omni Model Test with H100	Passed
MiMo-Audio E2E Test with H100	Passed
Qwen3-TTS E2E Test	Passed
OmniVoice E2E Test	Passed
Voxtral-TTS E2E Test	Passed
Bagel Text2Img Model Test with H100	Passed
Bagel Img2Img Model Test with H100	Passed
Bagel Online Serving Test with H100	Passed
CosyVoice3-TTS E2E Test	Passed

2.merge in local

Job Name	result
Simple Unit Test	Passed
Diffusion Model Test	Passed
Diffusion Images API LoRA E2E	Passed
Diffusion Model CPU offloading Test	Passed
Audio Generation Model Test	Passed
Diffusion Cache Backend Test	Passed
Diffusion Sequence Parallelism Test	Passed
Diffusion Tensor Parallelism Test	Passed
Diffusion GPU Worker Test	Passed
Engine Test	Passed
Omni Model Test	Passed
Qwen3-TTS CustomVoice E2E Test	Passed
Qwen3-TTS Base E2E Test	Passed
Omni Model Test with H100	Passed
Diffusion Image Edit Test with H100 (1 GPU)	Passed
Bagel Model Test with H100 (Real Weights)	Passed
Voxtral-TTS E2E Test	Passed

success in merge

summary log

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: wangyu <410167048@qq.com>

…tions Signed-off-by: wangyu <410167048@qq.com>

chatgpt-codex-connector · 2026-04-09T02:57:20Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

Signed-off-by: wangyu <410167048@qq.com>

…, video, and image generation. Introduce parameters for cache directory and force regeneration, enhancing performance and usability. Remove deprecated save_to_file logic and improve error handling for media processing. Signed-off-by: wangyu <410167048@qq.com>

hsliuustc0106 · 2026-04-10T11:54:58Z

cc @Gaohan123 @tzhouam @princepride @lishunyang12 @linyueqian @ZeldaHuang @wtomin @SamitHuang PTAL for your tests ownership files

lishunyang12 · 2026-04-11T17:32:57Z

+import pytest
+import torch
+
+from tests.helpers.env import _run_post_test_cleanup, _run_pre_test_cleanup


tests.helpers.env imports vllm.platforms and vllm_omni.platforms at module level, so loading this plugin at conftest time still pulls them in before default_env runs. Tbh this partially defeats the RFC #2299 goal — consider moving these imports inside clean_gpu_memory_between_tests so helpers.env only loads after session fixtures.

lishunyang12 · 2026-04-11T17:32:57Z

+    print("=" * 80)
+
+
+def _run_pre_test_cleanup(enable_force: bool = False) -> None:


These are in __all__ and imported from fixtures/env.py and helpers/runtime.py, so they're effectively public. Drop the leading underscore?

…orts of platform-specific modules until needed to ensure proper execution order of fixtures. Introduce a new function for forced GPU cleanup to streamline cleanup processes across different classes. Enhance memory monitoring logic for better clarity and performance during tests. Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

…ate test markers for better categorization in zimage_parallelism tests. Enhance GPU memory cleanup messages for improved debugging during test execution. Signed-off-by: wangyu <410167048@qq.com>

…e count discrepancies in MP4 validation. Update docstring for clarity on expected behavior and adjust frame count assertion logic. Signed-off-by: wangyu <410167048@qq.com>

… helpers from the runtime module. This change improves code organization and maintainability by consolidating cleanup functions under a common namespace. Signed-off-by: wangyu <410167048@qq.com>

…hem to a new helpers module. This change improves code organization and reusability, while also adding functionality to compute and assert SSIM and PSNR metrics for model outputs. The previous utility functions have been removed from the utils module to streamline the codebase. Signed-off-by: wangyu <410167048@qq.com>

david6666666 · 2026-04-15T14:40:58Z

+        if params.use_omni and params.stage_init_timeout is not None:
+            server_args = [*server_args, "--stage-init-timeout", str(params.stage_init_timeout)]
+        else:
+            server_args = [*server_args, "--stage-init-timeout", "600"]


This changes the non-omni path too: when use_omni=False, vllm_omni.entrypoints.cli.main forwards straight to upstream vllm_main() unless --omni is present, so --stage-init-timeout / --init-timeout are not recognized there. We still have use_omni=False coverage in tests/e2e/accuracy/conftest.py, so this will make those server launches fail. Can we keep these timeout args gated behind params.use_omni, like the old fixture did?

…ompatibility with non-omni paths. Timeout flags are now gated behind the use_omni parameter, aligning with legacy behavior and improving code clarity. Signed-off-by: wangyu <410167048@qq.com>

…ding Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

…ameter mapping functions Signed-off-by: wangyu <410167048@qq.com>

hsliuustc0106

Solid refactor overall. The modular split and media caching are clear improvements. A few issues to address.

hsliuustc0106 · 2026-04-16T11:01:47Z

+    # Marker for Buildkite log folding before pytest summary lines.
+    terminalreporter.write_sep("-", "Result Summary")
+
+


This eager import of tests.helpers.assertions pulls in transformers.pipeline at conftest load time (see assertions.py:12). The PR description says the root conftest should avoid "premature loading of heavy dependencies" — this defeats that goal. Either defer these re-exports via __getattr__ (like the runtime exports below), or move the transformers.pipeline import inside _load_gender_pipeline() in assertions.py.

hsliuustc0106 · 2026-04-16T11:01:47Z

+
+import numpy as np
+import soundfile as sf
+from PIL import Image


from transformers import pipeline at module level means any import of tests.helpers.assertions (including from tests.conftest) triggers a heavy transformers load. Move this into _load_gender_pipeline() — it's only used there and already guarded by a lazy singleton pattern.

hsliuustc0106 · 2026-04-16T11:01:47Z

    "H100: Tests that require H100 GPU",
    "L4: Tests that require L4 GPU",
+    "B60: Tests that require B60",
    "MI325: Tests that require MI325 GPU (AMD/ROCm)",


Duplicate B60 marker entry. Line 197 adds "B60: Tests that require B60" (no description) and line 199 already has "B60: Tests that require Intel Arc Pro B60 XPU". pytest may silently accept duplicates, but this is confusing. Remove the one added here.

hsliuustc0106 · 2026-04-16T11:01:47Z

+                    env_dict=params.env_dict,
+                    use_omni=params.use_omni,
+                )
+                if port


The with OmniServer(...) if port else OmniServer(...) pattern duplicates the entire constructor call just to optionally pass port. Simplify to:

kwargs = dict(model=model, server_args=server_args, env_dict=params.env_dict, use_omni=params.use_omni) if port: kwargs["port"] = port with OmniServer(**kwargs) as server:

…rmance Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

… advanced model Signed-off-by: wangyu <410167048@qq.com>

…conftest

Signed-off-by: wangyu <410167048@qq.com>

- Removed unused export `dummy_messages_from_mix_data` from `_STAGE_CONFIG_EXPORT_NAMES`. - Added `dummy_messages_from_mix_data` to the imports in multiple test files for consistency. - Adjusted the `_REPO_ROOT` path comment in `stage_config.py` for clarity. Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: wangyu <410167048@qq.com>

…dules (vllm-project#2620) Signed-off-by: wangyu <410167048@qq.com>

yenuo26 added 2 commits April 9, 2026 10:31

refactor conftest

b27df37

Signed-off-by: wangyu <410167048@qq.com>

Enhance error handling and logging in assertion and media helper func…

651c636

…tions Signed-off-by: wangyu <410167048@qq.com>

yenuo26 requested a review from hsliuustc0106 as a code owner April 9, 2026 02:57

yenuo26 changed the title ~~Conftest~~ [CI] Restructure vLLM-Omni Test Layout, Fixture Scope, and Support Modules Apr 9, 2026

yenuo26 and others added 2 commits April 9, 2026 11:01

Merge branch 'main' into conftest

bbfdd8a

Adapt to the latest version of the code.

b44ee2d

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 force-pushed the conftest branch from 55560e2 to b44ee2d Compare April 9, 2026 03:12

This was referenced Apr 9, 2026

[RFC]: Restructure vLLM-Omni Test Layout, Fixture Scope, and Support Modules #2299

Open

[RFC]: Restructure vLLM-Omni Test Layout, Fixture Scope, and Support Modules JiusiServe/vllm-omni#179

Closed

pjh4993 mentioned this pull request Apr 10, 2026

[CI/Build] Step A: Extract shared utilities from tests/conftest.py into tests/tools/ #2613

Closed

5 tasks

hsliuustc0106 requested review from Gaohan123, congw729, david6666666, linyueqian, lishunyang12, princepride, tzhouam and wtomin April 10, 2026 11:52

lishunyang12 reviewed Apr 11, 2026

View reviewed changes

yenuo26 added 2 commits April 13, 2026 19:34

Merge remote-tracking branch 'upstream/main' into conftest

c10d549

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 added the merge-test label to trigger buildkite merge test CI label Apr 13, 2026

Refactor test imports to use helpers for consistency and clarity. Upd…

3859a96

…ate test markers for better categorization in zimage_parallelism tests. Enhance GPU memory cleanup messages for improved debugging during test execution. Signed-off-by: wangyu <410167048@qq.com>

yenuo26 added ready label to trigger buildkite CI nightly-test label to trigger buildkite nightly test CI and removed merge-test label to trigger buildkite merge test CI ready label to trigger buildkite CI labels Apr 14, 2026

yenuo26 added 3 commits April 15, 2026 20:39

Enhance assert_video_valid function to accommodate codec-aligned fram…

bd6508d

…e count discrepancies in MP4 validation. Update docstring for clarity on expected behavior and adjust frame count assertion logic. Signed-off-by: wangyu <410167048@qq.com>

Refactor test imports in Qwen image edit and VoxCPM test files to use…

edf922b

… helpers from the runtime module. This change improves code organization and maintainability by consolidating cleanup functions under a common namespace. Signed-off-by: wangyu <410167048@qq.com>

david6666666 reviewed Apr 15, 2026

View reviewed changes

Refactor timeout argument handling in omni_server fixture to ensure c…

4a6c4e2

…ompatibility with non-omni paths. Timeout flags are now gated behind the use_omni parameter, aligning with legacy behavior and improving code clarity. Signed-off-by: wangyu <410167048@qq.com>

Gaohan123 added this to the v0.20.0 milestone Apr 16, 2026

yenuo26 added 2 commits April 16, 2026 16:17

Add pytest_terminal_summary hook to conftest.py for Buildkite log fol…

64e6f7f

…ding Signed-off-by: wangyu <410167048@qq.com>

Merge remote-tracking branch 'upstream/main' into conftest

e6a2c19

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 added omni-test label to trigger buildkite omni model test in nightly CI and removed nightly-test label to trigger buildkite nightly test CI labels Apr 16, 2026

Add conftest.py for DFX benchmarks with configuration loading and par…

8528717

…ameter mapping functions Signed-off-by: wangyu <410167048@qq.com>

hsliuustc0106 reviewed Apr 16, 2026

View reviewed changes

yenuo26 and others added 3 commits April 16, 2026 20:00

Refactor tests and configuration files for improved clarity and perfo…

52be8a1

…rmance Signed-off-by: wangyu <410167048@qq.com>

Merge branch 'main' into conftest

04c6a7e

Refactor import statement in e2e test for Flux2 Klein inpaint expansion

15bd4aa

Signed-off-by: wangyu <410167048@qq.com>

yenuo26 added ready label to trigger buildkite CI nightly-test label to trigger buildkite nightly test CI merge-test label to trigger buildkite merge test CI and removed omni-test label to trigger buildkite omni model test in nightly CI labels Apr 17, 2026

yenuo26 and others added 7 commits April 17, 2026 11:18

Fix import path in conftest.py for stability tests

cc7d706

Signed-off-by: wangyu <410167048@qq.com>

Merge branch 'main' into conftest

f9aecb2

Update pytest command in nightly test configuration to use marker for…

2bacffc

… advanced model Signed-off-by: wangyu <410167048@qq.com>

Merge branch 'conftest' of https://github.com/yenuo26/vllm-omni into …

27b9403

…conftest

Merge remote-tracking branch 'upstream/main' into conftest

5aa1b4b

Signed-off-by: wangyu <410167048@qq.com>

Refactor test imports and update helper module paths

d62b67a

Signed-off-by: wangyu <410167048@qq.com>

hsliuustc0106 merged commit 8a9add1 into vllm-project:main Apr 20, 2026
7 of 9 checks passed

Sy0307 mentioned this pull request Apr 20, 2026

[Bugfix] Sync main into dev/migrate-MR-v2 with semantic-safe conflict resolution #2954

Merged

10 tasks

yenuo26 deleted the conftest branch April 21, 2026 01:38

qinganrice pushed a commit to qinganrice/vllm-omni that referenced this pull request Apr 23, 2026

[CI] Restructure vLLM-Omni Test Layout, Fixture Scope, and Support Mo…

1b8dd96

…dules (vllm-project#2620) Signed-off-by: wangyu <410167048@qq.com>

		print("=" * 80)


		def _run_pre_test_cleanup(enable_force: bool = False) -> None:

		# Marker for Buildkite log folding before pytest summary lines.
		terminalreporter.write_sep("-", "Result Summary")

Conversation

yenuo26 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

chatgpt-codex-connector Bot commented Apr 9, 2026

Uh oh!

hsliuustc0106 commented Apr 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yenuo26 commented Apr 9, 2026 •

edited

Loading