Skip to content

[CI/Build] Add per-model reliability cases #9109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amy-why-3459 merged 8 commits into from
Oct 20, 2026
Merged

[CI/Build] Add per-model reliability cases #9109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f3ce132
Refactor reliability test file names and update dependencies
yenuo26 Oct 18, 2026
0928c04
Update reliability test file paths and improve type checking
yenuo26 Oct 19, 2026
1b1a61a
Refactor device parsing and enhance reliability test assertions
yenuo26 Oct 19, 2026
abd0be4
Update reliability test documentation for ProcessKillWorkerNoLoad sce…
yenuo26 Oct 20, 2026
08fbdc9
Enhance reliability test configurations and refactor helper functions
yenuo26 Oct 20, 2026
f646cf8
Refactor OmniServer parameters and remove logging functionality
yenuo26 Oct 20, 2026
95b9fc2
Merge branch 'main' into reliability_cases
yenuo26 Oct 20, 2026
113c99e
Merge branch 'main' into reliability_cases
yenuo26 Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .buildkite/amd/test-amd-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ steps:
source_file_dependencies: omni_qwen3_omni_reliability
commands:
- >-
pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py
pytest -s -v tests/dfx/reliability/test_qwen3_omni.py
-m "slow and rocm and MI325 and cards_2"

- label: "Reliability Test · wan22"
Expand All @@ -32,7 +32,7 @@ steps:
source_file_dependencies: diffusion_wan22_reliability
commands:
- >-
pytest -s -v tests/dfx/reliability/test_reliability_wan22.py
pytest -s -v tests/dfx/reliability/test_wan22.py
-m "slow and rocm and MI325 and cards_1"

- label: "Reliability Test · hunyuan-image"
Expand All @@ -44,7 +44,7 @@ steps:
source_file_dependencies: diffusion_hunyuan_image3_reliability
commands:
- >-
pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py
pytest -s -v tests/dfx/reliability/test_hunyuan_image.py
-m "slow and rocm and MI325 and cards_4"

- label: "Reliability Test · Invalid parameters · MI300 · 2-GPU"
Expand Down
30 changes: 27 additions & 3 deletions .buildkite/common/ci_source_file_dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,12 @@ source_file_dependencies:

omni_qwen3_omni_reliability:
- *qwen3_omni
- tests/dfx/reliability/test_reliability_qwen3_omni.py
- tests/dfx/reliability/test_qwen3_omni.py
- tests/dfx/reliability/scenarios/
- tests/dfx/reliability/helpers.py
- tests/dfx/conftest.py
- tests/dfx/reliability/conftest.py
- tests/helpers/runtime.py

@amy-why-3459 amy-why-3459 Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scenarios now depend on tests/dfx/conftest.py (assert_fault_exception, resolve_oom_device_spec, parse_stage_devices, create_reliability_omni_server_params), and this PR changes parse_stage_devices there. That file isn't in any of the four *_reliability keys or in source_filter_fallback (it's only under omni_qwen3_omni_duplex_perf), so a change limited to it won't select these jobs. Suggest adding tests/dfx/conftest.py next to tests/dfx/reliability/helpers.py in all four keys, including the new omni_minicpmo_4_5_reliability.

@yenuo26 yenuo26 Oct 20, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added tests/dfx/conftest.py next to tests/dfx/reliability/helpers.py on all four keys: omni_qwen3_omni_reliability, omni_minicpmo_4_5_reliability, diffusion_wan22_reliability, and diffusion_hunyuan_image3_reliability.

On 08fbdc959.


omni_qwen3_omni_doc:
- *qwen3_omni
Expand All @@ -337,6 +342,15 @@ source_file_dependencies:
- tests/e2e/online_serving/test_minicpmo_4_5.py
- tests/e2e/online_serving/test_minicpmo_4_5_expansion.py

omni_minicpmo_4_5_reliability:
- *minicpmo_4_5
- tests/dfx/reliability/test_minicpmo_4_5.py
- tests/dfx/reliability/scenarios/
- tests/dfx/reliability/helpers.py
- tests/dfx/conftest.py
- tests/dfx/reliability/conftest.py
- tests/helpers/runtime.py

omni_minicpmo_4_5_accuracy:
- *minicpmo_4_5
- tests/e2e/accuracy/minicpmo_4_5/
Expand Down Expand Up @@ -469,7 +483,12 @@ source_file_dependencies:

diffusion_wan22_reliability:
- *wan22
- tests/dfx/reliability/test_reliability_wan22.py
- tests/dfx/reliability/test_wan22.py
- tests/dfx/reliability/scenarios/
- tests/dfx/reliability/helpers.py
- tests/dfx/conftest.py
- tests/dfx/reliability/conftest.py
- tests/helpers/runtime.py

diffusion_image_to_video_doc:
- *wan22
Expand Down Expand Up @@ -560,7 +579,12 @@ source_file_dependencies:

diffusion_hunyuan_image3_reliability:
- *hunyuan_image3
- tests/dfx/reliability/test_reliability_hunyuan_image.py
- tests/dfx/reliability/test_hunyuan_image.py
- tests/dfx/reliability/scenarios/
- tests/dfx/reliability/helpers.py
- tests/dfx/conftest.py
- tests/dfx/reliability/conftest.py
- tests/helpers/runtime.py

diffusion_minimax_h3_function:
- *minimax_h3
Expand Down
13 changes: 10 additions & 3 deletions .buildkite/cuda/test-weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,23 +33,30 @@ steps:
source_file_dependencies: omni_qwen3_omni_reliability
timeout_in_minutes: 180
commands:
- pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m "slow"
- pytest -s -v tests/dfx/reliability/test_qwen3_omni.py -m "slow and cards_2"
mirror_hardwares: h100_2

- label: "Reliability Test · wan22"
source_file_dependencies: diffusion_wan22_reliability
timeout_in_minutes: 180
commands:
- pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m "slow"
- pytest -s -v tests/dfx/reliability/test_wan22.py -m "slow and cards_1"
mirror_hardwares: h100_1

- label: "Reliability Test · hunyuan-image"
source_file_dependencies: diffusion_hunyuan_image3_reliability
timeout_in_minutes: 180
commands:
- pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m "slow"
- pytest -s -v tests/dfx/reliability/test_hunyuan_image.py -m "slow and cards_4"
mirror_hardwares: h100_4

- label: "Reliability Test · minicpmo-4.5"
source_file_dependencies: omni_minicpmo_4_5_reliability
timeout_in_minutes: 180
commands:
- pytest -s -v tests/dfx/reliability/test_minicpmo_4_5.py -m "slow and cards_1"
mirror_hardwares: h100_1

- label: "Reliability Test · Invalid parameters · H100 · Single-GPU"
source_file_dependencies: reliability_invalid_param_h100
timeout_in_minutes: 90
Expand Down
9 changes: 6 additions & 3 deletions docs/contributing/ci/test_execution_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,16 @@ Failed jobs: 1/2
pytest -s -v dfx/stability/scripts/run_stability_wan22.py -m slow

# Reliability: Qwen3-Omni (H100 × 2)
pytest -s -v dfx/reliability/test_reliability_qwen3_omni.py -m slow
pytest -s -v dfx/reliability/test_qwen3_omni.py -m "slow and cards_2"

# Reliability: Wan2.2 (H100)
pytest -s -v dfx/reliability/test_reliability_wan22.py -m slow
pytest -s -v dfx/reliability/test_wan22.py -m "slow and cards_1"

# Reliability: HunyuanImage DiT (H100 × 4)
pytest -s -v dfx/reliability/test_reliability_hunyuan_image.py -m slow
pytest -s -v dfx/reliability/test_hunyuan_image.py -m "slow and cards_4"

# Reliability: MiniCPM-o 4.5 (H100). SIGTERM serve-root only, plus partial isolation.
pytest -s -v dfx/reliability/test_minicpmo_4_5.py -m "slow and cards_1"

```

Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/ci/test_system_overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ Through five levels (L1-L5) and common (Common) specifications, the system clari
<strong>Stability:</strong><br>
<code>tests/dfx/stability/tests/</code><br>
<strong>Reliability:</strong><br>
<code>tests/dfx/reliability/test_reliability_{model}.py</code>
<code>tests/dfx/reliability/test_{model}.py</code>
</td>
<td>
<a href="test_writing_guide.md#l5-level-testing-stability-and-reliability-testing">L5</a><br>
Expand Down
23 changes: 11 additions & 12 deletions docs/contributing/ci/test_writing_guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -575,10 +575,9 @@ L5 level testing focuses on the performance of model services under ***long-runn

- ***Stability Test Configuration***: `tests/dfx/stability/tests/test_qwen3_omni.json`, `tests/dfx/stability/tests/test_wan22.json` (one JSON per model / runner family)
- ***Reliability Test Suite*** (`tests/dfx/reliability/`):
- `test_reliability_qwen3_omni.py` — Qwen3-Omni chat / multimodal reliability (GPU OOM, process kill, recovery, error contract under `--async-chunk` vs default).
- `test_reliability_wan22.py` — Wan2.2 I2V video API reliability (`/v1/videos` under OOM and process kill, recovery).
- `test_reliability_hunyuan_image.py` — HunyuanImage-3.0-Instruct DiT-only reliability (`/v1/images/generations` under OOM and process kill; deploy `hunyuan_image3_dit.yaml`, H100 × 4).
- `helpers.py` — Shared primitives used by current suites: raw HTTP probes for `/v1/chat/completions` and `/health`, OpenAI-style error parsing, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
- `scenarios/` — one class per fault. The class owns the shared steps.
- `helpers.py` — `FaultModel`, signal lists, `scenario_params`, plus the shared primitives: raw HTTP probes, GPU OOM sidecar (`inject_gpu_oom` / `stop_gpu_oom_hogs`), and `pgrep`-based process-kill injector construction (`make_process_kill_fault_injector`).
- `test_<model>.py` — that model's deploy, probes, and the scenarios it runs. Each test calls the matching scenario class.
- `conftest.py` — `fault_injector` and `omni_server_after_fault` / `omni_server_after_fault_function` fixtures to run a callable **after** the server is ready.
- `README.md` — Short local run commands for this directory.

Expand All @@ -587,7 +586,7 @@ L5 level testing focuses on the performance of model services under ***long-runn
- ***Trigger Timing***: **`Weekly`** (weekly) or **`Days before Release`** (several days before a major release). Due to long execution times, the frequency is lower.
- ***Run Command***:
- ***Stability***: `pytest -s -v tests/dfx/stability/scripts/run_stability_qwen3_omni.py` or `pytest -s -v tests/dfx/stability/scripts/run_stability_wan22.py` (or add `run_stability_<model>.py` alongside a matching JSON config)
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_reliability_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`).
- ***Reliability***: `pytest -s -v tests/dfx/reliability/test_<model>.py -m slow` (current suites: `qwen3_omni`, `wan22`, `hunyuan_image`, `minicpmo_4_5`).
- ***Script Example***:

<details>
Expand Down Expand Up @@ -663,19 +662,19 @@ All other optional parameters follow the same rules as in L4 §3.4.

##### Reliability

Add pytest modules under `tests/dfx/reliability/` (for example `test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`). Reuse `helpers.py` and `conftest.py` rather than duplicating OOM / process-kill / raw HTTP plumbing.
Add `tests/dfx/reliability/test_<model>.py` with a `FaultModel` subclass for that model's deploy and probes, and call the scenario classes for the faults it runs. Reuse `scenarios/`, `helpers.py`, and `conftest.py` rather than duplicating OOM / process-kill / raw HTTP plumbing.

###### Parametrization and markers

- Each test module defines a **`RELIABILITY_SCENARIOS`** list (`test_name`, `server_params`: model, `stage_config_name` or diffusion `server_args`, etc.). **`create_reliability_omni_server_params()`** in `tests/dfx/conftest.py` resolves stage paths and builds **`OmniServerParams`** lists consumed by **`@pytest.mark.parametrize(..., indirect=True)`** on `omni_server` or `omni_server_function`.
- Cases are tagged **`@pytest.mark.slow`** for weekly / selective CI. GPU-heavy suites use **`@hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=...)`** (Qwen3-Omni **2** cards; Wan2.2 **1** card; HunyuanImage DiT **4** cards).
- The model's `scenarios` list (`test_name`, `server_params`) is turned into `OmniServerParams` by `create_reliability_omni_server_params()`. `scenario_params()` parametrizes `omni_server_function`.
- The model test module is tagged `pytest.mark.slow` and `hardware_marks(...)` (Qwen3-Omni **2** cards; Wan2.2 **1** card; HunyuanImage DiT **4** cards; MiniCPM-o 4.5 **1** card).
- **Process-kill** tests use POSIX **`pgrep` / `kill`**. The helper raises on Windows instead of skipping.

###### Adding a new model suite

1. Add `test_reliability_<model>.py` under `tests/dfx/reliability/`.
2. Define **`RELIABILITY_SCENARIOS`** and pass them through **`create_reliability_omni_server_params()`** with the correct deploy or e2e deploy-config directory (same pattern as existing files).
3. Reuse **`helpers`** for OOM / kill / raw HTTP; prefer **`assert_fault_exception()`** and **`resolve_oom_device_spec()`** from `tests/dfx/conftest.py` for consistent device selection vs deploy YAML.
4. Register **`slow`** (and **`hardware_test`** if needed); extend **`.buildkite/cuda/test-weekly.yml`** when the suite should run in weekly L5.
1. Add `test_<model>.py` under `tests/dfx/reliability/`.
2. Subclass `FaultModel` in that file with the deploy and probes. Call `scenario_params` and the scenario class `run` method for each fault this model runs.
3. Reuse **`helpers`** and **`scenarios`** for OOM / kill / raw HTTP; prefer **`assert_fault_exception()`** and **`resolve_oom_device_spec()`** from `tests/dfx/conftest.py` for consistent device selection vs deploy YAML.
4. Register **`slow`** and **`hardware_marks`** on the module; extend **`.buildkite/cuda/test-weekly.yml`** when the suite should run in weekly L5.

</details>
2 changes: 1 addition & 1 deletion docs/user_guide/fault_injection_reliability_matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ More failure modes (for example, network jitter and network interruption) will b

## Source of Conclusions

The behaviors and conclusions above are summarized from current fault injection validation results on **`Qwen3-Omni`**, **`Wan2.2`**, and **`HunyuanImage-3.0-Instruct`** (DiT-only, `/v1/images/generations`). Automated coverage lives under `tests/dfx/reliability/` (`test_reliability_qwen3_omni.py`, `test_reliability_wan22.py`, `test_reliability_hunyuan_image.py`) and runs weekly via `.buildkite/cuda/test-weekly.yml`.
The behaviors and conclusions above are summarized from current fault injection validation results on **`Qwen3-Omni`**, **`Wan2.2`**, and **`HunyuanImage-3.0-Instruct`** (DiT-only, `/v1/images/generations`). Automated coverage lives under `tests/dfx/reliability/` (`scenarios/`, `helpers.py`, and one `test_<model>.py` per model) and runs weekly via `.buildkite/cuda/test-weekly.yml`.

### Example: `SIGTERM` fault injection log (Qwen3-Omni)

Expand Down
6 changes: 3 additions & 3 deletions tests/buildkite/test_amd_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -896,9 +896,9 @@ def test_weekly_reliability_is_distinct_from_nightly() -> None:
nightly = AMD_NIGHTLY_PIPELINE.read_text(encoding="utf-8")
group = weekly["steps"][0]
jobs = {
"Reliability Test · qwen3-omni": ("mi300_2", "cards_2", "test_reliability_qwen3_omni.py"),
"Reliability Test · wan22": ("mi300_1", "cards_1", "test_reliability_wan22.py"),
"Reliability Test · hunyuan-image": ("mi300_4", "cards_4", "test_reliability_hunyuan_image.py"),
"Reliability Test · qwen3-omni": ("mi300_2", "cards_2", "tests/dfx/reliability/test_qwen3_omni.py"),
"Reliability Test · wan22": ("mi300_1", "cards_1", "tests/dfx/reliability/test_wan22.py"),
"Reliability Test · hunyuan-image": ("mi300_4", "cards_4", "tests/dfx/reliability/test_hunyuan_image.py"),
"Reliability Test · Invalid parameters · MI300 · 2-GPU": (
"mi300_2",
"cards_2",
Expand Down
Loading
Loading