[Test] add stability test case for wan2.2, qwen-tts, qwen3-omni and qwen-image model and modified conftest.py in test/dfx/

[zhumingjue138]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

add stability test case for wan2.2 model and modified conftest.py in test/dfx/

Test Plan

1、modified conftest.py in test/dfx/

2、Split the file "tests/dfx/stability/scripts/test_benchmark_stability.py" according to the model names and rename it as "tests/dfx/stability/scripts/test_stability_qwen3_omni.py" and "tests/dfx/stability/scripts/test_stability_wan22.py"

pytest -s -v tests/dfx/perf/scripts/run_benchmark.py
pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py
pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py
pytest -s -v tests/dfx/stability/scripts/test_stability_qwen_image.py
pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_tts.py

Test Result

pytest -s -v tests/dfx/perf/scripts/run_benchmark.py

pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_omni.py

[
    {
        "test_name": "test_qwen3_omni_stability_async_chunk",
        "server_params": {
            "model": "/home/models/Qwen/Qwen3-Omni-30B-A3B-Instruct",
            "stage_overrides": {
                "2": {
                    "max_num_batched_tokens": 1000000
                }
            },
            "extra_cli_args": ["--async-chunk"]
        },
        "benchmark_params": [
            {
                "dataset_name": "random-mm",
                "backend": "openai-chat-omni",
                "endpoint": "/v1/chat/completions",
                "duration_sec": 200,
                "request_rate": 0.3,
                "num_prompts_per_batch": 10,
                "random_input_len": {
                    "min": 0,
                    "max": 8000
                },
                "random_output_len": {
                    "min": 0,
                    "max": 1000
                },
                "random_range_ratio": 0.0,
                "random_mm_base_items_per_request": {
                    "min": 0,
                    "max": 6
                },
                "random_mm_num_mm_items_range_ratio": 0.0,
                "random_mm_limit_mm_per_prompt": {
                    "image": 2,
                    "video": 2,
                    "audio": 2
                },
                "random_mm_bucket_config": {
                    "(128-1024, 128-1024, 1)": 0.34,
                    "(256-1080, 256-1920, 2-16)": 0.33,
                    "(0, 1-60, 1-3)": 0.33
                },
                "ignore_eos": true,
                "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration"
            }
        ]
    }
]

pytest -s -v tests/dfx/stability/scripts/test_stability_qwen_image.py

[
    {
        "test_name": "test_qwen_image_stability",
        "server_params": {
            "model": "/nvme1n1p1/models/Qwen/Qwen-Image"
        },
        "benchmark_params": [
            {
                "dataset": "random",
                "task": "t2i",
                "backend": "vllm-omni",
                "duration_sec": 200,
                "max_concurrency": 1,
                "num_prompts_per_batch": 2,
                "width": {
                    "min": 512,
                    "max": 2048
                },
                "height": {
                    "min": 512,
                    "max": 2048
                },
                "num_inference_steps": 50,
                "enable_negative_prompt": true
            }
        ]
    }
]

pytest -s -v tests/dfx/stability/scripts/test_stability_qwen3_tts.py

[
    {
        "test_name": "test_qwen3_tts_stability",
        "server_params": {
            "model": "/nvme1n1p1/models/Qwen3-TTS-12Hz-1.7B-CustomVoice"
        },
        "benchmark_params": [
            {
                "dataset_name": "random",
                "backend": "openai-audio-speech",
                "endpoint": "/v1/audio/speech",
                "duration_sec": 120,
                "request_rate": 0.3,
                "num_prompts_per_batch": 3,
                "random_input_len": {
                    "min": 0,
                    "max": 1000
                },
                "random_output_len": {
                    "min": 0,
                    "max": 1000
                },
                "random_range_ratio": 0.0,
                "extra_body": {
                    "voice": "Vivian",
                    "language": "English"
                },
                "ignore_eos": true,
                "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration"
            }
        ]
    }
]

pytest -s -v tests/dfx/stability/scripts/test_stability_wan22.py

[
    {
        "test_name": "test_wan22_t2v_stability_v1_videos",
        "server_params": {
            "model": "/home/models/Wan-AI/Wan2.2-T2V-A14B-Diffusers",
            "serve_args": {
                "ulysses-degree": 1,
                "vae-patch-parallel-size": 2,
                "cfg-parallel-size": 1,
                "tensor-parallel-size": 1,
                "use-hsdp": true,
                "hsdp_shard_size": 2,
                "hsdp_replicate_size": 1,
                "vae-use-slicing": true,
                "vae-use-tiling": true
            }
        },
        "benchmark_params": [
            {
                "dataset": "random",
                "task": "t2v",
                "backend": "v1/videos",
                "duration_sec": 300,
                "max_concurrency": 1,
                "num_prompts_per_batch": 3,
                "enable_negative_prompt": true,
                "random_request_config": [
                    {"width": 854, "height": 480, "num_inference_steps": 3, "num_frames": 80, "fps": 16, "weight": 0.65},
                    {"width": 854, "height": 480, "num_inference_steps": 4, "num_frames": 120, "fps": 24, "weight": 0.25},
                    {"width": 1280, "height": 720, "num_inference_steps": 6, "num_frames": 80, "fps": 16, "weight": 0.1}
                ]
            }
        ]
    }
]

================= Serving Benchmark Result =================
Backend:                                 v1/videos
Model:                                   /home/models/Wan-AI/Wan2.2-T2V-A14B-Diffusers
Dataset:                                 random
Task:                                    t2v
--------------------------------------------------
Benchmark duration (s):                  170.36
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     3/3
--------------------------------------------------
Request throughput (req/s):              0.02
Latency Mean (s):                        56.7875
Latency Median (s):                      56.1269
Latency P99 (s):                         58.0920
Latency P95 (s):                         57.9316
--------------------------------------------------
Peak Memory Max (MB):                    51132.00
Peak Memory Mean (MB):                   50554.00
Peak Memory Median (MB):                 51040.00

============================================================
Metrics saved to /tmp/stability_diffusion_q30pi7rf.json
================= Serving Benchmark Result =================
Backend:                                 v1/videos
Model:                                   /home/models/Wan-AI/Wan2.2-T2V-A14B-Diffusers
Dataset:                                 random
Task:                                    t2v
--------------------------------------------------
Benchmark duration (s):                  173.02
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     3/3
--------------------------------------------------
Request throughput (req/s):              0.02
Latency Mean (s):                        57.6723
Latency Median (s):                      58.1305
Latency P99 (s):                         58.2935
Latency P95 (s):                         58.2802
--------------------------------------------------
Peak Memory Max (MB):                    51186.00
Peak Memory Mean (MB):                   50572.00
Peak Memory Median (MB):                 51040.00

============================================================
Metrics saved to /tmp/stability_diffusion_wj55pjs7.json
============ Stability Benchmark Summary ============
Successful requests:                     6
Failed requests:                         0
Total duration (s):                      352.24
==================================================

nightly CI

24h test:

============ Stability Benchmark Summary ============
Successful requests:                     1752      
Failed requests:                         68        
Total duration (s):                      93786.44  
==================================================

related issue: #2928

12h test:

[
    {
        "test_name": "test_wan22_t2v_stability_v1_videos",
        "server_params": {
            "model": "/nvme1n1p1/models/Wan2.2-I2V-A14B-Diffusers/snapshots/596658fd9ca6b7b71d5057529bbf319ecbc61d74",
            "serve_args": {
                "ulysses-degree": 2,
                "vae-patch-parallel-size": 2,
                "tensor-parallel-size": 1,
                "use-hsdp": true,
                "vae-use-slicing": true,
                "vae-use-tiling": true
            }
        },
        "benchmark_params": [
            {
                "dataset": "random",
                "task": "i2v",
                "backend": "v1/videos",
                "duration_sec": 43200,
                "max_concurrency": 1,
                "num_prompts_per_batch": 10,
                "enable_negative_prompt": true,
                "random_request_config": [
                    {"width": 832, "height": 480, "num_inference_steps": 2, "num_frames": 81, "fps": 16, "weight": 0.5},
                    {"width": 1280, "height": 720, "num_inference_steps": 2, "num_frames": 121, "fps": 16, "weight": 0.5}
                ]
            }
        ]
    }
]

result
============ Stability Benchmark Summary ============
Successful requests:                     490
Failed requests:                         0
Total duration (s):                      43387.18
==================================================

another 12h test

============ Stability Benchmark Summary ============
Successful requests:                     440
Failed requests:                         0
Total duration (s):                      40191.27
==================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

The test_wan22.json config only runs 3 prompts (num_prompts_per_batch=3) over 300 seconds with max_concurrency=1. Is this sufficient for a stability test? Consider increasing num_prompts_per_batch to get better coverage.

[yenuo26]

i think these can be moved to helpers.py

[zhumingjue138]

done

[zhumingjue138]

@Gaohan123 @hsliuustc0106 this pr is ready, can it be merged?