[Test] Add Qwen-tts test cases and unify the style of existing test cases

[yenuo26]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

1.Add new Qwen3-TTS E2E test cases: covering online /v1/audio/speech and offline inference, including streaming, concurrency, and various stage configurations (such as disabling cuda_graph).

2.Unify Qwen-TTS test style: online tests use OpenAIClientHandler.send_audio_speech_request, offline tests use OmniRunnerHandler.send_audio_speech_request, with input parameters fully aligned with end2end.py's additional_information + _estimate_prompt_len.

Test Plan

1.pytest -sv tests/e2e/offline_inference/test_qwen3_tts_customvoice.py
2.pytest -sv tests/e2e/online_serving/test_qwen3_tts_customvoice_expansion.py --run-level "advanced_model"
3.pytest -sv tests/e2e/online_serving/test_qwen3_tts_customvoice.py --run-level "advanced_model"
4.pytest -sv tests/e2e/online_serving/test_qwen3_tts_base.py --run-level "advanced_model"
5.pytest -sv tests/e2e/online_serving/test_qwen3_tts_base_expansion.py --run-level "advanced_model"

Test Result

============================================================== 1 passed, 18 warnings in 195.98s (0:03:15) ===============================================================

============================================================== 6 passed, 16 warnings in 827.43s (0:13:47) ===============================================================

============================================================== 2 passed, 17 warnings in 638.21s (0:10:38) ===============================================================

============================================================== 2 passed, 18 warnings in 281.52s (0:04:41) ===============================================================

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b874b652da

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[yenuo26]

@R2-Y please check it.
1.Are there any additional test scenarios that need to be supplemented?
2.Can it cover the current qwen-tts test cases? Do tests/e2e/online_serving/qwen_tts_base.py and tests/e2e/online_serving/test_qwen3_tts_websocket.py need to be retained, and should the related scenarios be covered in the E2E tests? I feel that some scenarios in them might be sufficiently covered by UT (Unit Tests).

[R2-Y]

@linyueqian PTAL, I think test customVoice is enough for our UT, do you think we still need test for base?

[linyueqian]

The tests only cover CustomVoice. There's no coverage for the Base model (voice cloning with ref_audio/ref_text). We've had several Base-specific bugs recently (#1731, #1774, #1754) that wouldn't be caught by CustomVoice-only tests. The Base model has a different code path (ICL prompt construction, ref_code context prepending in first decoder chunk, speaker encoder). Would be good to add at least one Base/voice-clone e2e test case.

[linyueqian]

Here are some Base model (voice cloning) test cases that would be good to add. These cover the ICL code path that has had several recent bugs (#1731, #1774, #1754).

# Add to test_qwen3_tts_expansion.py

BASE_MODEL = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
REF_AUDIO_URL = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_2.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

tts_base_server_params = [
    pytest.param(
        OmniServerParams(
            model=BASE_MODEL,
            stage_config_path=get_stage_config("qwen3_tts.yaml"),
            server_args=["--trust-remote-code", "--disable-log-stats"],
        ),
        id="base_async_chunk",
    ),
]


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_001(omni_server, openai_client) -> None:
    """
    Test voice cloning via Base model with ref_audio and ref_text.
    Exercises ICL prompt construction and ref_code context prepending
    in the first decoder chunk (regression coverage for #1731, #1774).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_002_streaming(omni_server, openai_client) -> None:
    """
    Test streaming voice cloning. The async_chunk path handles ref_code
    differently on the first chunk (put_req_chunk == 0 check in
    talker2code2wav_async_chunk).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": True,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_003_concurrent(omni_server, openai_client) -> None:
    """
    Test concurrent voice cloning. The ref_code prepend logic has
    potential race conditions under concurrency (issue #1861).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config, request_num=get_max_batch_size("few"))

Note: the Whisper similarity threshold (0.9) in assert_audio_speech_response might need to be more lenient for voice cloned audio since cloned voices sometimes have lower transcription accuracy.

[yenuo26]

Here are some Base model (voice cloning) test cases that would be good to add. These cover the ICL code path that has had several recent bugs (#1731, #1774, #1754).

# Add to test_qwen3_tts_expansion.py

BASE_MODEL = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
REF_AUDIO_URL = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_2.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

tts_base_server_params = [
    pytest.param(
        OmniServerParams(
            model=BASE_MODEL,
            stage_config_path=get_stage_config("qwen3_tts.yaml"),
            server_args=["--trust-remote-code", "--disable-log-stats"],
        ),
        id="base_async_chunk",
    ),
]


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_001(omni_server, openai_client) -> None:
    """
    Test voice cloning via Base model with ref_audio and ref_text.
    Exercises ICL prompt construction and ref_code context prepending
    in the first decoder chunk (regression coverage for #1731, #1774).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_002_streaming(omni_server, openai_client) -> None:
    """
    Test streaming voice cloning. The async_chunk path handles ref_code
    differently on the first chunk (put_req_chunk == 0 check in
    talker2code2wav_async_chunk).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": True,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_003_concurrent(omni_server, openai_client) -> None:
    """
    Test concurrent voice cloning. The ref_code prepend logic has
    potential race conditions under concurrency (issue #1861).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config, request_num=get_max_batch_size("few"))

Note: the Whisper similarity threshold (0.9) in assert_audio_speech_response might need to be more lenient for voice cloned audio since cloned voices sometimes have lower transcription accuracy.

Here are some Base model (voice cloning) test cases that would be good to add. These cover the ICL code path that has had several recent bugs (#1731, #1774, #1754).

# Add to test_qwen3_tts_expansion.py

BASE_MODEL = "Qwen/Qwen3-TTS-12Hz-1.7B-Base"
REF_AUDIO_URL = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_2.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

tts_base_server_params = [
    pytest.param(
        OmniServerParams(
            model=BASE_MODEL,
            stage_config_path=get_stage_config("qwen3_tts.yaml"),
            server_args=["--trust-remote-code", "--disable-log-stats"],
        ),
        id="base_async_chunk",
    ),
]


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_001(omni_server, openai_client) -> None:
    """
    Test voice cloning via Base model with ref_audio and ref_text.
    Exercises ICL prompt construction and ref_code context prepending
    in the first decoder chunk (regression coverage for #1731, #1774).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_002_streaming(omni_server, openai_client) -> None:
    """
    Test streaming voice cloning. The async_chunk path handles ref_code
    differently on the first chunk (put_req_chunk == 0 check in
    talker2code2wav_async_chunk).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": True,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config)


@pytest.mark.advanced_model
@pytest.mark.omni
@hardware_test(res={"cuda": "L4"}, num_cards=1)
@pytest.mark.parametrize("omni_server", tts_base_server_params, indirect=True)
def test_voice_clone_003_concurrent(omni_server, openai_client) -> None:
    """
    Test concurrent voice cloning. The ref_code prepend logic has
    potential race conditions under concurrency (issue #1861).
    """
    request_config = {
        "model": omni_server.model,
        "input": "The weather is nice today, perfect for a walk in the park.",
        "stream": False,
        "response_format": "wav",
        "task_type": "Base",
        "voice": "clone",
        "ref_audio": REF_AUDIO_URL,
        "ref_text": REF_TEXT,
    }
    openai_client.send_audio_speech_request(request_config, request_num=get_max_batch_size("few"))

Note: the Whisper similarity threshold (0.9) in assert_audio_speech_response might need to be more lenient for voice cloned audio since cloned voices sometimes have lower transcription accuracy.

I'm adding some base use cases, and then I found that when I input with these parameters, the returned audio content is only "It's nice today". Does this indicate any issues?

REF_AUDIO_URL = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_2.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

request_config = {
"input": "The weather is nice today, perfect for a walk in the park.",
"task_type": "Base",
"voice": "clone",
"ref_audio": REF_AUDIO_URL,
"ref_text": REF_TEXT,
}

[linyueqian]

Nice work! Could we also add Fish Speech S2 Pro tests? There are currently none in the repo. The existing example at examples/offline_inference/fish_speech/end2end.py and stage config can serve as references.

[linyueqian]

Could you test again on latest main? #1945 (merged Mar 17) fixed several Base voice clone issues including streaming quality and early stop-token handling.

the returned audio content is only "It's nice today"

This sounds like it could be the early stop-token issue that #1945 addressed.

[yenuo26]

Could you test again on latest main? #1945 (merged Mar 17) fixed several Base voice clone issues including streaming quality and early stop-token handling.

the returned audio content is only "It's nice today"

This sounds like it could be the early stop-token issue that #1945 addressed.

Could you test again on latest main? #1945 (merged Mar 17) fixed several Base voice clone issues including streaming quality and early stop-token handling.

the returned audio content is only "It's nice today"

This sounds like it could be the early stop-token issue that #1945 addressed.

After I updated the code, this issue was resolved.

[yenuo26]

@Gaohan123 @princepride please help to add a ready label

[yenuo26]

Nice work! Could we also add Fish Speech S2 Pro tests? There are currently none in the repo. The existing example at examples/offline_inference/fish_speech/end2end.py and stage config can serve as references.

Perhaps it can be added in a subsequent PR.

[linyueqian]

resolve conflicts please

[linyueqian]

Hi! I noticed this PR removes the HNR (Harmonic-to-Noise Ratio) distortion check that was added in #1945 (compute_hnr_db + assert hnr >= MIN_HNR_DB). That check was specifically designed to catch voice clone audio distortion — distorted audio can still pass Whisper transcription and gender classification, but fails the HNR threshold (< 1.2 dB).

Could you share why the HNR check was removed? Is there an issue with it (flaky, too slow, false positives), or was it unintentional during the test refactor?

Without it, the test suite loses the ability to detect the class of bugs fixed in #1945 (streaming voice clone distortion after first few seconds).

[yenuo26]

Hi! I noticed this PR removes the HNR (Harmonic-to-Noise Ratio) distortion check that was added in #1945 (compute_hnr_db + assert hnr >= MIN_HNR_DB). That check was specifically designed to catch voice clone audio distortion — distorted audio can still pass Whisper transcription and gender classification, but fails the HNR threshold (< 1.2 dB).

Could you share why the HNR check was removed? Is there an issue with it (flaky, too slow, false positives), or was it unintentional during the test refactor?

Without it, the test suite loses the ability to detect the class of bugs fixed in #1945 (streaming voice clone distortion after first few seconds).

OK，i will check it

[linyueqian]

LGTM