[Migrate]Move model-specific helpers out of serving_speech by sphinxkkkbc · Pull Request #6697 · vllm-project/vllm-omni

sphinxkkkbc · 2026-08-27T16:17:02Z

PLEASE FILL IN THE PR DESCRIPTION HERE.

Purpose

Move the scattered model-specific helpers out of serving_speech.py, as the validation/build/sampling_params lifecycle has been stabilized, and newly supported models already implement such helpers within their adapters instead of in the main serving file.

This work does not overlap with the M-series changes in #4855. The PR only relocates helpers that have a single model consumer. Potentially shared state, such as self._tts_tokenizer, remains unchanged. This is intended to be a structural refactor only, with no changes to serving behavior or model capabilities. The affected helpers are listed below.

The related tests have been updated to follow the new ownership boundaries. One potentially non-obvious case is build_tts_params: although it previously appeared to be a shared serving helper, its only consumer in online serving is Qwen3-TTS. It has therefore been moved into Qwen3TTSAdapter.

Corresponding tests in TestTTSMethods now call: server._adapter._build_tts_params(...) instead of: server._build_tts_params(...). This makes the tests reflect the actual model-specific ownership of the helper without changing the behavior under test.

One non-helper change involves Qwen3TTSAdapter.apply_sampling_overrides, which does not introduce behavioral regression. This PR only reorders the call sequence, and the final result remains identical to that on the main branch. This change was omitted in #5272.

Status

Self checked all affected helpers, not found regression behavior.

Migration Mapping Table

Original Function / Location	Migrated Location
`_conditioning_cache_salt`	`tts_adapters.base.conditioning_cache_salt`
`_estimate_ref_code_len`	`Qwen3TTSAdapter._estimate_ref_code_len`
`_estimate_prompt_len`	`Qwen3TTSAdapter._estimate_prompt_len`
`_build_tts_params`	`Qwen3TTSAdapter._build_tts_params`
`_build_qwen3_tts_request`	Inlined into `Qwen3TTSAdapter.build`
`_estimate_fish_ref_code_len`	`FishSpeechAdapter._estimate_ref_code_len`
`_estimate_fish_prompt_len`	`FishSpeechAdapter._estimate_prompt_len`
`_build_fish_speech_prompt`	`FishSpeechAdapter._build_prompt`
`_voxcpm2_encode`	`VoxCPM2Adapter._encode`
`_build_voxcpm2_prompt`	`VoxCPM2Adapter._build_prompt`
`_detect_moss_variant`	`_MossTTSAdapterBase._detect_moss_variant`
`_get_moss_processor`	`_MossTTSAdapterBase._get_moss_processor`
`_build_moss_tts_params`	`_MossTTSAdapterBase._build_moss_tts_params`
`_build_higgs_audio_v2_params`	`HiggsAudioV2Adapter._build_higgs_audio_v2_params`
`_resolve_higgs_audio_v2_processor`	`HiggsAudioV2Adapter._resolve_higgs_audio_v2_processor`
`_build_higgs_audio_v3_params`	`HiggsAudioV3Adapter._build_higgs_audio_v3_params`
`_resolve_higgs_audio_v3_ref_codes`	`HiggsAudioV3Adapter._resolve_higgs_audio_v3_ref_codes`
`_get_higgs_audio_v3_ref_codes`	`HiggsAudioV3Adapter._get_higgs_audio_v3_ref_codes`
`_put_higgs_audio_v3_ref_codes`	`HiggsAudioV3Adapter._put_higgs_audio_v3_ref_codes`
`_resolve_higgs_audio_v3_adapter`	`HiggsAudioV3Adapter._resolve_higgs_audio_v3_adapter`
`_coerce_ming_prompt_waveform`	`MingTTSAdapter._coerce_ming_prompt_waveform`
`_build_ming_reference_waveforms`	`MingTTSAdapter._build_ming_reference_waveforms`
`_parse_ming_instruction`	`MingTTSAdapter._parse_ming_instruction`
`_build_ming_dense_prompt`	`MingTTSAdapter._build_ming_dense_prompt`
`_parse_ming_instruction_fields`	Module function `tts_adapters.ming_tts.parse_ming_instruction_fields`
`_build_ming_flash_omni_prompt`	Inlined into `MingFlashOmniTTSAdapter.build`
`_build_voxtral_prompt`	`VoxtralTTSAdapter._build_prompt`
`_build_step_audio2_prompt`	Inlined into `StepAudio2Adapter.build`
`_build_cosyvoice3_prompt`	`CosyVoice3Adapter._build_prompt`
`_build_covo_audio_prompt`	Inlined into `CovoAudioAdapter.build`
`_get_audex_tokenizer`	Split into `AudexAdapter._get_tokenizer` and `AudexTTAAdapter._get_tokenizer`
`_build_glm_tts_prompt`	`GlmTTSAdapter._build_prompt`
`_glm_tts_text_tokenizer_and_frontend`	`GlmTTSAdapter._text_tokenizer_and_frontend`
`_estimate_glm_tts_text_token_len`	`GlmTTSAdapter._estimate_text_token_len`
`_build_glm_tts_prefill_metadata`	`GlmTTSAdapter._build_prefill_metadata`

Follow up

test_serving_speech.py should be reorganized because TestTTSMethods currently mixes tests for shared serving behavior with tests for model-specific helpers and validation. Several other test classes have the same issue. As a follow-up, these tests should be separated into shared serving tests and per-model adapter tests, potentially under a dedicated tts_adapters/ test directory or test_tts_adapter for validate/build/sampling/capabilities check, rather than continuing to add model-specific coverage to test_serving_speech.py

Test Plan

pytest -v tests/entrypoints/openai_api/test_audex_serving_guards.py
pytest -v tests/entrypoints/openai_api/test_serving_speech.py
pytest -v tests/entrypoints/openai_api/test_tts_adapter.py
pytest -v tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py
pytest -v tests/model_executor/models/ming_tts/test_ming_shared_modules.py

vLLM Version:
0.28.0
vLLM-Omni Commit:
76c3257

Test Result

No failed test.

Test File	Passed
`tests/entrypoints/openai_api/test_audex_serving_guards.py`	49
`tests/entrypoints/openai_api/test_serving_speech.py`	238
`tests/entrypoints/openai_api/test_tts_adapter.py`	32
`tests/model_executor/models/higgs_audio_v3/test_higgs_audio_v3.py`	77
`tests/model_executor/models/ming_tts/test_ming_shared_modules.py`	26

BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

…:sphinxkkkbc/vllm-omni into refactor/serving_speech_helper_migration

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

…:sphinxkkkbc/vllm-omni into refactor/serving_speech_helper_migration Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

hsliuustc0106 · 2026-08-28T01:39:15Z

This PR touches vllm_omni/entrypoints/, tests/entrypoints/, tests/model_executor/, tools/pre_commit/, vllm_omni/model_executor/ (23 files). Based on CODEOWNERS coverage of the changed files, the most-related reviewers appear to be:

@Gaohan123 @hsliuustc0106 @ywang96

Could one of you take a look when you get a chance? Thanks!

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

chatgpt-codex-connector · 2026-08-28T06:03:53Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

vllm-omni-review-bot · 2026-08-28T06:06:29Z

This PR appears to belong to: docs/design/module/entrypoints.md.

Module owners: @alex-jw-brooks @linyueqian @NickCao

@sphinxkkkbc, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

tzhouam · 2026-08-28T08:50:31Z

@vllm-omni-review-bot please review

…r/serving_speech_helper_migration Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com> # Conflicts: # tests/entrypoints/openai_api/test_serving_speech.py # tests/entrypoints/openai_api/test_tts_adapter.py # vllm_omni/entrypoints/openai/tts_adapters/base.py # vllm_omni/entrypoints/openai/tts_adapters/qwen3_tts.py

sphinxkkkbc · 2026-08-30T09:50:51Z

@vllm-omni-review-bot can you help review?

sphinxkkkbc · 2026-08-30T11:51:55Z

@linyueqian PTAL, the legacy pass still causes event-loop blocking(e.g. _get_tokenizer() in tts_adapter/audex or _get_moss_processor() in tts_adapter/moss_tts). I'm considering whether we should handle them all at once in this PR or keep the existing scope and leave it to another PR.

vllm-omni-review-bot · 2026-08-31T04:03:32Z

Omni ReviewBot triage note

Automated triage of commit ba1e0aed91b8 produced:

Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

…r/serving_speech_helper_migration

sphinxkkkbc · 2026-09-02T14:13:07Z

Resolved conflicts, @linyueqian PTAL, Thanks!

hsliuustc0106 · 2026-09-06T08:16:49Z

resolve conflcits

…r/serving_speech_helper_migration

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

…ect#6697) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com> Signed-off-by: ZhengWG <zwg0606@gmail.com>

…ect#6697) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

…ect#6697) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com> Signed-off-by: wenjie.yan <wenjyan@outlook.com>

main finished moving per-model helpers out of serving_speech.py (vllm-project#6697, RFC vllm-project#4327), and non-streaming delta accumulation is now driven by OutputPolicy.accumulate_nonstreaming rather than a hardcoded model-type check. The rebase takes main's serving_speech.py unchanged, so: - Vevo2Adapter.validate()/build() now own what _validate_vevo2_request / _build_vevo2_params did on the server. - build() returns OutputPolicy(accumulate_nonstreaming=True), replacing the ("moss_tts_nano", "vevo2") entry in the old hardcoded check. Without it the non-streaming response keeps only the final, empty sentinel chunk. - _resolve_ref_audio now returns (wav, sr, cache_key); unpack all three. The new adapter tests mock the server with spec=OmniOpenAIServingSpeech, so a call into a server helper that no longer exists fails in the test. Signed-off-by: Yash Jain <yashjainjain1704@gmail.com>

…ect#6697) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

sphinxkkkbc added 3 commits August 27, 2026 21:36

move model-specific helpers out of serving_speech

64412b3

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

move model-specific helpers out of serving_speech

fbfec4b

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

Merge branch 'refactor/serving_speech_helper_migration' of github.com…

d66e684

…:sphinxkkkbc/vllm-omni into refactor/serving_speech_helper_migration

sphinxkkkbc changed the title ~~[Refactor]Move model-specific helpers out of serving_speech~~ [Migrate]Move model-specific helpers out of serving_speech Aug 27, 2026

sphinxkkkbc added 3 commits August 28, 2026 00:32

fix bugs and behavior regression

1b6097c

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

fix bugs and behavior regression

f34c339

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

Merge branch 'refactor/serving_speech_helper_migration' of github.com…

477ec34

…:sphinxkkkbc/vllm-omni into refactor/serving_speech_helper_migration Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch from eb4f8e5 to 477ec34 Compare August 27, 2026 16:42

hsliuustc0106 added tts code related to tts models frontend code related to entrypoint refactor refactoring for better code scalability and quality labels Aug 28, 2026

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch 2 times, most recently from 76c3257 to 30c0b8a Compare August 28, 2026 05:55

remaining migration

a56e49f

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch from 30c0b8a to a56e49f Compare August 28, 2026 06:01

sphinxkkkbc marked this pull request as ready for review August 28, 2026 06:03

sphinxkkkbc requested review from Gaohan123, NickCao, Sy0307, alex-jw-brooks, amy-why-3459, gcanlin, hsliuustc0106, linyueqian, tzhouam, yenuo26 and ywang96 as code owners August 28, 2026 06:03

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch from bda6072 to 195ae73 Compare August 30, 2026 09:37

hsliuustc0106 approved these changes Aug 31, 2026

View reviewed changes

Merge branch 'main' of github.com:vllm-project/vllm-omni into refacto…

cdd3e6a

…r/serving_speech_helper_migration

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch from 1dc4055 to cdd3e6a Compare September 6, 2026 03:57

hsliuustc0106 added the high priority high priority issue, needs to be done asap label Sep 6, 2026

sphinxkkkbc added 2 commits September 6, 2026 17:00

Merge branch 'main' of github.com:vllm-project/vllm-omni into refacto…

0e6085b

…r/serving_speech_helper_migration

resolve conflicts

ba1e0ae

Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

sphinxkkkbc force-pushed the refactor/serving_speech_helper_migration branch from b6d689b to ba1e0ae Compare September 6, 2026 09:07

sphinxkkkbc requested a review from hsliuustc0106 September 6, 2026 10:52

hsliuustc0106 added ready label to trigger buildkite CI cuda-test Used to trigger vllm-omni cuda CI separately. labels Sep 6, 2026

hsliuustc0106 merged commit 043f03d into vllm-project:main Sep 8, 2026
6 checks passed

EchoHayate mentioned this pull request Sep 8, 2026

[Bugfix][TTS] Isolate shared Higgs v3 reference encode from request cancellation #7076

Merged

linyueqian mentioned this pull request Sep 9, 2026

[Frontend] Move non-streaming MP4 response encoding off the event loop #6579

Open

haic0 pushed a commit to haic0/vllm-omni that referenced this pull request Sep 9, 2026

[Migrate]Move model-specific helpers out of serving_speech (vllm-proj…

0a8cc52

…ect#6697) Signed-off-by: boatman <109857087+sphinxkkkbc@users.noreply.github.com>

yashkgp mentioned this pull request Sep 24, 2026

[Model] Add Vevo2 — unified AR + flow-matching TTS (#3391) #3422

Closed

9 of 11 tasks

Navigation Menu

Repository navigation

[Migrate]Move model-specific helpers out of serving_speech - #6697

[Migrate]Move model-specific helpers out of serving_speech#6697
hsliuustc0106 merged 11 commits into
vllm-project:mainfrom
sphinxkkkbc:refactor/serving_speech_helper_migration

sphinxkkkbc commented Aug 27, 2026 •

edited

Loading

Uh oh!

hsliuustc0106 commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

vllm-omni-review-bot commented Aug 28, 2026

Uh oh!

tzhouam commented Aug 28, 2026

Uh oh!

sphinxkkkbc commented Aug 30, 2026

Uh oh!

sphinxkkkbc commented Aug 30, 2026

Uh oh!

vllm-omni-review-bot commented Aug 31, 2026 •

edited

Loading

Uh oh!

sphinxkkkbc commented Sep 2, 2026

Uh oh!

hsliuustc0106 commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sphinxkkkbc commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Status

Follow up

Test Plan

Test Result

Uh oh!

hsliuustc0106 commented Aug 28, 2026

Uh oh!

chatgpt-codex-connector Bot commented Aug 28, 2026

Uh oh!

vllm-omni-review-bot commented Aug 28, 2026

Uh oh!

tzhouam commented Aug 28, 2026

Uh oh!

sphinxkkkbc commented Aug 30, 2026

Uh oh!

sphinxkkkbc commented Aug 30, 2026

Uh oh!

vllm-omni-review-bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot triage note

Uh oh!

sphinxkkkbc commented Sep 2, 2026

Uh oh!

hsliuustc0106 commented Sep 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sphinxkkkbc commented Aug 27, 2026 •

edited

Loading

vllm-omni-review-bot commented Aug 31, 2026 •

edited

Loading