Repository navigation
[New Model] Step-Audio2 - #464
Merged
linyueqian merged 5 commits intoJun 13, 2026
Merged
Conversation
Collaborator
|
@linyueqian PTAL |
linyueqian
reviewed
Dec 25, 2025
linyueqian
reviewed
Dec 25, 2025
28 of 61 tasks
linyueqian
reviewed
Dec 25, 2025
linyueqian
reviewed
Dec 25, 2025
linyueqian
reviewed
Dec 25, 2025
linyueqian
reviewed
Dec 25, 2025
wuli666
force-pushed
the
feature/step-audio2-integration
branch
3 times, most recently
from
December 29, 2025 14:44
18e9df2 to
f072adc
Compare
wuli666
marked this pull request as ready for review
December 29, 2025 16:03
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+304
to
+306
| generated_speech_tokens: torch.Tensor | list, | ||
| prompt_wav: str, | ||
| return_bytes: bool = True, |
There was a problem hiding this comment.
The Token2Wav wrapper mandates a prompt_wav argument with no default (step_audio2_token2wav.py lines 304-306), but the Stage 0→1 input processor only forwards audio token IDs and never adds a speaker path, and the stage config uses the generic GPUGenerationWorker. When Stage 1 runs it will call this forward with only the generated tokens, leading to a TypeError: forward() missing required argument 'prompt_wav' before any audio synthesis, so the two-stage pipeline cannot execute.
Useful? React with 👍 / 👎.
hsliuustc0106
left a comment
hsliuustc0106
left a comment
Collaborator
There was a problem hiding this comment.
please follow this guideline:
https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/model/adding_omni_model/
Contributor
Author
ok, I'll submit a version as soon as possible. |
Collaborator
|
I think this PR is almost ready for test and then merged, let's push it faster. |
Collaborator
wuhang2014
reviewed
Jan 5, 2026
linyueqian
reviewed
Jan 5, 2026
wuli666
force-pushed
the
feature/step-audio2-integration
branch
2 times, most recently
from
January 7, 2026 08:48
4f94598 to
a8d982b
Compare
Collaborator
|
linyueqian
reviewed
Jan 14, 2026
Closed
37 of 63 tasks
wuli666
force-pushed
the
feature/step-audio2-integration
branch
3 times, most recently
from
January 19, 2026 14:39
c2cdcb4 to
01aa912
Compare
wuli666
requested review from
Gaohan123,
congw729,
gcanlin,
tzhouam,
yenuo26 and
ywang96
as code owners
June 11, 2026 06:44
- test_step_audio2.py: import OmniRunner from tests.helpers.runtime and create_new_process_for_each_test from tests.helpers.process (the old conftest.py/utils.py were removed in the test-helpers refactor) - step_audio2_token2wav.py: hoist model_name_or_path out of the token2wav_path branch so _resolve_prompt_wav works when token2wav_path is configured (fixes UnboundLocalError) - stage_input_processors/step_audio2.py: migrate thinker2token2wav to the current calling convention (source_outputs, prompt, requires_multimodal_data), matching every other stage input processor; the stale (stage_list, engine_input_source, ...) signature caused KeyError: 0 at the Thinker->Token2Wav handoff - align the sync-path token2wav test assertion to the single-element-list contract used by the other async-chunk tests - inputs/preprocess.py: drop unused backwards-compat aliases (EmbedsInputs/SingletonInputs/MultiModalInputs) nothing referenced, restoring the file to upstream - mark required librosa imports with # noqa: TID251 (same as the minicpmo example) and apply ruff import ordering so pre-commit passes Signed-off-by: wuli666 <djjpro975@gmail.com>
wuli666
force-pushed
the
feature/step-audio2-integration
branch
from
June 11, 2026 06:55
44ef141 to
fbc5c55
Compare
linyueqian
added
ready
Collaborator
|
please fix ci and paste your successful ci run result here. i will remove ready label for now |
linyueqian
removed
ready
The "TTS . Step-Audio2 E2E Test" step collected 0 tests (exit 5): the two
weight-loading E2E tests were core_model while the step runs -m advanced_model.
Mark them advanced_model so the step collects them, and fix three issues that
otherwise made the tests unrunnable or false-green:
- step_audio2_ci.yaml: use a single Thinker stage. The audio->text / text-only
tests never emit audio tokens, so Token2Wav (stage 1) is skipped and the
orchestrator waits on the skipped stage forever (deadlock). Also drop the
Qwen2ForCausalLM hf_overrides so audio input goes through the real Thinker
audio encoder instead of vLLM`s renderer.
- Drop @create_new_process_for_each_test: the fork default crashes CUDA, and
the spawn path exits 0 without running the test (false green) because the
file has no __main__/RUNNING_IN_SUBPROCESS protocol. Run the runner directly
like the other E2E tests (Qwen3-TTS, Voxtral).
- OmniRequestOutput.request_output is a single RequestOutput, not a list:
use request_output.outputs[0] instead of request_output[0].outputs[0].
Verified on 2x GPU with the exact CI command:
pytest -s -v tests/e2e/offline_inference/test_step_audio2.py \
-m "advanced_model" --run-level "advanced_model"
=> 2 passed.
Signed-off-by: wuli666 <djjpro975@gmail.com>
Contributor
Author
|
@linyueqian CI fixed. Successful run on 2× GPU (full file, all 10 tests): The two weight-loading E2E tests also pass under the exact CI command (
|
linyueqian
approved these changes
Jun 12, 2026
linyueqian
added
ready
The `tests/diffusion tests/model_executor -m core_model and cpu` step exited 2 (1 error during collection): test_step_audio2_token2wav_async_chunk.py imports StepAudio2Token2WavForConditionalGeneration at module level, which pulls the optional step-audio2 extras (flashcosyvoice / s3tokenizer / hyperpyyaml). The CPU core_model image does not install them, so the import raised ModuleNotFoundError during collection and failed the whole sweep. - Guard the module-level import: skip (not error) when the optional extras are absent; a genuine import error (e.g. vLLM API drift) still surfaces. - Mark both Step-Audio2 model_executor unit tests [core_model, cpu] (matching ming_flash_omni / mimo_audio) so they are collected and run on the CPU sweep instead of being silently deselected. Verified (CPU): extras present -> 8 passed; simulating the CI image (extras absent) -> token2wav skips, async_chunk 4 passed, 0 collection errors. Signed-off-by: wuli666 <djjpro975@gmail.com>
Contributor
Author
|
@linyueqian PTAL |
linyueqian
added
ready
linyueqian
enabled auto-merge (squash)
June 12, 2026 23:05
linyueqian
added a commit
to linyueqian/vllm-omni
that referenced
this pull request
Jun 13, 2026
Resolve serving_speech.py conflict from Step-Audio2 (vllm-project#464): instead of the inline 'if self._tts_model_type == "step_audio2"' branch added on main, migrate Step-Audio2 onto the adapter framework (new StepAudio2Adapter delegating to the existing _validate_tts_request dispatcher + _build_step_audio2_prompt). Keeps the uniform adapter dispatch; registry now has 14 model-types. Also brings in vllm-project#4373 (Qwen3-TTS codes.ref leak), vllm-project#4397 (Ming-omni Mapping), Signed-off-by: Yueqian Lin <linyueqian@outlook.com> vllm-project#3763 (Wan2.2 SP) from main.
Nughm3
pushed a commit
to Nughm3/vllm-omni
that referenced
this pull request
Jun 18, 2026
Signed-off-by: linyueqian <linyueqian@outlook.com> Signed-off-by: wuli666 <djjpro975@gmail.com> Co-authored-by: linyueqian <linyueqian@outlook.com> Co-authored-by: wuli666 <djjpro975@gmail.com>
khairulkabir1661
pushed a commit
to khairulkabir1661/vllm-omni
that referenced
this pull request
Sep 25, 2026
Signed-off-by: linyueqian <linyueqian@outlook.com> Signed-off-by: wuli666 <djjpro975@gmail.com> Co-authored-by: linyueqian <linyueqian@outlook.com> Co-authored-by: wuli666 <djjpro975@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support complete Step-Audio2 integration with two-stage pipeline (Thinker → Token2Wav) for ASR, TTS, and S2ST tasks.
Architecture
Stage 0: Thinker (Qwen2 LLM + Audio Encoder) → text + audio tokens (ID ≥ 151696) │ │ async_chunk ▼ Stage 1: Token2Wav (Flow Model + HiFT Vocoder) → 24kHz waveform (streaming overlap-add)step_audio2_thinker.py: Stage-0 model. Qwen2-based LLM with audio encoder for speech understanding and audio token generation. Supports multimodal audio input via
StepAudio2Processor.step_audio2_token2wav.py: Stage-1 model. Flow model (conformer encoder + DiT estimator) + HiFT vocoder for audio token to 24kHz waveform conversion. Supports both streaming (per-request
_StreamStatewith chunked overlap-add) and non-streaming modes.step_audio2.py (stage_input_processors): Async chunk processor
thinker2token2wav_async_chunkthat extracts audio tokens from the AR token stream and sends them in chunks (chunk_size=25 + lookahead=3) to Token2Wav.serving_speech.py: Registered Step-Audio2 in
/v1/audio/speechendpoint with dedicated_build_step_audio2_prompt()that injects<tts_start>without chat template interference.Stage configs:
step_audio_2_async_chunk.yaml(streaming, recommended),step_audio_2.yaml(sequential),step_audio_2_asr.yaml(ASR-only, single stage).Related issues: #271
Installation:
pip install step-audio2(includes flashcosyvoice, cosyvoice2, s3tokenizer, hyperpyyaml, etc.)Purpose
Resolves #271
Test Plan
Tested on 4x RTX 3090 with
stepfun-ai/Step-Audio2-mini./v1/audio/speech, ASR and S2ST via/v1/chat/completionsend2end.pywith all three query types (audio_to_text, text_to_audio, audio_to_audio)bench_tts_serve.py(10 prompts, concurrency=1)Test Result
Online TTS (
/v1/audio/speech): clean 24kHz audio output ✅Online ASR (chat completions): correct transcription ✅
Online S2ST (chat completions): text + audio output ✅
Offline ASR/TTS/S2ST: all produce correct output ✅
Async chunk vs Sequential (4x RTX 3090, 10 prompts, concurrency=1):
vLLM-Omni vs Official Step-Audio2 (H100 80GB):