Skip to content

Migrate audiox examples - #4607

Merged
princepride merged 3 commits into
vllm-project:mainfrom
zzehli:migrate-audiox-examples
Jun 30, 2026
Merged

princepride merged 3 commits into
vllm-project:mainfrom
zzehli:migrate-audiox-examples

Conversation

@zzehli

@zzehli zzehli commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

[Feature] Migrate AudioX offline to standard task example + declare model_extras

Purpose

Closes #4540

Migrates AudioX off its bespoke examples onto the standard task example + a
declarative model_extras registration, following the reference migration in
#4225 (BagelPipeline / SenseNovaU1Pipeline).

What changed:

  • New vllm_omni/model_extras/audiox.py declaring the request/response param
    contract (AUDIOX_EXTRA_BODY_PARAMS, AUDIOX_EXTRA_OUTPUT_PARAMS).
  • Registered "AudioXPipeline" in vllm_omni/model_extras/registry.py, so
    AudioX knobs sent via extra_body (online) / --extra-body (offline) now flow
    through the standard apply_declared_extra_args path into
    sampling_params.extra_args instead of being dropped.
  • Folded all 6 tasks (t2a/t2m/v2a/v2m/tv2a/tv2m) into the standard
    examples/offline_inference/text_to_audio/text_to_audio.py via --task +
    --video + --extra-body. Stable Audio Open behavior is preserved (fallback path
    when the model is not registered).
  • Simplified the shared flags: collapsed the duplicate duration
    vocabularies onto a single --audio-length (→ audio length for Stable Audio,
    seconds_total for AudioX) and a single --audio-start (→ audio_start_in_s /
    seconds_start). Dropped the 6 AudioX-only pass-through flags
    (--seconds-start, --seconds-total, --sigma-min, --sigma-max,
    --cfg-rescale, --reference-audio); sampler/reference knobs now arrive via
    --extra-body only, matching the helios reference migration ([Feature]Migrate helios example #4569). AudioX runs
    default their output to audiox_output.wav.
  • Deleted the bespoke examples/offline_inference/audiox/ dir (folded into
    the standard offline example). Kept examples/online_serving/audiox/ as a
    dedicated online example, but switched its request contract from the old
    top-level extra_args escape hatch to flattened extra_body keys routed
    through apply_declared_extra_args.
  • Tests: added CPU registry unit tests in
    tests/model_extras/test_model_extras.py; updated the online e2e test to send
    sigma_* knobs through the now-declared extra_body path.
  • Docs: updated recipes/audiox/AudioX.md to document the extra_body knobs
    and the standard example invocations.

The AudioX pipeline itself needs no behavioral change — it already reads every
knob from extra_args (pipeline_audiox.py:824-918).

Test Plan

Offline

Text-to-audio (t2a):

python examples/offline_inference/text_to_audio/text_to_audio.py \
  --model zhangj1an/AudioX --task t2a \
  --prompt "Fireworks burst twice, then a clock ticking." \
  --output /tmp/audiox_t2a.wav

Text-to-music (t2m):

python examples/offline_inference/text_to_audio/text_to_audio.py \
  --model zhangj1an/AudioX --task t2m \
  --prompt "Uplifting ukulele tune for a travel vlog" \
  --output audiox_t2m.wav

Text+video-to-audio (tv2a, requires --video):

python examples/offline_inference/text_to_audio/text_to_audio.py \
  --model zhangj1an/AudioX --task tv2a \
  --prompt "drum beating sound and human talking" \
  --video https://zeyuet.github.io/AudioX/static/samples/V2M/1XeBotOFqHA.mp4 \
  --output audiox_tv2a.wav

Online

Start the server in one session:

bash examples/online_serving/audiox/run_server.sh

Calls the server to generate the audio file us the sme task:

python examples/online_serving/audiox/openai_chat_client.py --task t2m     --prompt "Uplifting ukulele tune for a travel vlog"     --output t2m-online.wav

Test Result

Wasn't able to upload the wav files directly here. Format not supported by gh. Here is the zip file for all 3 results from above:
audio-combined.zip

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AudioX migration to standard examples looks correct.

@zzehli

zzehli commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

@princepride Hi ZP, can you take a look at this PR? I made the migration based on your code.

@princepride princepride self-assigned this Jun 22, 2026
@zzehli
zzehli force-pushed the migrate-audiox-examples branch from 70d088b to 9e1e6da Compare June 24, 2026 03:12
@princepride

Copy link
Copy Markdown
Collaborator

@zzehli Hi, DCO failed

@zzehli
zzehli force-pushed the migrate-audiox-examples branch from 9d4518e to 6a1db2e Compare June 24, 2026 11:19
@zzehli

zzehli commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

@princepride One thing about this PR, there is currently no online_serving generic client for text_to_audio. Creating and moving existing model online examples to a new client (audiox and stable audio) would touch more files than the recurrent PR scope. So I only updated the existing audiox online_serving example folder to accomodate extra_body. I can create a subsequent PR to migrate text_to_audio online_serving examples to a single client. Let me know what you think.

@princepride

Copy link
Copy Markdown
Collaborator

@princepride One thing about this PR, there is currently no online_serving generic client for text_to_audio. Creating and moving existing model online examples to a new client (audiox and stable audio) would touch more files than the recurrent PR scope. So I only updated the existing audiox online_serving example folder to accomodate extra_body. I can create a subsequent PR to migrate text_to_audio online_serving examples to a single client. Let me know what you think.

Yeah, you can create a subsequent PR

zzehli added 3 commits June 29, 2026 08:20
Signed-off-by: zzehli <jaeli_ottawa@outlook.com>
Signed-off-by: zzehli <jaeli_ottawa@outlook.com>
Signed-off-by: zzehli <jaeli_ottawa@outlook.com>
@zzehli
zzehli force-pushed the migrate-audiox-examples branch from 6a1db2e to ce4b4a9 Compare June 29, 2026 12:21
@princepride princepride added the ready label to trigger buildkite CI label Jun 30, 2026
@princepride
princepride enabled auto-merge (squash) June 30, 2026 04:00
@princepride
princepride merged commit 0899a1a into vllm-project:main Jun 30, 2026
9 checks passed
wkutak pushed a commit to wkutak/vllm-omni-cosmos-genai-nim that referenced this pull request Jul 13, 2026
Signed-off-by: zzehli <jaeli_ottawa@outlook.com>
Signed-off-by: Wojciech Kutak <wkutak@nvidia.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
Signed-off-by: zzehli <jaeli_ottawa@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC sub-task] Migrate audiox example to standard task example + model_extras

3 participants