[Misc] move text-to-audio online examples to a unified folder by zzehli · Pull Request #4807 · vllm-project/vllm-omni

zzehli · 2026-06-30T16:37:58Z

Sub-task of #4539.

Purpose

Migrate the per-model online-serving examples for the two text/video-to-audio diffusion models into a single shared task folder, matching the structure of examples/online_serving/text_to_video and aligning with the RFC's "one standard task entrypoint per task" goal.

Consolidates examples/online_serving/audiox/ and examples/online_serving/stable_audio/ into one folder: examples/online_serving/text_to_audio/.
Because the two models are served through different endpoints, each gets its own server + curl pair (same pattern as text_to_video):
- Stable Audio Open (stabilityai/stable-audio-open-1.0) → POST /v1/audio/generate (JSON in, binary WAV out): run_server_stable_audio.sh, run_curl_stable_audio.sh.
- AudioX (zhangj1an/AudioX) → POST /v1/chat/completions with --model-class-name AudioXPipeline; task/sampler knobs declared in vllm_omni/model_extras/audiox.py are passed via extra_body, and the base64 WAV is decoded from choices[0].message.audio.data: run_server_audiox.sh, run_curl_audiox.sh (TASK-driven: t2a/t2m/v2a/v2m/tv2a/tv2m, with local videos inlined as a data:video/mp4;base64,... URI for v2*/tv2*).
Adds a unified README.md covering both models (start-server + curl recipes, the AudioX extra_body knob table, and a pointer to the shared offline entrypoint).
Deletes the old per-model example dirs and the bespoke Python clients (openai_chat_client.py, stable_audio_client.py).
Adds the missing Stable Audio online e2e test (tests/e2e/online_serving/test_stable_audio_online_expansion.py) so both migrated models are exercised on their standard online path; AudioX already had test_audiox_expansion.py.
Updates all references: docs/user_guide/examples/online_serving/text_to_audio.md (now covers both models + --8<-- includes the new scripts), adds the missing nav entry in docs/.nav.yml, and repoints recipes/StabilityAI/Stable-Audio-Open.md and recipes/audiox/AudioX.md at the new folder.

No engine/serving code changed — this is an examples + docs + tests migration; the legacy fallback in the serving layer is untouched per the RFC's incremental-transition note.

Test

vLLM Version: 0.24.0

vLLM-Omni Commit: fa3720a

See attachments for online-serving stable audio and audiox result. They are generated with the following commands.
Archive.zip

Run audiox server:

bash examples/online_serving/text_to_audio/run_server_audiox.sh

Curl results:

bash examples/online_serving/text_to_audio/run_curl_audiox.sh

Run stable audio server:

bash examples/online_serving/text_to_audio/run_server_stable_audio.sh

Curl results:

bash examples/online_serving/text_to_audio/run_curl_stable_audio.sh

chatgpt-codex-connector · 2026-06-30T16:38:05Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

zzehli · 2026-06-30T16:58:54Z

@princepride Hi zp, I made the online text-to-audio migration based on conversation in #4607 . Can you take a look?

princepride · 2026-07-19T06:14:20Z

@zzehli sorry for late review, can you resolve the conflicts?

zzehli · 2026-07-21T22:33:25Z

@princepride Resolved the merge conf. Can you take another look?

zzehli · 2026-08-20T13:26:59Z

@princepride I rebased this PR to main. It's ready to be viewed/merged.

Copilot

Pull request overview

This PR consolidates the per-model online-serving examples for text/video-to-audio diffusion into a single task folder (examples/online_serving/text_to_audio/), aligning the repository’s example structure with other unified task entrypoints and updating docs/recipes accordingly.

Changes:

Moved Stable Audio Open and AudioX online-serving scripts into a unified text_to_audio example folder with a combined README and per-model server/curl helpers.
Removed legacy per-model online-serving example directories and bespoke Python clients for Stable Audio and AudioX.
Added a new Stable Audio online E2E test and updated docs navigation + recipe links to point at the unified folder.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/e2e/online_serving/test_stable_audio_online_expansion.py	Adds Stable Audio Open online-serving E2E test for `/v1/audio/generate`.
recipes/StabilityAI/Stable-Audio-Open.md	Repoints recipe links/paths to the unified `text_to_audio` online example folder.
recipes/audiox/AudioX.md	Repoints AudioX recipe to the unified `text_to_audio` online example folder.
examples/online_serving/text_to_audio/run_server_stable_audio.sh	Adds Stable Audio Open server launch helper for `/v1/audio/generate`.
examples/online_serving/text_to_audio/run_server_audiox.sh	Updates AudioX server launch helper with clarified requirements.
examples/online_serving/text_to_audio/run_curl_stable_audio.sh	Adds Stable Audio Open curl helper that saves binary audio output.
examples/online_serving/text_to_audio/run_curl_audiox.sh	Adds AudioX curl helper that posts chat-completions + decodes base64 WAV.
examples/online_serving/text_to_audio/README.md	Adds unified README documenting both models’ online-serving flows.
examples/online_serving/stable_audio/stable_audio_client.py	Removes legacy Stable Audio Python client.
examples/online_serving/stable_audio/README.md	Removes legacy Stable Audio per-model README.
examples/online_serving/stable_audio/curl_examples.sh	Removes legacy Stable Audio curl examples script.
examples/online_serving/audiox/README.md	Removes legacy AudioX per-model README.
examples/online_serving/audiox/openai_chat_client.py	Removes legacy AudioX Python client.
docs/user_guide/examples/online_serving/text_to_audio.md	Updates user guide page to cover both models and include new scripts.
docs/.nav.yml	Adds missing nav entry for the text-to-audio online serving page.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        pytest.param(
+            OmniServerParams(model=model),
+            id="t2a",
+            marks=SINGLE_CARD_FEATURE_MARKS,
+        ),


+    responses = openai_client.send_audio_generate_http_request(request_config)
+    assert responses, "no response from /v1/audio/generate"
+    resp = responses[0]
+    assert resp.success, f"audio generate failed: {resp.status_code} {resp.error_message}"
+    assert resp.status_code == 200


+read -r -d '' PAYLOAD <<EOF || true
+{
+  "input": "${PROMPT}",
+  "negative_prompt": "${NEGATIVE_PROMPT}",
+  "audio_length": ${AUDIO_LENGTH},
+  "guidance_scale": ${GUIDANCE_SCALE},
+  "num_inference_steps": ${NUM_INFERENCE_STEPS},
+  "seed": ${SEED},
+  "response_format": "${RESPONSE_FORMAT}"
+}
+EOF


hsliuustc0106 · 2026-08-20T15:20:24Z

+    responses = openai_client.send_audio_generate_http_request(request_config)
+    assert responses, "no response from /v1/audio/generate"
+    resp = responses[0]
+    assert resp.success, f"audio generate failed: {resp.status_code} {resp.error_message}"


The assertions stop at HTTP success, but HttpResponse doesn't carry the response body — a 200 with an empty or non-WAV body would pass, while the docstring promises a non-empty WAV. Consider capturing r.content on HttpResponse in _http_response_from_requests and asserting non-empty + RIFF magic (mirrors assert_audio_valid in the offline test).

hsliuustc0106 · 2026-08-20T15:20:26Z

+
+read -r -d '' PAYLOAD <<EOF || true
+{
+  "input": "${PROMPT}",


PROMPT/NEGATIVE_PROMPT are string-interpolated into the JSON heredoc, so a value containing a double quote or backslash produces invalid JSON and a 400 from the server. The sibling run_curl_audiox.sh and the text_to_video curl scripts build payloads with jq -n --arg; doing the same here would make env overrides safe.

hsliuustc0106

LGTM. Examples/docs migration matches the RFC #4539 pattern and the e2e wiring mirrors the AudioX counterpart; the two inline comments are non-blocking nits.

Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

zzehli · 2026-08-25T12:52:36Z

remove audiox changes since it's no longer supported

zzehli · 2026-08-25T12:56:21Z

@princepride @hsliuustc0106 I removed audioX docs from the PR since it's no longer supported. Can you merge it again?

…roject#4807) Signed-off-by: zzehli <jaeli_ottawa@outlook.com> Signed-off-by: AndyZhou952 <jzhoubc@connect.ust.hk>

…roject#4807) Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

zzehli requested review from Gaohan123, david6666666, hsliuustc0106, yenuo26 and ywang96 as code owners June 30, 2026 16:37

zzehli force-pushed the text-to-audio-online-serving-examples branch from fa3720a to 1cda88d Compare June 30, 2026 16:40

zzehli changed the title ~~[Refactor]: move text-to-audio online examples to a unified folder~~ [Misc] move text-to-audio online examples to a unified folder Jun 30, 2026

hsliuustc0106 added frontend code related to entrypoint refactor refactoring for better code scalability and quality labels Jul 20, 2026 — with ChatGPT Codex Connector

zzehli force-pushed the text-to-audio-online-serving-examples branch from 1cda88d to 3a3b06c Compare July 21, 2026 22:32

zzehli force-pushed the text-to-audio-online-serving-examples branch from 3a3b06c to bd530f1 Compare August 20, 2026 13:01

zzehli requested a review from NickCao as a code owner August 20, 2026 13:01

hsliuustc0106 added the high priority high priority issue, needs to be done asap label Aug 20, 2026

hsliuustc0106 requested a lite review from Copilot August 20, 2026 15:06

Copilot AI reviewed Aug 20, 2026

View reviewed changes

hsliuustc0106 reviewed Aug 20, 2026

View reviewed changes

hsliuustc0106 added ready label to trigger buildkite CI npu-test labels Aug 20, 2026

hsliuustc0106 approved these changes Aug 20, 2026

View reviewed changes

Copilot started reviewing on behalf of hsliuustc0106 August 20, 2026 17:26

View session

princepride approved these changes Aug 20, 2026

View reviewed changes

princepride enabled auto-merge (squash) August 20, 2026 23:15

zzehli added 2 commits August 25, 2026 08:45

[Misc] Move text-to-audio online examples to a unified folder

124aba2

Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

[Doc] add text_to_audio doc

1b6af82

Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

zzehli added 2 commits August 25, 2026 08:45

[Doc] misc text_to_audio doc changes

2d371b0

Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

[Doc] remove online text_to_audio scripts

5ba9712

Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

auto-merge was automatically disabled August 25, 2026 12:52
Head branch was pushed to by a user without write access

zzehli force-pushed the text-to-audio-online-serving-examples branch from c4b087e to 5ba9712 Compare August 25, 2026 12:52

hsliuustc0106 merged commit 72beeae into vllm-project:main Aug 25, 2026
5 checks passed

JoseCarlosGarcia95 pushed a commit to valendra-tech/vllm-omni that referenced this pull request Sep 5, 2026

[Misc] move text-to-audio online examples to a unified folder (vllm-p…

6d0cdca

…roject#4807) Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026

[Misc] move text-to-audio online examples to a unified folder (vllm-p…

803a0fa

…roject#4807) Signed-off-by: zzehli <jaeli_ottawa@outlook.com>

Navigation Menu

Repository navigation

[Misc] move text-to-audio online examples to a unified folder - #4807

[Misc] move text-to-audio online examples to a unified folder#4807
hsliuustc0106 merged 4 commits into
vllm-project:mainfrom
zzehli:text-to-audio-online-serving-examples

zzehli commented Jun 30, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Jun 30, 2026

Uh oh!

zzehli commented Jun 30, 2026

Uh oh!

princepride commented Jul 19, 2026

Uh oh!

zzehli commented Jul 21, 2026

Uh oh!

zzehli commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

hsliuustc0106 Aug 20, 2026

Uh oh!

hsliuustc0106 Aug 20, 2026

Uh oh!

hsliuustc0106 left a comment •

edited

Loading

Uh oh!

zzehli commented Aug 25, 2026

Uh oh!

zzehli commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zzehli commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test

Uh oh!

chatgpt-codex-connector Bot commented Jun 30, 2026

Uh oh!

zzehli commented Jun 30, 2026

Uh oh!

princepride commented Jul 19, 2026

Uh oh!

zzehli commented Jul 21, 2026

Uh oh!

zzehli commented Aug 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

hsliuustc0106 Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zzehli commented Aug 25, 2026

Uh oh!

zzehli commented Aug 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zzehli commented Jun 30, 2026 •

edited

Loading

hsliuustc0106 left a comment •

edited

Loading