[Bugfix] Fix audio.format parameter silently ignored in chat completions

[oglok]

Purpose

Fix #4716 — The audio.format field in chat completion requests was silently ignored. The server always returned WAV audio regardless of the requested format (mp3, flac, opus, etc.).

Root cause: _create_audio_choice() in serving_chat.py hardcoded response_format="wav" when constructing the CreateAudio object. The encoding infrastructure in audio_utils_mixin.py already supports all formats — it was just never wired up.

Changes:

• serving_chat.py: Extract audio.format from the request and pass it to CreateAudio instead of hardcoding "wav". Maps OpenAI's "pcm16" to soundfile's "pcm". Defaults to "wav" when not specified.
• audio_utils_mixin.py: Add fallback to WAV with a warning when soundfile.write fails for the requested codec (e.g. MP3 on systems where libsndfile lacks LAME support).

Test Plan

vLLM Version: latest

vLLM-Omni Commit: 2e6c83c

1. Serve Qwen3-Omni and send chat completion requests with audio.format set to "mp3", "flac", and "wav"
2. Verify response audio magic bytes match the requested format (e.g. MP3 frame sync, fLaC header, RIFF header)
3. Verify backward compatibility: requests without audio.format still return WAV

Test Result

• ruff check and ruff format pass on both modified files
• Manual verification confirms the format parameter is now respected

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[NickCao]

soundfile uses a bundled version of libsndfile, not the system one, so this should not happen.

[NickCao]

And aac is not supported by sndfile from pypi, so we shall not accept that as a valid format.

[oglok]

Thanks Nick! I removed the try/except fallback entirely and also dropped AAC from the supported formats dict since the bundled libsndfile doesn't support it. Added input validation in serving_chat.py to reject invalid formats with a 400 error upfront.

[NickCao]

Per https://developers.openai.com/api/reference/resources/audio/subresources/speech/methods/create, pcm16 is not a valid format name, we may want to validate user input against that list.

Edit: this is the chat completion endpoint, https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create does use pcm16.

[oglok]

Thx!

[NickCao]

And _create_diffusion_chat_completion has the same problem.

[hsliuustc0106]

Bugfix looks correct.

[alex-jw-brooks]

Can you please add some tests to verify that the format conversion is correctly handled?

[alex-jw-brooks]

Since the type of request is known here, can you avoid you use properties on the object instead of getattr?

[oglok]

I've extracted the logic into a _resolve_audio_format() helper method that centralizes this, so the getattr is in one place only. Sounds good?

[alex-jw-brooks]

Can you centralize the supported formats to use a common def instead of inlining them here? They should also be in the protocol for audio, e.g., here, it would be better to use the same def so that we don't accidentally have this go out of sync with the protocol

[oglok]

I've added SUPPORTED_AUDIO_FORMATS, SUPPORTED_CHAT_AUDIO_FORMATS, and DEFAULT_AUDIO_FORMAT constants in protocol/audio.py. Both serving_chat.py and audio_utils_mixin.py now import from there.

[alex-jw-brooks]

Similarly, I think that we should pull the default audio format from the protocol to use what is here

[oglok]

deefault format now comes from DEFAULT_AUDIO_FORMAT imported from protocol/audio.py.

[alex-jw-brooks]

Can you also make sure the docs are in sync? I.e., the examples all document aac as a format (e.g., here), so removing it as a supported format would make the behavior out of sync with OpenAI spec

[oglok]

Addressed all review feedback:

• @NickCao — Fixed _create_diffusion_chat_completion to also use the new _resolve_audio_format() helper instead of hardcoded WAV.
• @alex-jw-brooks — Centralized formats into SUPPORTED_AUDIO_FORMATS, SUPPORTED_CHAT_AUDIO_FORMATS, and DEFAULT_AUDIO_FORMAT constants in protocol/audio.py. Added unit tests for format conversion (magic bytes, validation, pcm16 mapping, base64). Updated docs to remove aac from supported output format lists. The getattr is correct since audio is a Pydantic extra field, but it's now isolated in a single _resolve_audio_format() helper.

[alex-jw-brooks]

I think that this can result in bad behavior on chat completions, since audio format comes from the extra body in this case, so it doesn't have pydantic validation. E.g., if you make a stream request to Qwen3Omni for text+audio, it will yield the text part of the request without issues, then hit the error case, and silently shut the stream down instead of yielding the audio part. Some examples that may help:

# Assumes your server is running on port 8112, e.g., running with
# vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --enforce-eager --omni --port 8112

from openai import OpenAI

BASE = "http://localhost:8112/v1"
MODEL = "Qwen/Qwen3-Omni-30B-A3B-Instruct"

client = OpenAI(base_url=BASE, api_key="dummy")

def test_streaming_invalid_format():
    print("=== Streaming with audio.format='aac' ===")
    try:
        stream = client.chat.completions.create(
            model=MODEL,
            messages=[{"role": "user", "content": "Say hello in one sentence."}],
            modalities=["text", "audio"],
            audio={"voice": "alloy", "format": "aac"},
            stream=True,
        )
        chunk_count = 0
        has_audio = False
        for chunk in stream:
            chunk_count += 1
            raw = chunk.model_dump(exclude_none=True, exclude_unset=True)
            modality = raw.get("modality", "?")
            choices = raw.get("choices", [])
            finish = None
            content_preview = ""
            for c in choices:
                delta = c.get("delta", {})
                finish = c.get("finish_reason")
                content = delta.get("content", "")
                if modality == "audio":
                    has_audio = True
                    content_preview = f"[audio {len(content)}chars]"
                elif content:
                    content_preview = repr(content[:60])
                else:
                    content_preview = repr(delta)
            print(f"  chunk {chunk_count}: modality={modality} finish={finish} {content_preview}")

        print(f"  --- {chunk_count} chunks, has_audio={has_audio}")
        if not has_audio:
            print("  BUG: no audio chunks received despite requesting modalities=['audio']")
    except Exception as e:
        print(f"  Error: {type(e).__name__}: {e}")


if __name__ == "__main__":
    test_streaming_invalid_format()

Running ^, you'll see this:

(vllm_omni) [alex-jw-brooks@h100 scratch]$ python streaming_audio_format_bug.py 
=== Streaming with audio.format='aac' ===
  chunk 1: modality=text finish=None {'content': '', 'role': 'assistant'}
  chunk 2: modality=text finish=None 'Hello'
  chunk 3: modality=text finish=None '!'
  chunk 4: modality=text finish=None {'content': ''}
  --- 4 chunks, has_audio=False
  BUG: no audio chunks received despite requesting modalities=['audio']

Rather than handling it mid stream, I think we should probably validate this upfront, e.g., something like this:

    async def _create_chat_completion(
        self,
        request: ChatCompletionRequest,
        raw_request: Request | None = None,
    ) -> AsyncGenerator[str, None] | ChatCompletionResponse | ErrorResponse:
        # Before starting the stream, check the audio format
        audio_format_check = self._resolve_audio_format(request)
        if isinstance(audio_format_check, ErrorResponse):
            return audio_format_check

       ...
        # Handle diffusion mode

which should give a 400 and better align with the non-stream case. I think we can also add a unit tests to the expansion tests for tests/e2e/online_serving/test_qwen3_omni_expansion.py to make the stream validates. e.g.,

from openai import BadRequestError

@hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)
@pytest.mark.parametrize("omni_server", test_params[:1], indirect=True)
def test_invalid_audio_format_rejected(omni_server, openai_client) -> None:
    """Ensure we reject bad audio format prior to starting streaming."""
    messages = dummy_messages_from_mix_data(
        system_prompt=get_system_prompt(), content_text=get_prompt(),
    )
    with pytest.raises(BadRequestError, match="audio format"):
        openai_client.client.chat.completions.create(
            model=omni_server.model,
            messages=messages,
            modalities=["text", "audio"],
            audio={"voice": "alloy", "format": "aac"},
            stream=True,
        )

[oglok]

thank you for the detailed explanation @alex-jw-brooks , very much appreciated!

[oglok]

So I have moved the format validation upfront to _create_chat_completion (before the streaming/non-streaming branch), so an invalid format now returns a clean 400 before any chunks are sent. Removed the isinstance(choices_data, ErrorResponse) guards from the generators since they're no longer needed and added the E2E streaming test you suggested in test_qwen3_omni_expansion.py.