[Bugfix][Multimodal] Fix packed PyAV audio conversion

[waizuichougou]

Purpose

This fixes an end-to-end correctness failure in PyAV audio decoding. Packed or
integer decoder frames are misinterpreted as planar float audio, silently
producing a model input with the wrong time axis and out-of-range amplitudes.
The corrupted waveform then reaches downstream resampling and the audio
encoder, causing intelligible speech to produce unrelated or repetitive model
output instead of a transcription.

load_audio_pyav assumes AudioFrame.to_ndarray() always returns planar float
audio in (channels, samples) layout. PyAV instead returns packed stereo as
one interleaved row with shape (1, samples * 2). The current channel mean is
therefore applied across a single row and does nothing: left and right samples
remain interleaved but are presented downstream as one mono stream with twice
the real duration. In addition, astype(np.float32) changes only the dtype;
it does not normalize integer PCM values to the expected floating-point audio
range.

The failure is deterministic for the affected decoder output formats when the
requested sample rate matches the native rate, including the sr=None path
used by AudioMediaIO. The stereo PCM WAV, PCM MOV, and FLAC fixtures below
produce twice as many samples and integer-scale amplitudes. Planar integer
codecs such as ALAC keep the correct length but still produce raw integer-scale
amplitudes; the ALAC fixture below peaks around 9.7e8 instead of 0.452.
This affects the explicit pyav backend and direct load_audio_pyav callers
that preserve the native sample rate.

This change converts integer frames and frames requiring sample-rate conversion
to packed float with PyAV's audio resampler while preserving the source channel
layout. Existing float32 frames at the requested sample rate use a direct fast
path. Packed ndarrays are explicitly deinterleaved into (channels, samples)
before concatenation. Keeping resampler output packed is important because
exporting planar frames with many channels, including 7.1 audio, can terminate
the process inside PyAV. Integer PCM is normalized, mono reduction retains the
existing arithmetic-mean behavior, and the resampler is flushed after decoding
so buffered samples are included.

Test Plan

Validation

python -m pytest -q tests/multimodal/media/test_audio.py
python -m pytest -q tests/multimodal/media/test_audio.py tests/multimodal/test_audio.py
pre-commit run --files vllm/multimodal/media/audio.py
git diff --check

Codec matrix

A 30-second, 16 kHz stereo signal was encoded with FFmpeg into six
representative formats and decoded at its native sample rate. Two warm-up runs
were discarded, followed by 12 timed decodes for each unmodified and patched
path. The table reports median decode time.

Test Result

Tests and checks

• Media audio tests: 23 passed
• Media and audio utility tests: 70 passed
• All applicable pre-commit hooks: passed
• git diff --check: passed

Codec correctness and decode time

Codec/container	Old samples	Fixed samples	Old peak	Fixed peak	Old ms	Fixed ms
PCM16/WAV	960,000	480,000	18,022	0.452	16.84	34.56
PCM24/MOV	960,000	480,000	1.181e9	0.452	10.81	28.83
ALAC/M4A	480,000	480,000	9.702e8	0.452	10.72	21.44
AAC/M4A	480,256	480,256	0.452	0.452	15.46	15.50
MP3	480,000	480,000	0.438	0.438	24.47	24.39
FLAC	960,000	480,000	1.181e9	0.452	16.58	34.10

AAC and MP3 already decode as float32 and use a direct fast path, showing no
material regression. Integer formats require an additional format conversion
to produce normalized float32 output; this adds roughly 11-18 ms per 30
seconds of audio in this measurement.

Multi-channel correctness

Mono, stereo, 5.1, and 7.1 PCM inputs were decoded through the final packed
float path. Both mono=True and mono=False completed successfully. For
lossless PCM and FLAC, the deinterleaved per-channel output and arithmetic mono
output matched soundfile exactly, including a random 7.1 signal with distinct
content in every channel.

For the two-second random 7.1 fixture, the unmodified path returned 768,000
interleaved integer-scale samples instead of 96,000 mono samples. The fixed path
returned 96,000 normalized samples and matched the reference sample-for-sample.
Resampling 7.1 PCM from 48 kHz to 16 kHz also matched the reference within
1.2e-7 maximum absolute error. Duration and decoded-memory limits continued
to reject inputs at their configured boundaries.

Qwen2.5-Omni end-to-end validation

Qwen2.5-Omni-3B was run with deterministic decoding. A real recording of the
“Mary had a little lamb” passage was converted to stereo PCM16 WAV. The
unmodified and patched decoders processed the exact same bytes, and both
resulting arrays were sent to the same model instance with the same prompt and
sampling parameters.

The unmodified decoder returned 510,336 samples instead of 255,168 because the
two packed channels were interpreted as one continuous mono stream. Its peak
amplitude was 28,666 instead of 0.787 because signed PCM16 values were cast to
float32 without scaling. The downstream resampling and audio encoder therefore
received a waveform with both the wrong time axis and values far outside the
normal audio range.

Path	Samples	Peak amplitude	Model result
Unmodified	510,336	28,666	Returned an introduction followed by repeated “嗯” tokens until the output limit; none of the spoken English was transcribed
Patched	255,168	0.787	Transcribed the complete recording, including “Mary had a little lamb, its fleece was white as snow...”

The patched waveform matches the soundfile reference numerically. This shows
that the failure is not a cosmetic difference between decoder layouts: the
corrupted array changes a successful speech-recognition request into a fully
incorrect model response.

AI-assisted contribution

This contribution was developed with AI assistance.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

We have dropped pyav support, I don't think this is needed anymore? @Isotr0py

[DarkLight1337]

Sorry I misread the PR. In that case I'll have @Isotr0py review it

[waizuichougou]

@DarkLight1337 . Thanks for checking!