Repository navigation
[Bugfix][Multimodal] Normalize single-channel audio to 1D - #56691
DarkLight1337 merged 6 commits into
Conversation
Return a 1D waveform for mono inputs with an explicit channel axis so Whisper preprocessing accepts them and audio sample counts stay correct. Extend the existing NumPy, Torch and parser regressions for these layouts. Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Ylean <61880605+Woolgathererer@users.noreply.github.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
mergify
Bot
added
multi-modality
|
Does this issue occur for other processors? If not, it might be better to perform this conversion for WhisperProcessor only |
Thanks for the suggestion! I checked Qwen2-Audio shares Since |
|
@hmellor is this intentional for HF audio processors? |
@DarkLight1337 Following up on the processor-scope question: I checked both Whisper and Qwen2Audio. Both use the shared audio normalization path, and both fail on mono inputs shaped as (1, N) or (N, 1) before this fix. The patch only canonicalizes target_channels=1 to a 1D waveform; it does not change passthrough audio or multi-channel reduction behavior. Since normalize_audio already documents 1D output for mono targets, would you be comfortable approving the shared fix? If any processor intentionally requires a 2D mono input, please point it out so I can account for that case. |
It seems that most models fail in one way or another when receiving mono audio in a 2D array. Given this, I'd suggest making the fix global. I'll work with @eustlb to more consistently handle this on the Transformers side
|
|
/ci run |
|
❌ This PR is 1 commit behind upstream |
|
/ci run --allow-stale |
|
✅ Triggered Buildkite CI #92137 for commit
|
DarkLight1337
added
the
ready
|
✅ @Woolgathererer, CI is now available for this PR.
|
|
Pre-commit is failing on main, trying to fix it now... |
|
/ci run |
|
✅ Triggered Buildkite CI #92157 for commit |
|
Hi @Woolgathererer, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
The current pre-commit failure looks relevant now, please fix it. |
|
/ci run |
|
✅ Triggered Buildkite CI #92781 for commit |
|
Hi @Woolgathererer, the pre-commit checks have failed. Please run: uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, |
|
Please fix pre-commit |
CI selector (shadow): 18 test steps (47 jobs) instead of 54 (70 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (18)
Would skip (today's rules run them) (44)
Would add (today's rules do not run them) (8)
AMD mirrors: would skip (38)
AMD mirrors: would add (6)
2 changed files · base |
sry, on the way |
Signed-off-by: Ylean <61880605+Woolgathererer@users.noreply.github.com>
Head branch was pushed to by a user without write access
|
/ci run |
|
❌ This PR is 1 commit behind upstream |
|
/ci run |
|
✅ Triggered Buildkite CI #92783 for commit |
…ct#56691) Signed-off-by: Ylean <61880605+Woolgathererer@users.noreply.github.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
5 participants
Purpose
Mono audio with an explicit channel dimension can fail in Whisper preprocessing.
For example, passing a
(1, 16000)waveform at 16 kHz through the offline audiotuple path raises
ValueError: Failed to apply WhisperProcessor, while the samesamples in a
(16000,)array work.normalize_audioreturns early when the input channel count already matchesthe target, leaving a 2D array for mono input. This also makes
AudioProcessorItems.get_audio_length()report 1 instead of 16000.Return the only channel as a 1D waveform when the target is mono. Extend the
existing NumPy, Torch and parser tests to cover both channel layouts and the
single-sample case.
This reproduces without resampling. #47693 changes the order of normalization
and resampling, #56023 addresses unsupported multi-channel reduction, and
#51022 optimizes downmixing. Their current patches leave this early return
unchanged.
Test Plan
Run the audio and parser tests on CPU:
Also compare the same one-second WAV in
(16000,),(1, 16000)and(16000, 1)layouts through vLLM's Whisper multimodal processor, using the official
openai/whisper-tinyconfiguration, tokenizer and feature extractor.Test Result
fa1b3b1922plus this patch:71 passed, 1 skipped, 8 deselected.
three layouts produce exactly equal features, prompt tokens and placeholder
ranges. Feature shape is
(1, 80, 3000)and parsed audio length is 16000.The local environment has no matching torchaudio build. An earlier full audio
test run had 55 passes and 7 failures due to that missing dependency; the
filtered run excludes those cases and the default-backend assertion. The
skipped parser test requires CUDA. Whisper validation covers CPU preprocessing;
model generation and accuracy evaluation have not been run.
AI assistance was used, and I reviewed all the changes and tests.