[arun-elr]

Fixes #534

What does this PR do?

In the Dynamo base image, torchcodec loads the base's video-only FFmpeg 8 before the full FFmpeg 6.1 the Dockerfile installs, so every audio upload fails to decode ("Invalid data found when processing input"). A host without FFmpeg's shared libraries fails the same way today. Model.load_audio now goes through the shared load_audio_file: libsndfile (soundfile) first, torchcodec for the containers libsndfile cannot open. WAV, FLAC, OGG and MP3 no longer depend on the FFmpeg build.

• Decode path. load_audio in mstar/model/base.py decodes via load_audio_file: soundfile first, torchcodec second, torchaudio resample, channel mean. [API server][Model][Security] Refuse audio sample rates whose resampling kernel is unbounded #609's sample-rate bound in load_audio_file refuses a crafted header rate before torchaudio builds its kernel, which FFmpeg's resampler never needed on this path.
• CI. pip install soundfile in CPU Core (.github/workflows/ci.yml): the new tests, and 26 cases in test_media_paths.py, skip without it.
• Test. test/modular/test_base_load_audio.py: a torchcodec stand-in that fails as FFmpeg 8 does; 4 containers at 16 and 24 kHz; a crafted 100,003 or 48,001 Hz header refused before torchaudio resamples it.

How was it tested?

On b81a833 (head, on main f17b321), CI's install plus soundfile: CPU Core, the ci.yml paths with -n 4: 5,064 passed, 610 skipped, 5 failed: three LingBot-World shell tests that need ftfy and scipy, which CI installs and this venv lacks (they fail the same way on main here), and two signal tests (test_e2e_validators.py, test_eval_framework.py) that fail only under nohup, which this run used, and pass when run directly; test_base_load_audio.py with test_resample_rate_bound.py: 187 passed. Earlier, on 6b0a375 (the head's own diff, on main 422d785), CPU, CI's install plus soundfile: ruff check . is clean. The CPU core list then: 2660 passed, 203 skipped, 2 failed (test_wan_vae_decode.py; the same 2 fail on main in this environment). python -m pytest test/modular/test_base_load_audio.py: 12 passed (11 fail on main, and the 2 crafted-rate cases fail on 2539338, before the rebase onto #609; the torchcodec fallback case passes on main too). Decode against main's torchcodec path (FFmpeg 6.1, 3 s clips): a 16 kHz mono WAV is sample-identical; details in "Decoded audio against main".

Real-user checks on one H200 (served on b733780, the same commits on #609's previous head dec613d): Whisper-large-v3 served from this PR and from main on 115 uploads (WAV in 10 encodings, FLAC, MP3, OGG, M4A, 16 sample rates, crafted and broken headers, 30 min files). With torchcodec hidden, main answers 103 of them with a 500 and this PR serves 94, as main does with FFmpeg; with FFmpeg both serve 95, 84 of 92 transcripts are identical (the rest differ at resampler level), and three of main's 500s for broken WAVs become a 200 or a 400. Before the rebase onto #609, one 320 KB WAV with a 768,001 Hz header pushed the server process to 192 GB until the kernel killed it.

Not tested

The Dynamo image itself (needs an image build); the stand-in fails the way the issue's FFmpeg 8 does, and an H200 without torchcodec stands in for it in the served check.

Not here: OmniVoice's own load_audio still uses torchcodec only. A FLAC whose header leaves the sample count out is refused by libsndfile's whole-file read and still needs FFmpeg (#708). M4A and WebM in the Dynamo image still need the issue's option 1 (keep FFmpeg 8 off the M* stage's loader path).

Merge order with #644

#644 also rewrites Model.load_audio (a torchcodec decode capped at 30 min). Neither PR is stacked on the other: whichever lands second resolves load_audio to load_audio_file, which #644 caps, so the result keeps libsndfile first and the cap.

Decoded audio against main

Synthetic 3 s clips (220 Hz tone plus noise), decoded to 16 kHz mono by main's AudioDecoder(path, sample_rate=16000, num_channels=1) and by load_audio_file(path, 16000); torch 2.12.1, torchcodec 0.17.0, Ubuntu's FFmpeg 6.1, libsndfile 1.2.2.

input	difference
WAV, 16 kHz, mono	identical (same 48000 samples)
WAV, 24 kHz, mono	35.1 dB SNR (torchaudio's resampler vs FFmpeg's)
WAV, 16 kHz, stereo	main is 1.4142x louder: FFmpeg downmixes to (L+R)/sqrt(2), this PR averages, as librosa (librosa.load(mono=True)) and M*'s other decode paths do
OGG Vorbis, 48 kHz / MP3, 24 kHz, mono	38.4 / 34.0 dB SNR (different decoders and resamplers)

Same sample count in every case.

Tests in detail

test_base_load_audio.py swaps in a torchcodec.decoders.AudioDecoder that raises "Could not open input file ... Invalid data found when processing input" and records its calls:

• WAV, FLAC, OGG Vorbis and MP3, each mono at 16 kHz and stereo at 24 kHz, decode to 16 kHz mono float32 at the right length and level, without a torchcodec call (8 cases).
• A 16 kHz float WAV returns exactly the file's samples.
• An M4A still goes to torchcodec, and its error is unchanged.

Checklist

• ruff check . passes
• Added or updated tests / docs where relevant

AI assistance: written with AI coding agent

[github-actions]

Automated review

This PR changes Model.load_audio to decode through the existing load_audio_file helper. That helper tries libsndfile (soundfile) first, then falls back to torchcodec when soundfile is missing or can't open the container. It downmixes by averaging channels and resamples with torchaudio. The PR also adds a CPU test with a fake torchcodec that fails the way the video-only FFmpeg does, registers that test in test/cpu-core.txt, and installs soundfile in the CPU Core CI job. The code matches the description.

Findings: none.

I checked the following:

• Dynamo images get the new path. Both Dynamo Dockerfiles install .[all] (which includes soundfile) and libsndfile1.
• Installs without soundfile behave as on main. For example, the asr extra doesn't include soundfile; there, the ImportError is caught and decoding falls back to torchcodec as before.
• Resampling needs no new dependency. torchaudio is already a core dependency, so the resample step adds nothing to install.
• Other models are unaffected. The models that override load_audio (Qwen3-TTS, Chatterbox, OmniVoice, Nemotron duplex) don't go through this path.

The description is accurate and states what was tested and what wasn't. It also discloses the one behavior change for current users: stereo input now comes out 1/√2 as loud as FFmpeg's downmix produced.

---

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[merceod]

Checked on klone (1bca43a, same own files as this head). libsndfile first works for the uploads tried, and with #609 under it the 48,001 Hz and 100,003 Hz uploads are refused in 0.02-0.05 s. The first head opened the unbounded-resample hole (#589) on every model through the base loader (a 3 s WAV at 48,001 Hz took 4.5 s and 13 GB, at 100,003 Hz 9.2 s and 26.5 GB); stacking on #609 closed it and the description says so now. CPU tests pass, ruff clean. Nothing to fix.

[NSagan271]

🤖 Automated live test, AI-generated (Claude Code). Not checked line by line by a human. Numbers come from one allocation, so treat small deltas as provisional.

Live test: [Model][Bug fix] Decode request audio with libsndfile first, for a video-only FFmpeg

Works: with FFmpeg's audio decoders removed, WAV, FLAC, OGG and MP3 transcribe on the PR and fail on base.

1x H200, Whisper large-v3, Python frontend, SHM transport · PR head 253933824 vs main dd1baa889 · head now 8f7d86522: rebased onto #609 (sample-rate checks in the same loader) plus a test; retested under the same shim: served Whisper rows match the tested head (3/3), and on CPU Model.load_audio gives the same lengths and RMS; a 2 MHz WAV is now refused (#609). PR tests pass (12); since the retest, head is 4086c0726: same own diff (#609 merged to main)

Claim	Result
Uploads decode without FFmpeg audio decoders	✅ simulated by an LD_PRELOAD shim on the real FFmpeg 5 libraries, so avcodec_find_decoder finds no audio decoder. WAV 16 kHz, WAV 24 kHz stereo, FLAC, OGG and MP3 return 200 with the correct transcript (transcriptions and /generate, 3/3, both heads). Base returns 400 for all 7 formats
M4A and WebM still go to torchcodec	✅ 400 on both arms and both PR heads
Same transcripts with a normal FFmpeg	✅ identical across 7 formats × 3 runs per arm

Other

• Edge cases: under the shim, a FLAC whose header leaves the sample count out still fails on the PR. libsndfile refuses it and the request falls back to torchcodec (CPU check, both PR heads).
• Not simulated: the Dynamo image's real FFmpeg 8 load order and its "Invalid data found" error. Under the simulation, base fails with a 400 rather than a 500.

[arun-elr]

Thanks for the live test.

under the shim, a FLAC whose header leaves the sample count out still fails on the PR. libsndfile refuses it and the request falls back to torchcodec

• Real, outside this PR's change: libsndfile reports such a FLAC as 2**63 - 1 frames, soundfile.read sizes its buffer from that and raises "array is too big", and load_audio_file falls back to torchcodec, which needs FFmpeg's audio decoders. Reproduced on CPU (soundfile 0.14.0, libsndfile 1.2.2, torchcodec hidden). With a full FFmpeg the fallback decodes it, as on main. Tracked in [Model] A FLAC whose header has no sample count still needs FFmpeg in load_audio_file #708: the fix reads such a file in blocks, as [Qwen3-TTS][Bug fix] Count a reference clip whose header has no sample count as it decodes, instead of refusing it #700 does for Qwen3-TTS reference clips, so it fits after [Qwen3-TTS][Bug fix] Count a reference clip whose header has no sample count as it decodes, instead of refusing it #700 and [API server][Security] Bound request audio by its decoded length, not its header #644 rather than here.

Not simulated: the Dynamo image's real FFmpeg 8 load order and its "Invalid data found" error. Under the simulation, base fails with a 400 rather than a 500.

• Noise for the claim: the shim and the PR's own stand-in both take FFmpeg's audio decoding away, which is what this change routes around; the status main returns in that state depends on the error FFmpeg raises, not on this PR. The image build stays under Not tested.

Since the approval: no code change. The description now has a merge-order note for #644 (both rewrite Model.load_audio; whichever lands second keeps load_audio_file, which #644 caps) and links #708. The tested head and the current head have the same own diff.