Repository navigation
[TTS][API server] Read OmniVoice and Chatterbox reference audio with libsndfile first, and check Chatterbox's at the API - #779
Conversation
…libsndfile first, and check Chatterbox's at the API OmniVoice decoded its cloning reference with torchcodec only, so on a server whose FFmpeg has no audio decoders (the base image mstar-project#639 fixed Model.load_audio for) even a WAV reference failed; its API check used torchcodec too. Chatterbox's adapter never checked ref_audio at all: a bad clip failed only after admission, and a reference up to the data worker's 30-minute cap was accepted where the other voice-clone adapters stop at 60 s. - check_ref_audio gains decoder="soundfile+torchcodec", the base loader's order: libsndfile, then FFmpeg for what it cannot open. - OmniVoice's load_audio uses load_audio_file at the codec rate, and its adapter checks with that mode. - Chatterbox's adapter goes through _resolve_ref_audio with that mode, like the OmniVoice and Qwen3-TTS adapters, so it also takes bare base64. A reference not at the codec rate is now resampled by torchaudio instead of FFmpeg's resampler, so OmniVoice's cloned output can move slightly for it.
Automated reviewOmniVoice's Findings: none. I traced the new check mode against both loaders ( The description covers what was run, with the counts. It also mentions the torchaudio resampling side effect and that speaker similarity wasn't checked on GPU, which is the one open item. Since non-24 kHz references now go through a different resampler, a short before/after listen or a speaker-similarity number on one or two such clips would back up the "differs slightly" claim. Advisory, not a merge gate. Rules: |
Routing Chatterbox's ref_audio through the shared check means bare base64 is now accepted, and a clip over 60 s or below 8 kHz is a 400 instead of being trimmed by the model. Update the adapter docstring, the MAX_REF_AUDIO_SECONDS comment and docs/models.rst to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| seconds = count_sound_file_seconds(path, MAX_REF_AUDIO_SECONDS) | ||
| _require_ref_audio_seconds(seconds) | ||
| try: | ||
| _check_ref_audio_soundfile(path) |
There was a problem hiding this comment.
This only checks the header of the file. If the header is fine but the rest of the file is corrupt, this still passes. Just calling load_audio_file with the max_seconds argument would be better and will keep the check here in sync with the checks that are run later after admission.
Also should add a check to make sure the file is not empty.
A WAV with a valid header and 0 frames passed check_ref_audio in both libsndfile modes: sound_file_seconds reads 0 frames as "not declared", the count then gives 0 s, and only the upper bound was checked.
|
Fixed the empty file case in 5d4e214. The full decode already runs in the model's load_audio after admission, so a corrupt body fails that request there |
…sts create their upload dir 0700 mstar-project#779 checks the reference audio at the API, which saves it through the per-upload dir check; under umask 002 a plain mkdir gives 0775, which that check refuses before the case under test.
…sts create their upload dir 0700 mstar-project#779 checks the reference audio at the API, which saves it through the per-upload dir check; under umask 002 a plain mkdir gives 0775, which that check refuses before the case under test.
Conflicts: - mstar/model/nemotron_duplex/nemotron_duplex_model.py: imports take this branch's ClientChunk with main's KVLabel/LabelLayout/ProcessPromptOutput/Span (#832); process_prompt keeps this branch's body and comment and returns main's ProcessPromptOutput with the kv_layout, beside main's kv_labels and this branch's process_stream_chunk. - test/modular/test_audio_decode_limit.py: this branch's (load, error, match, seconds) rows plus main's OmniVoice row (#779). - test/modular/test_nemotron_duplex_model.py: this branch's tests (audio_in, session cap, appends) kept; every process_prompt result they read goes through .new_input_tensors, as main's #832 returns ProcessPromptOutput. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sts create their upload dir 0700 mstar-project#779 checks the reference audio at the API, which saves it through the per-upload dir check; under umask 002 a plain mkdir gives 0775, which that check refuses before the case under test.
…sts create their upload dir 0700 mstar-project#779 checks the reference audio at the API, which saves it through the per-upload dir check; under umask 002 a plain mkdir gives 0775, which that check refuses before the case under test.
…upload dir 0700 mstar-project#659 added test_voxcpm2_model.py after this branch was written. Its media_dirs fixture made the upload dir with a plain mkdir, which under umask 002 is 0775; the per-upload check refuses that before the case under test, as it did for mstar-project#779's and mstar-project#807's tests.
What does this PR do?
Two small fixes around voice-cloning reference audio:
Model.load_audio). It now usesload_audio_file(libsndfile first, then torchcodec), andcheck_ref_audiogets a matchingsoundfile+torchcodecmode.check_ref_audio, so a bad or very long clip only failed after admission. It now goes through_resolve_ref_audiolike the OmniVoice and Qwen3-TTS adapters, which also means it accepts bare base64.One side effect: a reference that isn't at 24 kHz is now resampled by torchaudio instead of FFmpeg, so OmniVoice output can differ slightly for those.
How was it tested?
test_omnivoice_reads_a_wav_reference_without_ffmpeg(with the video-only FFmpeg stub fromtest_base_load_audio.py): a WAV reference goes through the adapter check andload_audiowithout touching torchcodec. On main the adapter refuses it.test_media_paths.py,test_audio_decode_limit.pyandtest_chatterbox_model.pyfor the new decode order and the Chatterbox check.test_openai_router.py,test_qwen3_tts_reference_audio.pyandtest/omnivoice: 566 passed, 7 skipped (CPU, torch 2.9.1).Checklist
ruff check .passes