[Refactor] Remove dependency on librosa by NickCao · Pull Request #2273 · vllm-project/vllm-omni

NickCao · 2026-03-27T14:25:39Z

Purpose

Replace librosa.load() with vllm.multimodal.media.audio.load_audio() and librosa.resample() with vllm.multimodal.audio.resample_audio_resampy(), reusing the functions introduced in vllm#37058.

Test Plan

# Pytest tests
pytest tests/entrypoints/openai_api/test_serving_speech.py -v
pytest tests/entrypoints/openai_api/test_serving_speech_stream.py -v
pytest tests/model_executor/models/qwen3_tts/test_cuda_graph_decoder.py -v
pytest tests/engine/test_arg_utils.py -v
# CosyVoice3 E2E
python examples/offline_inference/cosyvoice3/verify_e2e_cosyvoice.py \
  --model Fun-CosyVoice3-0.5B \
  --tokenizer Fun-CosyVoice3-0.5B/CosyVoice-BlankEN \
  --audio-path "" \
  --prompt-text "Mary had a little lamb, its fleece was white as snow, and everywhere that Mary went, the lamb was sure to go." \
  --prompt "The quick brown fox jumps over the lazy dog. This is a test of clear speech synthesis."
# Qwen2.5-Omni E2E
python examples/offline_inference/qwen2_5_omni/end2end.py --query-type text
# Speaker embedding extraction
python examples/online_serving/qwen3_tts/speaker_embedding_interpolation.py \
  --model Qwen/Qwen3-TTS-12Hz-0.6B-Base --device cuda \
  extract --audio voice.wav --output embedding.json

Test Result

PASS

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

lishunyang12

looks good overall, couple small things

chatgpt-codex-connector · 2026-04-06T13:15:22Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

NickCao · 2026-04-06T13:15:43Z

main branch is now targeting vllm 0.19.0, this PR can be merged.

NickCao · 2026-04-06T13:37:55Z

Ok it seems the remainder of the librosa functions also have counterparts in torchaudio, I'm expanding the scope of this PR to completely drop the dependency on librosa.

NickCao · 2026-04-06T19:27:58Z

Done, the dependency on librosa is now fully dropped. See also vllm-project/vllm#39079

tzhouam

Please also update the Dockerfiles, thanks.

NickCao · 2026-04-10T01:54:02Z

Please also update the Dockerfiles, thanks.

Done, manually inspected the wheels (for x86_64 linux), they do contain the native libraries.

tzhouam

LGTM

tzhouam · 2026-04-10T07:44:14Z

@gcanlin Please have a look. The CI failure seems to be caused by the main branch instead of this PR.

gcanlin · 2026-04-10T07:49:40Z

It seems the failure is related to this PR?

tzhouam · 2026-04-10T07:51:38Z

It seems the failure is related to this PR?

@gcanlin
similar errors can be found in the main branch nightly tests:
https://buildkite.com/vllm/vllm-omni/builds/6235/steps/canvas

…_audio Signed-off-by: Nick Cao <ncao@redhat.com>

…le_audio_resampy Signed-off-by: Nick Cao <ncao@redhat.com>

…scale_fbanks Signed-off-by: Nick Cao <ncao@redhat.com>

Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Nick Cao <ncao@redhat.com>

Uses of librosa has been replaced with wrapper functions from vllm (using soundfile and pyav internally). Both soundfile and pyav wheels have the native libraries bundled (libsndfile and ffmpeg), thus no additional installation is required on the host, making these docs outdated and misleading. Signed-off-by: Nick Cao <ncao@redhat.com>

Signed-off-by: Nick Cao <ncao@redhat.com>

These libraries are already bundled in the pyav and soundfile wheels Signed-off-by: Nick Cao <ncao@redhat.com>

Signed-off-by: Nick Cao <ncao@redhat.com>

Isotr0py

LGTM. Thanks!

linyueqian

LGTM

gcanlin · 2026-04-12T15:04:37Z

    apt-get install -y --no-install-recommends --fix-missing \
    curl \
    espeak-ng \
-    ffmpeg \


@xuechendi Could you check whether XPU have the same issue #2708?

Thanks, LGTM

Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Claude <noreply@anthropic.com>

lishunyang12 reviewed Apr 2, 2026

View reviewed changes

Comment thread vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py Outdated

Comment thread examples/online_serving/qwen3_tts/speaker_embedding_interpolation.py

NickCao force-pushed the feature/replace-librosa-with-resampy branch 2 times, most recently from 1180487 to 4ffbb8f Compare April 6, 2026 13:15

NickCao marked this pull request as ready for review April 6, 2026 13:15

NickCao requested a review from hsliuustc0106 as a code owner April 6, 2026 13:15

NickCao requested a review from lishunyang12 April 6, 2026 13:15

NickCao force-pushed the feature/replace-librosa-with-resampy branch from 4ffbb8f to 008c0fc Compare April 6, 2026 19:26

NickCao changed the title ~~[Refactor] Replace librosa.load/resample with upstream vllm audio utilities~~ [Refactor] Remove dependency on librosa Apr 6, 2026

NickCao force-pushed the feature/replace-librosa-with-resampy branch 2 times, most recently from 2d7454e to c1a598e Compare April 7, 2026 14:08

tzhouam added the ready label to trigger buildkite CI label Apr 9, 2026

tzhouam self-requested a review April 9, 2026 07:30

tzhouam requested changes Apr 10, 2026

View reviewed changes

NickCao force-pushed the feature/replace-librosa-with-resampy branch from df97f2b to 8204fe1 Compare April 10, 2026 01:53

tzhouam self-requested a review April 10, 2026 03:00

tzhouam approved these changes Apr 10, 2026

View reviewed changes

tzhouam added the nightly-test label to trigger buildkite nightly test CI label Apr 10, 2026

tzhouam requested a review from gcanlin April 10, 2026 07:43

gcanlin requested review from DarkLight1337, Gaohan123, Isotr0py and linyueqian April 10, 2026 07:50

gcanlin reviewed Apr 10, 2026

View reviewed changes

Comment thread vllm_omni/utils/audio.py

NickCao and others added 8 commits April 10, 2026 08:30

[Refactor] Replace librosa.load with vllm.multimodal.media.audio.load…

78bcebf

…_audio Signed-off-by: Nick Cao <ncao@redhat.com>

[Refactor] Replace librosa.resample with vllm.multimodal.audio.resamp…

6ce346e

…le_audio_resampy Signed-off-by: Nick Cao <ncao@redhat.com>

[Refactor] Replace librosa.filters.mel with torchaudio.functional.mel…

2359b5a

…scale_fbanks Signed-off-by: Nick Cao <ncao@redhat.com>

[Refactor] Replace librosa.effects.time_stretch with torchaudio

292a2f0

Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Nick Cao <ncao@redhat.com>

[Refactor] Drop dependency on now unused librosa

67fe193

Signed-off-by: Nick Cao <ncao@redhat.com>

[Refactor] Drop unused ffmpeg and libsndfile1 from Dockerfile

1c68c42

These libraries are already bundled in the pyav and soundfile wheels Signed-off-by: Nick Cao <ncao@redhat.com>

[Test] Add unit test for mel_filter_bank audio utility

2560d54

Signed-off-by: Nick Cao <ncao@redhat.com>

NickCao force-pushed the feature/replace-librosa-with-resampy branch from f2b6b6b to 2560d54 Compare April 10, 2026 12:38

Isotr0py reviewed Apr 10, 2026

View reviewed changes

Comment thread docker/Dockerfile.cuda

Isotr0py approved these changes Apr 10, 2026

View reviewed changes

gcanlin removed the nightly-test label to trigger buildkite nightly test CI label Apr 10, 2026

linyueqian approved these changes Apr 10, 2026

View reviewed changes

Isotr0py merged commit 2bc183f into vllm-project:main Apr 10, 2026
8 checks passed

tjtanaa mentioned this pull request Apr 12, 2026

[ROCm] [CI] Fix environment issue #2708

Merged

5 tasks

gcanlin reviewed Apr 12, 2026

View reviewed changes

daixinning pushed a commit to daixinning/vllm-omni that referenced this pull request Apr 13, 2026

[Refactor] Remove dependency on librosa (vllm-project#2273)

748337b

Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Claude <noreply@anthropic.com>

alex-jw-brooks mentioned this pull request Apr 15, 2026

[Bug]: qwen_omni_utils this module does not exist #2800

Closed

1 task

NickCao mentioned this pull request Apr 21, 2026

[Bugfix] treewide: drop references to librosa #2996

Merged

5 tasks

Conversation

NickCao commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

lishunyang12 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Apr 6, 2026

Uh oh!

NickCao commented Apr 6, 2026

Uh oh!

NickCao commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NickCao commented Apr 6, 2026

Uh oh!

tzhouam left a comment

Choose a reason for hiding this comment

Uh oh!

NickCao commented Apr 10, 2026

Uh oh!

tzhouam left a comment

Choose a reason for hiding this comment

Uh oh!

tzhouam commented Apr 10, 2026

Uh oh!

gcanlin commented Apr 10, 2026

Uh oh!

tzhouam commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Isotr0py left a comment

Choose a reason for hiding this comment

Uh oh!

linyueqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gcanlin Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

xuechendi Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

NickCao commented Mar 27, 2026 •

edited

Loading

NickCao commented Apr 6, 2026 •

edited

Loading

tzhouam commented Apr 10, 2026 •

edited

Loading