chore: replace decord2 with torchvision/torchaudio for media loading by kajalj22 · Pull Request #3107 · NVIDIA-NeMo/RL

kajalj22 · 2026-07-07T21:54:01Z

Summary

Replaces `decord2` with `torchvision`/`torchaudio` and removes bundled codec libraries from the shipped container.

Video/audio decoding

Replace `load_video(backend="decord")` with `load_video(backend="torchvision")` in `multimodal_utils.py` and `processors.py`
Replace `decord.AudioReader` fallback with `torchaudio.load` + `torchaudio.functional.resample` in `multimodal_utils.py`
Replace `import decord` in `datasets/utils.py` (`load_audio_from_file`) with `torchaudio`

Container changes

Block `decord2` via `"decord2; sys_platform == 'never'"` uv override — removes its bundled codec libs from the container (pulled transitively by `sglang → qwen-vl-utils[decord]` on aarch64)
Block `av` (PyAV) via `"av; sys_platform == 'never'"` uv override — removes its bundled codec libs (only needed by MB's optional WAN diffusion path, installed at test time via `scripts/install_diffusion_deps.sh`)
Remove `torchaudio` and `torchcodec` from `[project.dependencies]` and drop the `pytorch-cu130` sources entry — the container receives the PyPI torchaudio stub (no bundled FFmpeg) via sglang/vllm; the full pytorch-cu130 build is not shipped
Add `torchaudio.*` to `pyrefly.toml` `replace-imports-with-any` so lazy imports are not flagged by the type checker

Audio install script

Add `scripts/install_audio_deps.sh` — installs `torchaudio==2.11.0` (pytorch-cu130 build) + `torchcodec` before audio tests. Idempotent: checks for bundled FFmpeg libs rather than just importability. Uses `--reinstall-package torchaudio` to upgrade from the PyPI stub.
Single hook in `tests/test_suites/llm/common.env` with a name-based check (`[[ "${BASH_SOURCE[1]:-}" =~ (omni|audio|avqa) ]]`) — covers all audio/omni VLM test scripts via the shared `common.env` (which `vlm/common.env` symlinks to); also added directly to `tests/functional/audio_grpo_megatron.sh`, `eval_audio.sh`, and `eval_daily_omni.sh`. LLM tests are unaffected.
Add prerequisite note to `docs/guides/grpo-audio.md`

Motivation

`decord2` and `av` (PyAV) bundle codec libraries as hard dynamic link dependencies that cannot be stripped without breaking the import.

Supersedes #2922, which attempted to delete codec `.so` files at Docker build time but broke `import decord`.

Test plan

CI pipeline 57825799: all 10 tests passed (~2h 52m)
Audio/omni tests self-install torchaudio via `scripts/install_audio_deps.sh` before running
LLM tests unaffected: `torchaudio` is lazily imported inside audio functions, never called by LLM training paths

🤖 Generated with Claude Code

decord2 bundles libx264, libx265, libopenh264, and libmp3lame as hard dynamic dependencies of its core shared library. These codec libs carry MPEG LA / HEVC Advance royalty obligations on distribution. Replace the two decord usages in multimodal_utils.py: - Video loading: load_video backend="decord" → backend="torchvision" (torchvision==0.26.0 is already a project dependency) - Audio fallback: decord.AudioReader → torchaudio.load + functional.resample (torchaudio==2.11.0 is already a project dependency) Remove decord2 from pyproject.toml dependencies. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

copy-pr-bot · 2026-07-07T21:54:04Z

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 · 2026-07-07T22:30:14Z

/ok to test 25b19e2

…ision

- Replace decord.AudioReader in load_audio_from_file with torchaudio - Add torchaudio.* to pyrefly replace-imports-with-any to suppress import-error for the lazy torchaudio import in multimodal_utils.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 · 2026-07-09T14:54:12Z

/ok to test 029b9ee

av bundles libx264, libx265, libopenh264, and libmp3lame — the same royalty-obligating codec libs we removed with decord2. It is pulled in transitively by qwen-vl-utils (a megatron-bridge dep) but is only actually needed by megatron-bridge's optional WAN diffusion path, which installs it at test time via scripts/install_diffusion_deps.sh. Add "av; sys_platform == 'never'" to override-dependencies, mirroring the same approach already taken in Megatron-Bridge's own pyproject.toml (commit f4b34b4). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

…d2 via uv override - Switch processors.py video loading from backend="decord" to backend="torchvision" (missed in the initial decord→torchvision pass) - Add decord2 to override-dependencies with sys_platform=='never' so sglang's transitive pull of qwen-vl-utils[decord] on aarch64 Linux doesn't land it in the container Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

copy-pr-bot · 2026-07-09T16:24:31Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…ision

kajalj22 · 2026-07-09T21:38:53Z

/ok to test 3e21f62

torchaudio was only in override-dependencies (which pins version but doesn't install the package). Moving it to [project.dependencies] alongside torchvision so it's actually installed in the container. Without this, load_audio_from_file raises ModuleNotFoundError on torchaudio. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

torchaudio 2.11.0 defaults to torchcodec as its first audio backend. torchcodec's native .so files were already present in the container (via ray worker venvs) but the Python package wasn't in the main venv, causing ModuleNotFoundError when load_audio_from_file is called from the dataloader. torchcodec links against system FFmpeg (LGPL-only) rather than bundling its own copy, so it carries no royalty-bearing codec libs (no libx264/ libx265/libmp3lame). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

…ll script - Remove torchaudio and torchcodec from [project.dependencies] and drop the pytorch-cu130 sources entry so the container receives only the PyPI torchaudio stub (~1.7 MB, no bundled FFmpeg) via sglang/vllm transitive deps. The pytorch-cu130 build (bundled FFmpeg with H.264/AAC decoders) is no longer shipped. - Add scripts/install_audio_deps.sh: installs torchaudio==2.11.0 (pytorch-cu130, with FFmpeg) + torchcodec at test time. Uses --reinstall-package torchaudio to upgrade from the PyPI stub. Check is idempotent (looks for bundled libav* in torchaudio/lib/, not just import success, so the PyPI stub does not falsely suppress the install). - Hook install_audio_deps.sh into all audio/omni test scripts (tests/test_suites/vlm/*omni*.sh, tests/functional/audio_grpo_megatron.sh, eval_audio.sh, eval_daily_omni.sh). LLM tests are unaffected. - Add prerequisite callout to docs/guides/grpo-audio.md. Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 · 2026-07-16T00:37:12Z

/ok to test 2fe5480

yuekaizhang · 2026-07-16T05:23:36Z

@kajalj22 Could you help update the audio dependency install methods in https://github.com/NVIDIA-NeMo/RL/blob/main/docs/guides/grpo-audio-visual.md and the related nightly tests?

… guide Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

…ision

kajalj22 · 2026-07-16T07:16:09Z

/ok to test bf18ead

Also fix two stale comments in pyproject.toml: - drop torchaudio from the uv.sources index comment - update script path reference from scripts/ to tools/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 · 2026-07-16T21:43:50Z

/ok to test 3a0d641

chtruong814 · 2026-07-17T22:25:02Z

/ok to test

copy-pr-bot · 2026-07-17T22:25:05Z

/ok to test

@chtruong814, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

chtruong814 · 2026-07-17T23:04:37Z

/ok to test 0af12fe

…ing (#3107) (#3277) Signed-off-by: Kajal Jain <kajalj@nvidia.com> Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: kajalj22 <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…3107) Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

kajalj22 requested review from a team as code owners July 7, 2026 21:54

kajalj22 temporarily deployed to public July 7, 2026 21:54 — with GitHub Actions Inactive

kajalj22 temporarily deployed to public July 7, 2026 21:57 — with GitHub Actions Inactive

chore: update uv.lock to remove decord2

aff511d

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 temporarily deployed to public July 7, 2026 21:59 — with GitHub Actions Inactive

kajalj22 temporarily deployed to public July 7, 2026 22:00 — with GitHub Actions Inactive

kajalj22 temporarily deployed to public July 7, 2026 22:04 — with GitHub Actions Inactive

kajalj22 added CI:L1 Run doctests, unit tests, and functional tests r0.7.0 labels Jul 7, 2026

chore: merge main into replace-decord-torchvision

25b19e2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 and others added 2 commits July 9, 2026 09:45

Merge remote-tracking branch 'origin/main' into replace-decord-torchv…

2445e54

…ision

kajalj22 and others added 2 commits July 9, 2026 10:14

Merge remote-tracking branch 'origin/main' into replace-decord-torchv…

3e21f62

…ision

chtruong814 previously approved these changes Jul 9, 2026

View reviewed changes

kajalj22 dismissed chtruong814’s stale review via e52aac7 July 10, 2026 21:31

kajalj22 and others added 2 commits July 11, 2026 22:35

kajalj22 requested a review from a team as a code owner July 13, 2026 19:47

kajalj22 requested a review from chtruong814 July 16, 2026 03:19

chtruong814 previously approved these changes Jul 16, 2026

View reviewed changes

docs: add audio/video dependency install section to grpo-audio-visual…

6204d56

… guide Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Kajal Jain <kajalj@nvidia.com>

kajalj22 dismissed chtruong814’s stale review via 6204d56 July 16, 2026 07:14

Merge remote-tracking branch 'origin/main' into replace-decord-torchv…

bf18ead

…ision

kajalj22 enabled auto-merge (squash) July 16, 2026 07:38

kajalj22 requested a review from chtruong814 July 16, 2026 07:38

terrykong reviewed Jul 16, 2026

View reviewed changes

Comment thread tools/install_audio_deps.sh

terrykong reviewed Jul 16, 2026

View reviewed changes

Comment thread tools/install_audio_deps.sh

yuki-97 reviewed Jul 16, 2026

View reviewed changes

Comment thread pyproject.toml

kajalj22 mentioned this pull request Jul 16, 2026

Make install_audio_deps.sh robust to torch version upgrades #3254

Closed

kajalj22 requested review from terrykong and yuki-97 July 16, 2026 21:41

terrykong approved these changes Jul 16, 2026

View reviewed changes

Merge branch 'main' into replace-decord-torchvision

3a0d641

Merge branch 'main' into replace-decord-torchvision

0af12fe

kajalj22 merged commit 52be987 into main Jul 18, 2026
176 of 179 checks passed

kajalj22 deleted the replace-decord-torchvision branch July 18, 2026 01:46

chtruong814 mentioned this pull request Jul 18, 2026

cp: chore: replace decord2 with torchvision/torchaudio for media loading (#3107) #3277

Merged

4 tasks

ashors1 pushed a commit that referenced this pull request Jul 22, 2026

chore: replace decord2 with torchvision/torchaudio for media loading (#…

be9f88f

…3107) Signed-off-by: Kajal Jain <kajalj@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

anubhutivyas mentioned this pull request Aug 6, 2026

chore(customizer): fix CVEs in RL image NVIDIA-NeMo/nemo-platform#1150

Merged

15 tasks

Uh oh!

Conversation

kajalj22 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Video/audio decoding

Container changes

Audio install script

Motivation

Test plan

Uh oh!

copy-pr-bot Bot commented Jul 7, 2026

Uh oh!

kajalj22 commented Jul 7, 2026

Uh oh!

kajalj22 commented Jul 9, 2026

Uh oh!

copy-pr-bot Bot commented Jul 9, 2026

Uh oh!

kajalj22 commented Jul 9, 2026

Uh oh!

kajalj22 commented Jul 16, 2026

Uh oh!

yuekaizhang commented Jul 16, 2026

Uh oh!

kajalj22 commented Jul 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kajalj22 commented Jul 16, 2026

Uh oh!

chtruong814 commented Jul 17, 2026

Uh oh!

copy-pr-bot Bot commented Jul 17, 2026

Uh oh!

chtruong814 commented Jul 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kajalj22 commented Jul 7, 2026 •

edited

Loading