Skip to content

feat(media): actionable errors for unsupported video/audio codecs (OPS-7779) - #11872

Closed
dmitry-tokarev-nv wants to merge 33 commits into
mainfrom
dtokarev/media-codec-error-ux
Closed

feat(media): actionable errors for unsupported video/audio codecs (OPS-7779)#11872
dmitry-tokarev-nv wants to merge 33 commits into
mainfrom
dtokarev/media-codec-error-ux

Conversation

@dmitry-tokarev-nv

@dmitry-tokarev-nv dmitry-tokarev-nv commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Summary

The runtime images ship a VP8/VP9-only media stack — H.264/H.265/AAC and the OpenCV/decord backend decoders are omitted. Today, sending an unsupported video format yields an opaque low-level error (Decoder not found, No module named 'cv2', No module named 'decord'). This adds actionable guidance at each decode entry point so a user knows which decoder is missing and how to enable it:

  • Frontend media decoder (decoders/video.rs): on decode failure, explain that only VP8/VP9 are built in and how to fix — re-encode to VP9 (ffmpeg -i in.mp4 -c:v libvpx-vp9 -an out.webm) or run a codec-enabled image — while preserving the original ffmpeg error for diagnostics.
  • vLLM backend (video_loader.py): catch the lazily-imported OpenCV ImportError and point to pip install opencv-python-headless (or send VP8/VP9, which the frontend decodes).
  • SGLang encode worker: preflight the decord decoder for video requests and point to pip install decord2, instead of a bare No module named 'decord' surfaced from deep inside SGLang.

Stacked on #11836 (which removes the codecs) so it carries the right context; rebase onto main once #11836 lands.

Validation

  • py_compile + ruff format/check (pinned v0.5.2) clean on all changed files.
  • Added tests: the vLLM ImportError guidance path, and the SGLang preflight for both decoder-present and decoder-absent. The existing extract-media-urls tests stay green by stubbing the preflight. Preflight helper logic unit-verified locally.
  • Manual test on colima with a purpose-built VP9-only ffmpeg confirmed the demuxer still names the codec (h264/aac) with no decoder present, so these messages can be specific.
  • The Rust change is a simple map_err wrap; full compile/build is exercised by CI.

Linear: OPS-7779

🤖 Generated with Claude Code


Open in Devin Review

dmitry-tokarev-nv and others added 26 commits July 14, 2026 01:42
…stage (OPS-7665)

Add a compliance scan that fails the image build if a media-codec
library/binary ships outside our in-tree ffmpeg allowlist. We build an
in-tree ffmpeg to a narrow media set; this keeps the finished image aligned
by catching a distro libav*, a wheel-bundled libavcodec, libx264/265, or a
stray/imageio-bundled ffmpeg pulled in by a base-image or dependency bump.

The gate is a filesystem scan (bundled codec .so files don't appear as SBOM
components) plus an optional SBOM version floor, driven by
container/compliance/policy/codec_policy.yaml. Matches under our in-tree
allow_paths pass; a reasoned per-entry exception (e.g. NVIDIA DALI's bundled
libav*, owned by its own team) is logged but does not fail. It runs in the
licenses stage of compliance.Dockerfile, so a violation fails the build.

Unit-tested against the real bundled-library paths; see OPS-7665 for background.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
… allowlist (OPS-7665)

The in-tree ffmpeg restricted encoders/muxers but still shipped ffmpeg's full
default decoder set (291 video + 212 audio decoders) in the libav*.so copied
into every runtime image. Narrow it to the media formats we actually build and
use.

Switch the wheel_builder configure to an allowlist: blanket
--disable-decoders/--disable-demuxers/--disable-parsers/--disable-protocols
plus exactly the components the two real paths need:
- encode CLI (imageio): rawvideo demux/decode over a pipe -> h264_nvenc/vp9
- Rust media-ffmpeg VideoDecoder: VP8/VP9 in mp4/webm/mkv
Image decode does not use ffmpeg (Rust `image` crate), so no still-image
decoders are enabled. HW NVDEC (h264_cuvid/hevc_cuvid) can be re-added if a
decode feature needs it.

Also bump ffmpeg 8.1 -> 8.1.2 (an upstream maintenance release) to pick up
security fixes and trim the decode surface further.

Test fixtures are re-encoded to VP9-in-mp4; filenames and the Rust/Python tests
that consume them are unchanged. Validated by building ffmpeg 8.1.2 with this
exact allowlist and decoding every fixture to the expected frame counts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…m images (OPS-7625)

Upstream TRT-LLM made cv2 an optional video extra
(NVIDIA/TensorRT-LLM#16206) and no longer preinstalls it on main; its
imports were already function-local in 1.3.x, so nothing in the image
needs it at import time. Align the trtllm runtime images with that:
uninstall the package (via the system interpreter -- with VIRTUAL_ENV
set, plain pip targets the venv and exits 0 without touching the
system-site install) and mirror the removal in the pre_runtime
whiteout, since the squash COPY cannot represent deletions. Build
fails if cv2 survives either stage, and a dependencies test guards
against a base-image bump reintroducing it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Remove codec-bearing Python packages inherited from the SGLang base, build the Dynamo runtime without media-ffmpeg, and fail the image build if FFmpeg, H.264, H.265, or AAC libraries remain.

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
The ARM64 Inkling base installs the distribution as decord2, whose wheel bundles FFmpeg, x264, and x265 shared libraries. Remove that package and its residual files before the final codec guard.

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Address review: hoist the `pathlib` import in test_no_opencv.py to module
scope per the repo import-placement rule, and remove the internal ticket
reference from the docstring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Move the optional `packaging.version` import to module scope and narrow the
  fallback to ImportError only, so a malformed version string surfaces (and
  fails the scan) instead of silently degrading to the naive comparator.
- Rename unused unpacked values in the tests (RUF059).
- Drop internal ticket references from the policy/scanner comments.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Scan /tmp and /run: only the virtual kernel filesystems (proc/sys/dev)
  are pruned now, so a codec left in /tmp still gets flagged.
- Fail when an explicitly-supplied --sbom path is missing, instead of
  silently disabling the version-floor gate.
- Flag denied SBOM components that carry no version (cannot prove they
  meet the fixed floor) rather than letting them pass.
- Add the required pytest markers to the test module and cover the three
  behaviors above.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Drop internal ticket references from the wheel_builder + media README comments.
- Make the fixture-regeneration ffmpeg command in the video.rs docstring
  executable by adding the output filename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The TRT-LLM diffusion encode path (imageio -> ffmpeg -c:v h264_nvenc -> mp4)
regressed after the decoder allowlist. The h264 *parser* (not the H.264
decoder) is needed for the mp4 muxer to package the h264_nvenc bitstream, so
add it back. A parser carries no codec implementation, so this does not
reintroduce any software H.264 encode/decode surface; h264_nvenc remains the
NVIDIA hardware encoder (the sanctioned path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The vllm-openai base ships PyAV, opencv, decord, torchcodec, and PyNvVideoCodec,
each bundling its own full ffmpeg with software H.264/H.265/AAC (PyAV and decord
also ship GPL libx264/libx265). Dynamo's vLLM component imports none of them, so
they are unused decode-side dead weight. Uninstall + remove them, with a guard
that fails the build if cv2/av survive.

The in-tree LGPL ffmpeg and imageio-ffmpeg are kept for the omni HW video-encode
path (h264_nvenc, the sanctioned NVIDIA hardware encoder). This drops software
video *decode* from the vLLM images (mirrors the SGLang codec removal).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…python-headless' into dtokarev/ops-7665-codec-compliance
DALI vendors its full ffmpeg stack under nvidia/dali/.libs — libavcodec plus
the libsw*/libpostproc companions. The exception only matched libav*, so DALI's
libswscale tripped the gate and failed the trtllm build. Waive the whole
vendored dir instead, and cover it in the test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The narrow in-tree ffmpeg protocol allowlist enabled only file,pipe. On
ffmpeg 8.x, `ffmpeg -i -` (reading rawvideo frames from stdin, as the
imageio encode path does) resolves stdin through the `fd:` protocol, not
`pipe:`. With fd disabled, opening the stdin input fails immediately with
"Error opening input: Protocol not found. Did you mean file:fd:?", breaking
the video encode path.

Add fd to the protocol allowlist. It is pure file-descriptor I/O and
carries no codec implementation, so it does not widen the media-format
surface. Reproduced and confirmed against a locally built allowlisted
ffmpeg: file,pipe fails to open stdin; file,pipe,fd opens it cleanly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…ages

The vllm and sglang runtime images intentionally omit certain media-decoder
libraries (opencv for vLLM's VideoMediaIO backend decode, decord for SGLang's
video path). The video multimodal serve tests exercise those decode paths, so
without the decoder present they fail at request time ("No module named
'cv2'" / "No module named 'decord'").

Rather than skip that coverage, let a serve config opt into an ephemeral,
test-only decoder install via a reserved env key (DYN_TEST_ONLY_PIP_INSTALL).
common.run_serve_deployment installs the named package(s) into the runtime
container before the server launches, so the worker can import them; the
shipped image is unaffected. Session-deduped and a no-op unless a config sets
the key.

Wired for the three vLLM video topologies (opencv-python-headless) and the two
SGLang video configs (decord2). Verified on faithful codec-stripped rc.11
images: each decode path reproduces the exact failure when stripped and decodes
successfully once the package is installed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
… path

Narrow the in-tree ffmpeg to the media formats we actually ship: video
encode uses VP9 (libvpx_vp9) only. NVENC is no longer enabled and the
nv-codec-headers are no longer fetched, so the build produces no H.264
encoder, decoder, or parser in any framework image. A post-build guard
fails the build if an H.264/H.265/AAC/NVENC/CUVID surface reappears in
the in-tree ffmpeg.

The Python encode paths (the shared common util and the sglang video
generation handler) switch from h264_nvenc to libvpx-vp9 and drop the
libx264 fallback. VP9-in-mp4 is valid and decodes with the existing
VP8/VP9 decoder allowlist.

Validation: the trtllm video_diffusion serve test passes end-to-end with
VP9 encoding (frames -> libvpx-vp9 -> mp4 -> validated response).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com>

Co-authored-by: Dan Gil <dagil-nvidia@users.noreply.github.com>
…guards) + trtllm attribution

Addresses review feedback on the media-codec compliance gate:

- scan_codecs: CodecPolicy.load now fails closed. A missing/misspelled
  deny_globs previously became an empty list, silently passing every
  image. It now validates the policy is a mapping with a non-empty
  deny_globs and that every exception carries glob, reason, and owner.
- codec_policy: add the auditwheel hash-renamed variants
  (libx264-*/libx265-*/libfdk-aac-*/libpostproc-*.so*) so a wheel that
  vendors a renamed codec lib under .libs/ can't bypass the denylist.
- vllm_runtime: the purge guard now checks decord, torchcodec, and
  PyNvVideoCodec in addition to cv2 and av. The uninstall is best-effort
  (|| true), so a survivor would otherwise pass silently.
- native_packages: list trtllm-runtime[-efa] under the ffmpeg and libvpx
  entries. The TensorRT-LLM runtime copies the in-tree ffmpeg + libvpx
  from the build stage, so its generated legal notices must attribute
  them too.

Tests: add fail-closed cases (missing/empty deny_globs, non-mapping doc,
exception missing owner) and a case proving a hash-renamed libx264 is denied.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Pre-commit (ruff-format) flagged the deny_globs validation message as
exceeding the line length; wrap it to match the sibling raise statements.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The post-build guard failed the image build on 'aac_adtstoasc' listed by
`ffmpeg -bsfs`. Bitstream filters (aac_adtstoasc, h264_mp4toannexb, ...)
only reframe an already-encoded stream and are pulled in as mov/mp4 muxer
dependencies; they carry no codec implementation, so matching their names
is a false positive. Guard the implementation-carrying surfaces only
(encoders/decoders/parsers), which are VP9-only by construction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv requested review from a team as code owners July 18, 2026 12:49
The vLLM-Omni video output path encoded via diffusers.export_to_video,
whose imageio backend infers the codec from the .mp4 extension and picks
H.264 (libx264). The runtime image ships no H.264 encoder, so video
generation failed at the encode step with "No valid H.264 encoder was
found with the ffmpeg installation" (seen on the omni_i2v/omni_t2v serve
tests).

Encode with the in-tree VP9 (libvpx-vp9) encoder instead, via the shared
encode_to_video_bytes helper already used by the TRT-LLM video handler.
Omni frames are PIL images, so frames_to_numpy converts them to the
(T,H,W,C) uint8 array the helper expects. VP9-in-mp4 is valid and decodes
with the VP8/VP9 allowlist; the response still advertises output_format=mp4.

Update the DiffusionFormatter video tests to patch the new encode path
(frames_to_numpy + encode_to_video_bytes) instead of export_to_video.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/media-codec-error-ux branch from 39df4a7 to f40f06d Compare July 20, 2026 21:13
The SGLang video-generation handler encodes output with libvpx-vp9, but the
SGLang image shipped no ffmpeg at all, so encode failed with "unknown file
extension" (imageio had no FFMPEG plugin/binary). Aggregated and LLM paths
never needed it, which is why only video generation regressed.

Ship the same VP9-only in-tree ffmpeg the vLLM and TRT-LLM images already carry:
- wheel_builder builds the ffmpeg for SGLang too (VP9 only; no H.264/H.265/AAC).
- sglang_runtime copies the CLI + libav*/libvpx from wheel_builder (CUDA),
  installs imageio-ffmpeg from source (no bundled binary), and points
  IMAGEIO_FFMPEG_EXE at the in-tree CLI. A build guard asserts the shipped
  ffmpeg exposes VP9 and no H.264/H.265/AAC/NVENC encoder.
- The final file guard now allows the VP9 ffmpeg while still failing on any
  H.264/H.265/AAC implementation; the media-codec scan already re-permits the
  in-tree ffmpeg only under /usr/local and catches stray third-party libav*.
- native_packages records ffmpeg + libvpx for the sglang images; the context.yaml
  note is corrected to match.

Validated on gpu-ts: the shipped ffmpeg + imageio_ffmpeg make VP9 encode succeed
(codec=vp9).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/ops-7665-codec-compliance branch from 07c6ea4 to 53982bc Compare July 20, 2026 23:56
@dmitry-tokarev-nv
dmitry-tokarev-nv requested a review from a team as a code owner July 20, 2026 23:56
dmitry-tokarev-nv and others added 2 commits July 21, 2026 01:58
The video-output encode path had no pre_merge test that exercised the shipped
ffmpeg: the tests that touch it mock imageio/encode_to_video_bytes, and the
real-encode serve tests are post_merge/nightly. That let two failures ship
green through the PR pipeline (a removed H.264 default; an image with no
encoder).

Add a pre_merge, gpu_0 test per backend that drives the real handler encode
(DiffusionFormatter._encode_video / SGLang _frames_to_video / trtllm
encode_to_video_bytes) through the baked ffmpeg and asserts the output stream
is vp9 — catching both "no encoder" and "wrong codec". Add the same VP9 build
guard to the vLLM and TRT-LLM runtimes (SGLang's rides with its ffmpeg commit).

Validated on gpu-ts: all three backends encode vp9; guards pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Flip the vLLM/SGLang/TRT-LLM Test + Multi-GPU lanes to
"(pre_merge or post_merge)" so the media-codec work gets full serve-test
coverage (incl. omni_i2v/omni_t2v video-output encode) before merge.

Marked TEMP-11836; revert to pre_merge-only before merging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/ops-7665-codec-compliance branch from 53982bc to 3c8eabc Compare July 20, 2026 23:59
dmitry-tokarev-nv and others added 3 commits July 21, 2026 01:59
The runtime images build a royalty-free media stack — VP8/VP9 only, with
no H.264/H.265/AAC and without the OpenCV/decord backend decoders. Before
this change a user who sent an unsupported format got an opaque low-level
error. Give actionable guidance at each decode entry point:

- Frontend media decoder (Rust): on decode failure, explain that only
  VP8/VP9 are built in and how to fix (re-encode to VP9, or run an image
  with the codec), preserving the original ffmpeg error for diagnostics.
- vLLM backend video loader: catch the lazy OpenCV ImportError and point
  the operator to `pip install opencv-python-headless` (or send VP8/VP9).
- SGLang encode worker: preflight the `decord` decoder for video requests
  and point to `pip install decord2` instead of a bare
  "No module named 'decord'" surfaced from deep inside SGLang.

Tests cover the vLLM ImportError guidance and the SGLang preflight
(decoder present and absent); the existing extract-media-urls tests stay
green by stubbing the preflight.

Stacked on the media-codec-compliance branch (#11836) since that is what
removes the codecs; rebase onto main once #11836 lands.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Document which image/video codecs the shipped images decode (VP8/VP9
video; common still images; OpenCV/decord backend decoders omitted) and
how to enable others — re-encode to VP9, or install the backend decoder
(`opencv-python-headless` for vLLM, `decord2` for SGLang). Mirrors the
actionable runtime error messages. Wired into the Multimodal nav and
linked from the multimodal README support matrix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Review feedback on the actionable-error messages:
- sglang encode worker: run the decord preflight only AFTER media-URL
  validation, so a malformed/Decoded-variant video item fails with the
  existing payload error rather than a spurious 'install decord'.
- vLLM video_loader: the backend URL path decodes every video_url (any
  codec) via OpenCV, so VP8/VP9 does not avoid it — point to installing
  OpenCV, or enabling frontend decoding (which sends decoded frames).
- frontend video.rs: Decoder::new can fail on malformed input too; keep
  the original ffmpeg error prominent and frame the VP8/VP9 guidance
  conditionally so bad payloads aren't misreported as a codec problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/media-codec-error-ux branch from f40f06d to 968d8cc Compare July 21, 2026 00:00
@dmitry-tokarev-nv dmitry-tokarev-nv changed the title feat(media): actionable errors for unsupported video/audio codecs feat(media): actionable errors for unsupported video/audio codecs (OPS-7779) Jul 22, 2026
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/ops-7665-codec-compliance branch from d79e13b to cb0063b Compare July 29, 2026 15:40
@dmitry-tokarev-nv
dmitry-tokarev-nv requested review from a team as code owners July 29, 2026 15:40
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/ops-7665-codec-compliance branch 3 times, most recently from 116728e to ea49322 Compare July 30, 2026 12:03
@dmitry-tokarev-nv
dmitry-tokarev-nv force-pushed the dtokarev/ops-7665-codec-compliance branch from 9aaf999 to 6e29800 Compare July 31, 2026 03:20
Base automatically changed from dtokarev/ops-7665-codec-compliance to main August 5, 2026 15:31
@dmitry-tokarev-nv

Copy link
Copy Markdown
Contributor Author

Superseded by #12725, which reimplements this against the post-codec-removal / post-NVDEC / post-#12051 tree. Everything this PR touched was rewritten since July: the blanket decord preflight would now break H.264/H.265 requests that NVDEC serves without decord, the recommended installs must use the validated version bounds from #12051, and the docs page is covered by video-decode-gpu-requirements.md plus additional-media-decoders.md. The three review findings raised here are carried into #12725 as design constraints (preflight after payload validation; no implication that VP8/VP9 avoids the backend decoder; original error kept prominent with conditional guidance).

@dmitry-tokarev-nv
dmitry-tokarev-nv deleted the dtokarev/media-codec-error-ux branch August 10, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` multimodal size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants