feat(media): actionable errors for unsupported video/audio codecs (OPS-7779) - #11872
Closed
dmitry-tokarev-nv wants to merge 33 commits into
Closed
feat(media): actionable errors for unsupported video/audio codecs (OPS-7779)#11872dmitry-tokarev-nv wants to merge 33 commits into
dmitry-tokarev-nv wants to merge 33 commits into
Conversation
…stage (OPS-7665) Add a compliance scan that fails the image build if a media-codec library/binary ships outside our in-tree ffmpeg allowlist. We build an in-tree ffmpeg to a narrow media set; this keeps the finished image aligned by catching a distro libav*, a wheel-bundled libavcodec, libx264/265, or a stray/imageio-bundled ffmpeg pulled in by a base-image or dependency bump. The gate is a filesystem scan (bundled codec .so files don't appear as SBOM components) plus an optional SBOM version floor, driven by container/compliance/policy/codec_policy.yaml. Matches under our in-tree allow_paths pass; a reasoned per-entry exception (e.g. NVIDIA DALI's bundled libav*, owned by its own team) is logged but does not fail. It runs in the licenses stage of compliance.Dockerfile, so a violation fails the build. Unit-tested against the real bundled-library paths; see OPS-7665 for background. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
… allowlist (OPS-7665) The in-tree ffmpeg restricted encoders/muxers but still shipped ffmpeg's full default decoder set (291 video + 212 audio decoders) in the libav*.so copied into every runtime image. Narrow it to the media formats we actually build and use. Switch the wheel_builder configure to an allowlist: blanket --disable-decoders/--disable-demuxers/--disable-parsers/--disable-protocols plus exactly the components the two real paths need: - encode CLI (imageio): rawvideo demux/decode over a pipe -> h264_nvenc/vp9 - Rust media-ffmpeg VideoDecoder: VP8/VP9 in mp4/webm/mkv Image decode does not use ffmpeg (Rust `image` crate), so no still-image decoders are enabled. HW NVDEC (h264_cuvid/hevc_cuvid) can be re-added if a decode feature needs it. Also bump ffmpeg 8.1 -> 8.1.2 (an upstream maintenance release) to pick up security fixes and trim the decode surface further. Test fixtures are re-encoded to VP9-in-mp4; filenames and the Rust/Python tests that consume them are unchanged. Validated by building ffmpeg 8.1.2 with this exact allowlist and decoding every fixture to the expected frame counts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…m images (OPS-7625) Upstream TRT-LLM made cv2 an optional video extra (NVIDIA/TensorRT-LLM#16206) and no longer preinstalls it on main; its imports were already function-local in 1.3.x, so nothing in the image needs it at import time. Align the trtllm runtime images with that: uninstall the package (via the system interpreter -- with VIRTUAL_ENV set, plain pip targets the venv and exits 0 without touching the system-site install) and mirror the removal in the pre_runtime whiteout, since the squash COPY cannot represent deletions. Build fails if cv2 survives either stage, and a dependencies test guards against a base-image bump reintroducing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Remove codec-bearing Python packages inherited from the SGLang base, build the Dynamo runtime without media-ffmpeg, and fail the image build if FFmpeg, H.264, H.265, or AAC libraries remain. Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
The ARM64 Inkling base installs the distribution as decord2, whose wheel bundles FFmpeg, x264, and x265 shared libraries. Remove that package and its residual files before the final codec guard. Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Address review: hoist the `pathlib` import in test_no_opencv.py to module scope per the repo import-placement rule, and remove the internal ticket reference from the docstring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Move the optional `packaging.version` import to module scope and narrow the fallback to ImportError only, so a malformed version string surfaces (and fails the scan) instead of silently degrading to the naive comparator. - Rename unused unpacked values in the tests (RUF059). - Drop internal ticket references from the policy/scanner comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Scan /tmp and /run: only the virtual kernel filesystems (proc/sys/dev) are pruned now, so a codec left in /tmp still gets flagged. - Fail when an explicitly-supplied --sbom path is missing, instead of silently disabling the version-floor gate. - Flag denied SBOM components that carry no version (cannot prove they meet the fixed floor) rather than letting them pass. - Add the required pytest markers to the test module and cover the three behaviors above. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
- Drop internal ticket references from the wheel_builder + media README comments. - Make the fixture-regeneration ffmpeg command in the video.rs docstring executable by adding the output filename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The TRT-LLM diffusion encode path (imageio -> ffmpeg -c:v h264_nvenc -> mp4) regressed after the decoder allowlist. The h264 *parser* (not the H.264 decoder) is needed for the mp4 muxer to package the h264_nvenc bitstream, so add it back. A parser carries no codec implementation, so this does not reintroduce any software H.264 encode/decode surface; h264_nvenc remains the NVIDIA hardware encoder (the sanctioned path). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The vllm-openai base ships PyAV, opencv, decord, torchcodec, and PyNvVideoCodec, each bundling its own full ffmpeg with software H.264/H.265/AAC (PyAV and decord also ship GPL libx264/libx265). Dynamo's vLLM component imports none of them, so they are unused decode-side dead weight. Uninstall + remove them, with a guard that fails the build if cv2/av survive. The in-tree LGPL ffmpeg and imageio-ffmpeg are kept for the omni HW video-encode path (h264_nvenc, the sanctioned NVIDIA hardware encoder). This drops software video *decode* from the vLLM images (mirrors the SGLang codec removal). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…python-headless' into dtokarev/ops-7665-codec-compliance
…to dtokarev/ops-7665-codec-compliance
DALI vendors its full ffmpeg stack under nvidia/dali/.libs — libavcodec plus the libsw*/libpostproc companions. The exception only matched libav*, so DALI's libswscale tripped the gate and failed the trtllm build. Waive the whole vendored dir instead, and cover it in the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…to dtokarev/ops-7665-codec-compliance
The narrow in-tree ffmpeg protocol allowlist enabled only file,pipe. On ffmpeg 8.x, `ffmpeg -i -` (reading rawvideo frames from stdin, as the imageio encode path does) resolves stdin through the `fd:` protocol, not `pipe:`. With fd disabled, opening the stdin input fails immediately with "Error opening input: Protocol not found. Did you mean file:fd:?", breaking the video encode path. Add fd to the protocol allowlist. It is pure file-descriptor I/O and carries no codec implementation, so it does not widen the media-format surface. Reproduced and confirmed against a locally built allowlisted ffmpeg: file,pipe fails to open stdin; file,pipe,fd opens it cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
…ages
The vllm and sglang runtime images intentionally omit certain media-decoder
libraries (opencv for vLLM's VideoMediaIO backend decode, decord for SGLang's
video path). The video multimodal serve tests exercise those decode paths, so
without the decoder present they fail at request time ("No module named
'cv2'" / "No module named 'decord'").
Rather than skip that coverage, let a serve config opt into an ephemeral,
test-only decoder install via a reserved env key (DYN_TEST_ONLY_PIP_INSTALL).
common.run_serve_deployment installs the named package(s) into the runtime
container before the server launches, so the worker can import them; the
shipped image is unaffected. Session-deduped and a no-op unless a config sets
the key.
Wired for the three vLLM video topologies (opencv-python-headless) and the two
SGLang video configs (decord2). Verified on faithful codec-stripped rc.11
images: each decode path reproduces the exact failure when stripped and decodes
successfully once the package is installed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
… path Narrow the in-tree ffmpeg to the media formats we actually ship: video encode uses VP9 (libvpx_vp9) only. NVENC is no longer enabled and the nv-codec-headers are no longer fetched, so the build produces no H.264 encoder, decoder, or parser in any framework image. A post-build guard fails the build if an H.264/H.265/AAC/NVENC/CUVID surface reappears in the in-tree ffmpeg. The Python encode paths (the shared common util and the sglang video generation handler) switch from h264_nvenc to libvpx-vp9 and drop the libx264 fallback. VP9-in-mp4 is valid and decodes with the existing VP8/VP9 decoder allowlist. Validation: the trtllm video_diffusion serve test passes end-to-end with VP9 encoding (frames -> libvpx-vp9 -> mp4 -> validated response). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Dan Gil <dagil-nvidia@users.noreply.github.com>
…guards) + trtllm attribution Addresses review feedback on the media-codec compliance gate: - scan_codecs: CodecPolicy.load now fails closed. A missing/misspelled deny_globs previously became an empty list, silently passing every image. It now validates the policy is a mapping with a non-empty deny_globs and that every exception carries glob, reason, and owner. - codec_policy: add the auditwheel hash-renamed variants (libx264-*/libx265-*/libfdk-aac-*/libpostproc-*.so*) so a wheel that vendors a renamed codec lib under .libs/ can't bypass the denylist. - vllm_runtime: the purge guard now checks decord, torchcodec, and PyNvVideoCodec in addition to cv2 and av. The uninstall is best-effort (|| true), so a survivor would otherwise pass silently. - native_packages: list trtllm-runtime[-efa] under the ffmpeg and libvpx entries. The TensorRT-LLM runtime copies the in-tree ffmpeg + libvpx from the build stage, so its generated legal notices must attribute them too. Tests: add fail-closed cases (missing/empty deny_globs, non-mapping doc, exception missing owner) and a case proving a hash-renamed libx264 is denied. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Pre-commit (ruff-format) flagged the deny_globs validation message as exceeding the line length; wrap it to match the sibling raise statements. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The post-build guard failed the image build on 'aac_adtstoasc' listed by `ffmpeg -bsfs`. Bitstream filters (aac_adtstoasc, h264_mp4toannexb, ...) only reframe an already-encoded stream and are pulled in as mov/mp4 muxer dependencies; they carry no codec implementation, so matching their names is a false positive. Guard the implementation-carrying surfaces only (encoders/decoders/parsers), which are VP9-only by construction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
The vLLM-Omni video output path encoded via diffusers.export_to_video, whose imageio backend infers the codec from the .mp4 extension and picks H.264 (libx264). The runtime image ships no H.264 encoder, so video generation failed at the encode step with "No valid H.264 encoder was found with the ffmpeg installation" (seen on the omni_i2v/omni_t2v serve tests). Encode with the in-tree VP9 (libvpx-vp9) encoder instead, via the shared encode_to_video_bytes helper already used by the TRT-LLM video handler. Omni frames are PIL images, so frames_to_numpy converts them to the (T,H,W,C) uint8 array the helper expects. VP9-in-mp4 is valid and decodes with the VP8/VP9 allowlist; the response still advertises output_format=mp4. Update the DiffusionFormatter video tests to patch the new encode path (frames_to_numpy + encode_to_video_bytes) instead of export_to_video. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
dmitry-tokarev-nv
force-pushed
the
dtokarev/media-codec-error-ux
branch
from
July 20, 2026 21:13
39df4a7 to
f40f06d
Compare
The SGLang video-generation handler encodes output with libvpx-vp9, but the SGLang image shipped no ffmpeg at all, so encode failed with "unknown file extension" (imageio had no FFMPEG plugin/binary). Aggregated and LLM paths never needed it, which is why only video generation regressed. Ship the same VP9-only in-tree ffmpeg the vLLM and TRT-LLM images already carry: - wheel_builder builds the ffmpeg for SGLang too (VP9 only; no H.264/H.265/AAC). - sglang_runtime copies the CLI + libav*/libvpx from wheel_builder (CUDA), installs imageio-ffmpeg from source (no bundled binary), and points IMAGEIO_FFMPEG_EXE at the in-tree CLI. A build guard asserts the shipped ffmpeg exposes VP9 and no H.264/H.265/AAC/NVENC encoder. - The final file guard now allows the VP9 ffmpeg while still failing on any H.264/H.265/AAC implementation; the media-codec scan already re-permits the in-tree ffmpeg only under /usr/local and catches stray third-party libav*. - native_packages records ffmpeg + libvpx for the sglang images; the context.yaml note is corrected to match. Validated on gpu-ts: the shipped ffmpeg + imageio_ffmpeg make VP9 encode succeed (codec=vp9). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
dmitry-tokarev-nv
force-pushed
the
dtokarev/ops-7665-codec-compliance
branch
from
July 20, 2026 23:56
07c6ea4 to
53982bc
Compare
The video-output encode path had no pre_merge test that exercised the shipped ffmpeg: the tests that touch it mock imageio/encode_to_video_bytes, and the real-encode serve tests are post_merge/nightly. That let two failures ship green through the PR pipeline (a removed H.264 default; an image with no encoder). Add a pre_merge, gpu_0 test per backend that drives the real handler encode (DiffusionFormatter._encode_video / SGLang _frames_to_video / trtllm encode_to_video_bytes) through the baked ffmpeg and asserts the output stream is vp9 — catching both "no encoder" and "wrong codec". Add the same VP9 build guard to the vLLM and TRT-LLM runtimes (SGLang's rides with its ffmpeg commit). Validated on gpu-ts: all three backends encode vp9; guards pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Flip the vLLM/SGLang/TRT-LLM Test + Multi-GPU lanes to "(pre_merge or post_merge)" so the media-codec work gets full serve-test coverage (incl. omni_i2v/omni_t2v video-output encode) before merge. Marked TEMP-11836; revert to pre_merge-only before merging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
dmitry-tokarev-nv
force-pushed
the
dtokarev/ops-7665-codec-compliance
branch
from
July 20, 2026 23:59
53982bc to
3c8eabc
Compare
The runtime images build a royalty-free media stack — VP8/VP9 only, with no H.264/H.265/AAC and without the OpenCV/decord backend decoders. Before this change a user who sent an unsupported format got an opaque low-level error. Give actionable guidance at each decode entry point: - Frontend media decoder (Rust): on decode failure, explain that only VP8/VP9 are built in and how to fix (re-encode to VP9, or run an image with the codec), preserving the original ffmpeg error for diagnostics. - vLLM backend video loader: catch the lazy OpenCV ImportError and point the operator to `pip install opencv-python-headless` (or send VP8/VP9). - SGLang encode worker: preflight the `decord` decoder for video requests and point to `pip install decord2` instead of a bare "No module named 'decord'" surfaced from deep inside SGLang. Tests cover the vLLM ImportError guidance and the SGLang preflight (decoder present and absent); the existing extract-media-urls tests stay green by stubbing the preflight. Stacked on the media-codec-compliance branch (#11836) since that is what removes the codecs; rebase onto main once #11836 lands. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Document which image/video codecs the shipped images decode (VP8/VP9 video; common still images; OpenCV/decord backend decoders omitted) and how to enable others — re-encode to VP9, or install the backend decoder (`opencv-python-headless` for vLLM, `decord2` for SGLang). Mirrors the actionable runtime error messages. Wired into the Multimodal nav and linked from the multimodal README support matrix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
Review feedback on the actionable-error messages: - sglang encode worker: run the decord preflight only AFTER media-URL validation, so a malformed/Decoded-variant video item fails with the existing payload error rather than a spurious 'install decord'. - vLLM video_loader: the backend URL path decodes every video_url (any codec) via OpenCV, so VP8/VP9 does not avoid it — point to installing OpenCV, or enabling frontend decoding (which sends decoded frames). - frontend video.rs: Decoder::new can fail on malformed input too; keep the original ffmpeg error prominent and frame the VP8/VP9 guidance conditionally so bad payloads aren't misreported as a codec problem. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>
dmitry-tokarev-nv
force-pushed
the
dtokarev/media-codec-error-ux
branch
from
July 21, 2026 00:00
f40f06d to
968d8cc
Compare
dmitry-tokarev-nv
force-pushed
the
dtokarev/ops-7665-codec-compliance
branch
from
July 29, 2026 15:40
d79e13b to
cb0063b
Compare
dmitry-tokarev-nv
force-pushed
the
dtokarev/ops-7665-codec-compliance
branch
3 times, most recently
from
July 30, 2026 12:03
116728e to
ea49322
Compare
dmitry-tokarev-nv
force-pushed
the
dtokarev/ops-7665-codec-compliance
branch
from
July 31, 2026 03:20
9aaf999 to
6e29800
Compare
Contributor
Author
|
Superseded by #12725, which reimplements this against the post-codec-removal / post-NVDEC / post-#12051 tree. Everything this PR touched was rewritten since July: the blanket decord preflight would now break H.264/H.265 requests that NVDEC serves without decord, the recommended installs must use the validated version bounds from #12051, and the docs page is covered by video-decode-gpu-requirements.md plus additional-media-decoders.md. The three review findings raised here are carried into #12725 as design constraints (preflight after payload validation; no implication that VP8/VP9 avoids the backend decoder; original error kept prominent with conditional guidance). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The runtime images ship a VP8/VP9-only media stack — H.264/H.265/AAC and the OpenCV/decord backend decoders are omitted. Today, sending an unsupported video format yields an opaque low-level error (
Decoder not found,No module named 'cv2',No module named 'decord'). This adds actionable guidance at each decode entry point so a user knows which decoder is missing and how to enable it:decoders/video.rs): on decode failure, explain that only VP8/VP9 are built in and how to fix — re-encode to VP9 (ffmpeg -i in.mp4 -c:v libvpx-vp9 -an out.webm) or run a codec-enabled image — while preserving the original ffmpeg error for diagnostics.video_loader.py): catch the lazily-imported OpenCVImportErrorand point topip install opencv-python-headless(or send VP8/VP9, which the frontend decodes).decorddecoder for video requests and point topip install decord2, instead of a bareNo module named 'decord'surfaced from deep inside SGLang.Stacked on #11836 (which removes the codecs) so it carries the right context; rebase onto
mainonce #11836 lands.Validation
py_compile+ruff format/check(pinned v0.5.2) clean on all changed files.ImportErrorguidance path, and the SGLang preflight for both decoder-present and decoder-absent. The existing extract-media-urls tests stay green by stubbing the preflight. Preflight helper logic unit-verified locally.h264/aac) with no decoder present, so these messages can be specific.map_errwrap; full compile/build is exercised by CI.Linear: OPS-7779
🤖 Generated with Claude Code