Limit CPU video decoder codec support by JanuszL · Pull Request #6352 · NVIDIA/DALI

JanuszL · 2026-05-18T11:23:05Z

Limit CPU video decoder codec support

Restrict the CPU frames decoder to codecs supported by the currently
compiled libavcodec configuration. H264 and HEVC are no longer
advertised for the CPU variant while VP8, VP9, and MJPEG remain
enabled.

Make ReadRegularFrame mark end-of-stream by setting next_frame_idx_
to -1 when the index reaches NumFrames(), mirroring the existing
guard in ReadFlushFrame. Without this, codecs with no decoder latency
(VP9 on the new test inputs) deliver the final frame via the regular
path, leaving next_frame_idx_ at NumFrames() and causing
VideoInput depletion to be reported one batch late.

Reset the decoder when an indexed next frame falls outside the valid
range, avoiding reuse of an invalid decoder position.

Update video decoder tests to expect CPU failures for unsupported
codecs instead of skipping only MPEG4. Use VP9 CFR/VFR test inputs
and device-less CPU pipelines where appropriate. Point the CFR/VFR
reference frame folders at `frames_{1,2}_vp9/` so CPU decode of the
new VP9 fixtures matches at the existing eps=10 tolerance. Drop the
CPU HEVC frames-decoder tests (`ConstantFrameRateHevc`,
`VariableFrameRateHevc`, `VariableFrameRateHevcNoIndex`) — HEVC is
no longer in the CPU codec allow-list.

Tolerate up to 16 isolated subpixel deviations exceeding eps in
TestVideo::CompareFrame (out of ~2.7M subpixels per frame). The CPU
VP9 decode path occasionally produces a single byte that differs by
~32 — a SIMD glitch inside libavcodec/sws_scale that Valgrind cannot
instrument. The budget is orders of magnitude below what any genuine
regression would produce, so test sensitivity is preserved.

In dali/test/python/input/test_video.py, filter out h264 from the
round-robin fixture (the unsuffixed test_{1,2}.mp4 in cfr//vfr/
are h264) and restrict test_video_input_audio_stream to the mixed
backend — the only DALI_extra video with an audio stream is h264.

Category:

Bug fix (non-breaking change which fixes an issue)

Description:

Restricts the CPU video frames decoder to the codecs supported by the
currently compiled libavcodec configuration. H264 and HEVC are no longer
advertised for the CPU variant, while VP8, VP9, and MJPEG remain enabled.

ReadRegularFrame now mirrors ReadFlushFrame and signals end-of-stream
by setting next_frame_idx_ to -1 once NumFrames() is reached, so
codecs with no decoder latency report depletion immediately instead of
one batch late.

Resets the decoder when an indexed next frame falls outside the valid
range, avoiding reuse of an invalid decoder position.

The video decoder tests now expect CPU failures for unsupported codecs
instead of skipping only MPEG4. The affected CFR/VFR test inputs are
switched to VP9 variants, and CPU pipelines use device_id=None where
appropriate. The CFR/VFR reference frame folders are repointed at the
new VP9-derived frames_{1,2}_vp9/ so CPU decode matches at the
existing eps=10 tolerance. CPU HEVC frames-decoder tests are removed.

TestVideo::CompareFrame now tolerates up to 16 isolated subpixel
deviations exceeding eps per frame (out of ~2.7M). The CPU VP9 decode
path occasionally produces a single byte that differs by ~32 — a SIMD
glitch inside libavcodec/sws_scale that Valgrind cannot instrument.
The budget is orders of magnitude below any genuine regression, so
test sensitivity is preserved.

dali/test/python/input/test_video.py filters out h264 from the
round-robin fixture and restricts test_video_input_audio_stream to
the mixed backend — the only DALI_extra video with an audio stream is
h264, which CPU can no longer decode.

Additional information:

Affected modules and functionalities:

CPU video frames decoder: codec allow-list and ReadRegularFrame
end-of-stream signalling.
Video frames decoder seek path (reset when seek target is out of
range).
Video decoder gtests: reference frame paths, dropped CPU HEVC cases,
relaxed CompareFrame tolerance.
dali/test/python/input/test_video.py: h264 fixture filter and
audio-stream test backend restriction.
DALI deps and DALI_extra version pins.

Key points relevant for the review:

DALI_DEPS_VERSION and DALI_EXTRA_VERSION are temporary ToDo
placeholders until the corresponding dali_deps and dali_extra
repository changes merge.
CPU unsupported-codec behavior is now tested as an expected failure
instead of being skipped for a subset of codecs.
The ReadRegularFrame EOS guard is required for VideoInput
depletion to fire on the right batch with VP9 inputs (h264 hid the
off-by-one through its decoder latency, which routed the tail
through ReadFlushFrame).
The 16-subpixel tolerance in CompareFrame is a flake mitigation,
not a tolerance loosening: a real codec/colorspace bug would touch
thousands of subpixels.

Tests:

Not run locally.

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-4712

JanuszL · 2026-05-18T11:25:40Z

NVIDIA/DALI_extra#135 & NVIDIA/DALI_deps#162 are related to this change.

dali-automaton · 2026-05-18T11:26:02Z

CI MESSAGE: [51666287]: BUILD STARTED

greptile-apps · 2026-05-18T11:27:30Z

Greptile Summary

This PR restricts the CPU video frames decoder to codecs supported by the compiled libavcodec (VP8, VP9, MJPEG only — H264 and HEVC removed), fixes an off-by-one end-of-stream signal in ReadRegularFrame, and resets the decoder when a seek target falls outside the valid index range.

Core decoder changes: SelectVideoStream now uses a correctly-sized std::array<AVCodecID, 3>, ReadRegularFrame sets next_frame_idx_ = -1 on exhaustion (mirroring the existing ReadFlushFrame guard), and SeekFrame gains a reset path for out-of-bounds indexed positions.
Test infrastructure updates: CPU HEVC gtest cases are dropped; unsupported-CPU-codec paths now assert a RuntimeError instead of being skipped; cfr/vfr reference fixtures are repointed to new VP9 variants; CompareFrame tolerates up to 16 isolated subpixel deviations to suppress an sws_scale SIMD flake.
CI setup: TL0_videoreader_test/test.sh is updated to copy the VP9 multi-resolution directory and use the VP9 Sintel container.

Confidence Score: 4/5

Safe to merge once any outstanding questions around NumFrames() re-entrancy after Reset() are confirmed; the codec restriction and EOS-fix logic are sound.

The codec allow-list change, EOS guard, and seek-reset path are all well-reasoned and internally consistent. The early NumFrames() call in ReadNextFrame is the right approach to pre-populate the index, but its interaction with Reset() — which zeroes next_frame_idx_ and reopens the stream — means ParseNumFrames() could be re-invoked after every seek-triggered reset. Whether that is safe depends on ParseNumFrames() being idempotent after the file is reopened.

frames_decoder_cpu.cc — specifically the NumFrames() / ParseNumFrames() interaction after Reset() in a seek-heavy workload.

Important Files Changed

Filename	Overview
dali/operators/video/frames_decoder_cpu.cc	Restricts supported codecs to VP8/VP9/MJPEG (array size correctly fixed to 3), adds EOS guard in ReadRegularFrame, pre-populates index via NumFrames() on first frame, and tolerates AVERROR_EOF from avcodec_send_packet(null). Logic looks sound; one open question around NumFrames() re-entrancy after Reset().
dali/operators/video/frames_decoder_base.cc	SeekFrame gains a guard to reset the decoder when next_frame_idx_ is beyond NumFrames() (guarded by HasIndex()), mirroring the existing negative-index reset. Reset() sets next_frame_idx_=0, so the assert after the new condition holds.
dali/operators/video/video_test.cc	CompareFrame switched from early-exit on first mismatch to counting all bad subpixels; tolerates up to 16 isolated deviations to suppress VP9/sws_scale SIMD flake. Reference frame paths updated to _vp9 variants. Copyright header is stale (2021-2022).
dali/test/python/decoder/test_video.py	Adds assert_unsupported_cpu_codec helper and exercises all previously-skipped CPU unsupported codecs as expected-failure tests; fixes device_id=None for CPU pipelines; updates cfr/vfr paths to VP9 fixtures; removes pipe.build() in several places relying on auto-build.
dali/test/python/input/test_video.py	Filters out H264 (test_1/test_2.mp4) from the round-robin fixture; switches audio-stream test to sintel_trailer_vp9.mp4. The test_video_input_audio_stream docstring may no longer accurately describe the scenario if the VP9 file has no audio track.
qa/TL0_videoreader_test/test.sh	Switches resolution test fixture from the full video_resolution tree to the VP9 subdirectory; switches sintel container to the VP9 variant. The previous find -name 'vp9' bug was fixed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ReadNextFrame called] --> B{next_frame_idx_ == 0?}
    B -->|yes| C[Call NumFrames to populate index]
    C --> D{next_frame_idx_ == -1?}
    B -->|no| D
    D -->|yes| E[return false EOS]
    D -->|no| F{flush_state_?}
    F -->|no| G[ReadRegularFrame]
    F -->|yes| H[ReadFlushFrame]
    G --> I{frame received?}
    I -->|yes| J[CopyToOutput + increment idx]
    J --> K{idx >= NumFrames?}
    K -->|yes| L[idx = -1]
    K -->|no| M[return true]
    L --> M
    I -->|no| N[send null packet, flush_state_=true, return false]
    H --> O{frame received?}
    O -->|no| P[flush_state_=false, idx=-1, return false]
    O -->|yes| Q[CopyToOutput + increment idx]
    Q --> R{idx >= NumFrames?}
    R -->|yes| S[idx = -1]
    R -->|no| T[return true]
    S --> T

_{Reviews (21): Last reviewed commit: "Typo fix" | Re-trigger Greptile}

greptile-apps · 2026-05-18T11:27:35Z

@@ -1 +1 @@
-b270f29e9d7655512e7e8eaf055cca4d19b55f55
+ToDo


[Bug] Version pin files set to literal ToDo placeholder.

Both DALI_DEPS_VERSION and DALI_EXTRA_VERSION now contain the string ToDo instead of a commit SHA. Any CI job that reads these files to fetch the matching dali_deps / dali_extra artefacts will either fail outright or pick up an incorrect/stale revision, breaking reproducibility for the entire build. These files should be updated to the real commit SHAs before this PR merges (or the dependent PRs should land first).

JanuszL · 2026-05-18T22:02:45Z

@greptile review

JanuszL · 2026-05-18T22:19:10Z

@greptile review

JanuszL · 2026-05-18T22:27:17Z

@greptile review

greptile-apps · 2026-05-18T22:38:03Z

    ++next_frame_idx_;
+    if (next_frame_idx_ >= NumFrames()) {
+      next_frame_idx_ = -1;
+      LOG_LINE << "Next frame index out of bounds (regular), setting to -1" << std::endl;
+    }
    return true;


The new EOS guard calls NumFrames() unconditionally on every decoded frame, which can invoke ParseNumFrames() when no index is built and nb_frames is zero in the container. ParseNumFrames() reads all remaining demuxer packets to completion, so the very first frame's increment will exhaust the packet stream and cause all subsequent av_read_frame calls to return EOF — silently dropping every frame after the first. The existing guard in ReadFlushFrame has the same limitation (documented with a TODO) but that function runs only after the demuxer is already exhausted. The fix is to guard the check with HasIndex(), mirroring the SeekFrame condition added in this same PR.

Suggested change

++next_frame_idx_;

if (next_frame_idx_ >= NumFrames()) {

next_frame_idx_ = -1;

LOG_LINE << "Next frame index out of bounds (regular), setting to -1" << std::endl;

}

return true;

++next_frame_idx_;

// TODO(awolant): Figure out how to handle this during index building

// Or when NumFrames is unavailable

if (HasIndex() && next_frame_idx_ >= NumFrames()) {

next_frame_idx_ = -1;

LOG_LINE << "Next frame index out of bounds (regular), setting to -1" << std::endl;

}

return true;

greptile-apps · 2026-05-18T22:38:04Z

    batch_size = 3
    pipe = test_pipeline(batch_size=batch_size, num_threads=3, device_id=0)


test_multichannel_fill_value hard-codes device_id=0 even though the test body uses fn.experimental.decoders.video which is a CPU/mixed operator; on a device-less CI machine this will fail at pipeline construction. Other tests in this PR were correctly updated to derive device_id from device, so this one was apparently missed.

Suggested change

batch_size = 3

pipe = test_pipeline(batch_size=batch_size, num_threads=3, device_id=0)

batch_size = 3

device_id = None if device == "cpu" else 0

pipe = test_pipeline(batch_size=batch_size, num_threads=3, device_id=device_id)

dali-automaton · 2026-05-18T22:39:35Z

CI MESSAGE: [51735694]: BUILD STARTED

dali-automaton · 2026-05-19T04:49:59Z

CI MESSAGE: [51735694]: BUILD FAILED

dali-automaton · 2026-05-19T05:12:53Z

CI MESSAGE: [51770028]: BUILD STARTED

greptile-apps · 2026-05-19T05:20:28Z

        # test overflow of frame_buffer_
        filenames.append(f"{get_dali_extra_path()}/db/video/cfr_test.mp4")
-        filenames = filter(lambda filename: "mpeg4" not in filename, filenames)
        filenames = filter(lambda filename: "hevc" not in filename, filenames)
        filenames = filter(lambda filename: "av1" not in filename, filenames)
+        if device == "cpu":
+            # some formats are not yet supported in the CPU operator itself
+            filenames = filter(lambda filename: "mpeg4" not in filename, filenames)
+            filenames = filter(
+                lambda filename: "test_1.mp4" not in filename and "test_2.mp4" not in filename,
+                filenames,
+            )
        filenames = cycle(filenames)


cfr_test.mp4 is H264 and is not filtered for CPU.

The DALI_extra README.rst shows cfr_test.mp4 is generated with -c:v libx264, so it's H264. This file is appended to filenames before the CPU-conditional filters run. The CPU block filters mpeg4, test_1.mp4, and test_2.mp4, but cfr_test.mp4 slips through. When the CPU decoder encounters it, SelectVideoStream issues a DALI_WARN and returns false, causing a RuntimeError that fails the test.

The fix is to guard the filenames.append call (or add "cfr_test" not in filename to the CPU filter) so that the H264-specific overflow fixture is only used when the device can decode H264.

dali-automaton · 2026-05-19T05:53:16Z

CI MESSAGE: [51774937]: BUILD STARTED

greptile-apps · 2026-05-19T05:58:29Z

+        if device == "gpu":
+            filenames.append(f"{get_dali_extra_path()}/db/video/cfr_test.mp4")


[Bug] Dead condition — cfr_test.mp4 is never appended for the "mixed" device.

test_multi_gpu_video is decorated with @params("cpu", "mixed"), so device is never "gpu". The frame-buffer overflow fixture (cfr_test.mp4) is silently skipped for the "mixed" case, which was the very path the file was added to stress. The condition should be device == "mixed" (or device != "cpu").

Suggested change

if device == "gpu":

filenames.append(f"{get_dali_extra_path()}/db/video/cfr_test.mp4")

if device == "mixed":

filenames.append(f"{get_dali_extra_path()}/db/video/cfr_test.mp4")

dali-automaton · 2026-05-19T06:15:51Z

CI MESSAGE: [51777473]: BUILD STARTED

dali-automaton · 2026-05-19T07:49:13Z

CI MESSAGE: [51777473]: BUILD FAILED

dali-automaton · 2026-05-20T07:44:21Z

CI MESSAGE: [51925234]: BUILD FAILED

dali-automaton · 2026-05-20T07:51:19Z

CI MESSAGE: [51925957]: BUILD STARTED

jantonguirao · 2026-05-22T10:31:24Z



-@params("cpu", "gpu")
+@params("gpu")


[Coverage] Dropping CPU from @params here is a real regression in test surface, not just a fixture-availability artifact. The test uses /db/video/full_dynamic_range/video.mp4 (H.264), which the new CPU codec policy rejects — so the only fix is a VP9 version of that fixture in DALI_extra. Worth filing a follow-up so this CPU path doesn't quietly stay uncovered, or extending the PR #135 set with video_vp9.mp4 here.

The code path that detect full range videos applies only to h264/h265, and for VP9 we would need to implement if separately. I will add this to our ToDo list.

Follow-up #6367

jantonguirao · 2026-05-22T10:31:24Z

-@params(*device_values)
+# The only available test video with an audio stream (sintel) is h264, which is not supported by
+# the CPU variant of the operator. Run this test only on the mixed (GPU) backend.
+@params("mixed")


[Coverage / Question] Same shape as the test_full_range_video_in_memory coverage loss — CPU dropped because the only audio-stream fixture is H.264. However: NVIDIA/DALI_extra#135 adds sintel_trailer_vp9.mp4. Does that file retain the audio stream from the original sintel? If yes, this test can stay parameterized over both devices by switching the input to the VP9 variant. If no, worth re-encoding to preserve audio so we don't lose this CPU-side audio-stream regression check permanently.

jantonguirao · 2026-05-22T10:31:24Z

 }

+unsupported_cpu_codec_error = r"is not supported by the CPU variant of this operator\."
+unsupported_cpu_codecs = {"h264", "hevc", "mpeg4"}


[Drift risk — separate from the maintainability comment on the original review] This set claims to enumerate "unsupported on CPU" codecs but is missing AV1, which is also unsupported on the CPU path (no decoder in libavcodec build + commented out in frames_decoder_cpu.cc:223). It works today only because both test_decoder_operator_codec_support and test_reader_operator_codec_support early-skip AV1 with raise SkipTest("...Ampere+ GPUs..."). If/when AV1 becomes generally available and that skip is removed, the codec-rejection branch will be silently bypassed and the test will attempt a real decode on CPU. Add "av1" here defensively, or guard the skip with if codec == "av1" and not has_ampere(): raise SkipTest(...).

dali-automaton · 2026-05-22T12:12:54Z

CI MESSAGE: [52225500]: BUILD STARTED

greptile-apps · 2026-05-22T12:17:36Z

 @params(*device_values)
 def test_video_input_audio_stream(device):
    """


Comment contradicts decorator — and audio-stream coverage may be lost.

The block comment says "Run this test only on the mixed (GPU) backend", but the decorator @params(*device_values) expands to @params("cpu", "mixed"), so the CPU case still runs. The file was switched from sintel_trailer-720p.mp4 (h264 + audio) to sintel_trailer_vp9.mp4: if the VP9 re-encode preserved the audio track, CPU can handle it and the comment is simply stale. If the audio track was dropped during re-encoding, neither device is actually testing the "video decoding when audio stream is present" scenario described in the docstring — the test would pass vacuously. Please confirm whether sintel_trailer_vp9.mp4 retains an audio stream, and update the comment (or restrict the decorator to @params("mixed")) accordingly.

dali-automaton · 2026-05-22T12:20:22Z

CI MESSAGE: [52226023]: BUILD STARTED

Restrict the CPU frames decoder to codecs supported by the currently compiled libavcodec configuration. H264 and HEVC are no longer advertised for the CPU variant while VP8, VP9, and MJPEG remain enabled. Make `ReadRegularFrame` mark end-of-stream by setting `next_frame_idx_` to -1 when the index reaches `NumFrames()`, mirroring the existing guard in `ReadFlushFrame`. Without this, codecs with no decoder latency (VP9 on the new test inputs) deliver the final frame via the regular path, leaving `next_frame_idx_` at `NumFrames()` and causing `VideoInput` depletion to be reported one batch late. Reset the decoder when an indexed next frame falls outside the valid range, avoiding reuse of an invalid decoder position. Update video decoder tests to expect CPU failures for unsupported codecs instead of skipping only MPEG4. Use VP9 CFR/VFR test inputs and device-less CPU pipelines where appropriate. Point the CFR/VFR reference frame folders at `frames_{1,2}_vp9/` so CPU decode of the new VP9 fixtures matches at the existing eps=10 tolerance. Drop the CPU HEVC frames-decoder tests (`ConstantFrameRateHevc`, `VariableFrameRateHevc`, `VariableFrameRateHevcNoIndex`) — HEVC is no longer in the CPU codec allow-list. Tolerate up to 16 isolated subpixel deviations exceeding eps in `TestVideo::CompareFrame` (out of ~2.7M subpixels per frame). The CPU VP9 decode path occasionally produces a single byte that differs by ~32 — a SIMD glitch inside libavcodec/sws_scale that Valgrind cannot instrument. The budget is orders of magnitude below what any genuine regression would produce, so test sensitivity is preserved. In `dali/test/python/input/test_video.py`, filter out h264 from the round-robin fixture (the unsuffixed `test_{1,2}.mp4` in `cfr/`/`vfr/` are h264) and restrict `test_video_input_audio_stream` to the mixed backend — the only DALI_extra video with an audio stream is h264. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-05-22T13:21:44Z

CI MESSAGE: [52231011]: BUILD STARTED

dali-automaton · 2026-05-22T19:54:16Z

CI MESSAGE: [52231011]: BUILD PASSED

szalpal · 2026-05-25T11:20:08Z

 bool FramesDecoderCpu::ReadNextFrame(uint8_t *data) {
  LOG_LINE << "FramesDecoderCpu::ReadNextFrame: next_frame_idx_=" << next_frame_idx_ << std::endl;
+  if (next_frame_idx_ == 0) {
+    // call NumFrames() to populate the index before nay frames are read


I believe there's a typo here? Should it be new frames are read?

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-05-25T11:35:09Z

CI MESSAGE: [52531032]: BUILD STARTED

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

dali-automaton · 2026-05-25T16:03:10Z

CI MESSAGE: [52552124]: BUILD STARTED

dali-automaton · 2026-05-25T17:45:32Z

CI MESSAGE: [52552124]: BUILD FAILED

dali-automaton · 2026-05-26T05:05:51Z

CI MESSAGE: [52552124]: BUILD PASSED

dali-automaton · 2026-05-26T06:57:18Z

CI MESSAGE: [52531032]: BUILD PASSED

Limit CPU video decoder codec support Restrict the CPU frames decoder to codecs supported by the currently compiled libavcodec configuration. H264 and HEVC are no longer advertised for the CPU variant while VP8, VP9, and MJPEG remain enabled. Make ReadRegularFrame mark end-of-stream by setting next_frame_idx_ to -1 when the index reaches NumFrames(), mirroring the existing guard in ReadFlushFrame. Without this, codecs with no decoder latency (VP9 on the new test inputs) deliver the final frame via the regular path, leaving next_frame_idx_ at NumFrames() and causing VideoInput depletion to be reported one batch late. Reset the decoder when an indexed next frame falls outside the valid range, avoiding reuse of an invalid decoder position. Update video decoder tests to expect CPU failures for unsupported codecs instead of skipping only MPEG4. Use VP9 CFR/VFR test inputs and device-less CPU pipelines where appropriate. Point the CFR/VFR reference frame folders at `frames_{1,2}_vp9/` so CPU decode of the new VP9 fixtures matches at the existing eps=10 tolerance. Drop the CPU HEVC frames-decoder tests (`ConstantFrameRateHevc`, `VariableFrameRateHevc`, `VariableFrameRateHevcNoIndex`) — HEVC is no longer in the CPU codec allow-list. Tolerate up to 16 isolated subpixel deviations exceeding eps in TestVideo::CompareFrame (out of ~2.7M subpixels per frame). The CPU VP9 decode path occasionally produces a single byte that differs by ~32 — a SIMD glitch inside libavcodec/sws_scale that Valgrind cannot instrument. The budget is orders of magnitude below what any genuine regression would produce, so test sensitivity is preserved. In dali/test/python/input/test_video.py, filter out h264 from the round-robin fixture (the unsuffixed test_{1,2}.mp4 in cfr//vfr/ are h264) and restrict test_video_input_audio_stream to the mixed backend — the only DALI_extra video with an audio stream is h264. Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

…265/AAC Adopts the approach NVIDIA DALI took when it trimmed its own FFmpeg build (NVIDIA/DALI#6352): rather than skipping tests for codecs it no longer supports, assert the absence, so a silent reintroduction is caught. We already guard this at build time -- wheel_builder enumerates -encoders/-decoders/-parsers and fails the build on a match -- but that runs in the builder stage, so it proves what was produced, not what the runtime image ships. A base-image bump, a stray copy, or an ffmpeg earlier on PATH would all slip past it. This checks the same property from where it matters, resolving ffmpeg the way the encode path does, and reuses the guard's own pattern rather than a second one that could drift. One entry point per framework rather than one module marked with all three. tests/conftest.py turns a framework marker into a skip when that framework's module is absent, and it does so per marker: an item carrying vllm+sglang+trtllm is skipped in every image, because no image has all three. The first version did exactly that and skipped 6/6 in the vLLM image citing tensorrt_llm -- green while proving nothing, which is the failure this PR exists to prevent. Verified on GPU across all three runtime images: 1 executed and passed in each. Every absence assertion is paired with a positive one: VP9 must be present. A missing or broken ffmpeg would otherwise satisfy 'no H.264 decoder' trivially. Bitstream filters stay unchecked for the reason the build guard documents: they reframe an already-encoded stream, carry no implementation, and h264_mp4toannexb is needed to feed hardware decode. DALI reached the same conclusion. Add enumerate_bundled_decoders.py beside the codec scan. The scan gates which media libraries may appear in an image; this answers what a library that is allowed to be there actually implements, which is what reviewing a third-party waiver needs. It walks av_codec_iterate rather than reading symbols or strings: these libraries are stripped, so nm reports nothing even for codecs that are present, and codec long names survive for parsers and bitstream filters. Record on the DALI waiver that it is expected to narrow. Upstream DALI has restricted its vendored ffmpeg; the build currently consumed predates that, and enumeration still registers h264, hevc, aac, aac_fixed and aac_latm. Noting it keeps the waiver reviewable rather than assumed permanent. Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

The TensorRT-LLM images decode video through cv2, which they no longer ship, so NVDEC covers H.264/H.265 and VP8/VP9/AV1 have no decode path. Nothing exercised a codec the image cannot decode, so a base-image bump that reintroduced cv2, or a routing change that sent VP9 down the hardware path, would both have gone unnoticed. Assert the failure instead of leaving the gap untested -- the approach NVIDIA DALI took when it trimmed its own FFmpeg build (NVIDIA/DALI#6352). Verified against a real image (1.3.0rc22) rather than assumed: VP9 probes as vp9, does not route to NVDEC, and returns HTTP 400 carrying the upstream reason. The assertions are scoped to what this repo owns -- the status, the "Failed to load video (<url>)" prefix, and that the cause is preserved rather than swallowed. The upstream text itself is asserted by identity, not by wording, so a vendor rewording cannot fail the test for no defect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

The TensorRT-LLM base image ships DALI 2.1.0, which vendors an ffmpeg registering 446 decoders including h264, hevc, aac, aac_fixed and aac_latm. Upstream restricted that build (NVIDIA/DALI_deps#162, NVIDIA/DALI#6352) and released it in 2.1.1, which registers 440 with all five absent and vp8/vp9/mjpeg/av1 retained. Upgrade rather than delete. The vendored libraries cannot be trimmed on their own -- libavcodec, libavfilter, libavformat, libavutil and libswscale are each DT_NEEDED by libdali.so and its siblings, so removing them breaks `import nvidia.dali` outright. Removing the whole package would work (nothing declares it, nothing imports it across 43k files scanned, and the suite passes without it) but it belongs to TensorRT-LLM, and a version bump clears the codecs without taking a package out of someone else's image. A whiteout entry comes with it. pre_runtime rebases on upstream and overlays runtime_full, and DALI's libraries are hash-named, so an upgrade renames rather than overwrites: without dropping the old tree first, the base image's 2.1.0 libraries would ship beside the upgraded ones and the upgrade would achieve nothing while every check still passed. The guard enumerates what the library registers rather than trusting the version string, so a wheel that reintroduces a decoder fails the build. The same check is added to the shipped-image test, where it was verified to fail on 2.1.0 before being trusted to pass afterwards. Validated on GPU: unit suite green, and the TensorRT-LLM integration lane green -- 16 passed including aggregated_multimodal_video_nvdec, video_diffusion, image_diffusion and the E/PD multimodal cases. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Dmitry Tokarev <dtokarev@nvidia.com>

This was referenced May 18, 2026

Add CFR and VFR videos based on VP9 format NVIDIA/DALI_extra#135

Merged

Trim some codecs from FFmpeg build NVIDIA/DALI_deps#162

Merged

greptile-apps Bot reviewed May 18, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch 2 times, most recently from 13fd2c1 to 8375cb2 Compare May 18, 2026 22:02

JanuszL force-pushed the remove_cpu_codecs branch from 8375cb2 to 9389ee9 Compare May 18, 2026 22:18

JanuszL force-pushed the remove_cpu_codecs branch from 9389ee9 to d364075 Compare May 18, 2026 22:27

greptile-apps Bot reviewed May 18, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch 2 times, most recently from a8f64cc to fba22e3 Compare May 19, 2026 05:11

greptile-apps Bot reviewed May 19, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch from fba22e3 to c87a527 Compare May 19, 2026 05:51

greptile-apps Bot reviewed May 19, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch from c87a527 to 4e1a23c Compare May 19, 2026 06:14

JanuszL force-pushed the remove_cpu_codecs branch from 4e1a23c to fd7b970 Compare May 20, 2026 07:42

JanuszL force-pushed the remove_cpu_codecs branch from fd7b970 to dfa6ff4 Compare May 20, 2026 07:49

jantonguirao reviewed May 22, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch 2 times, most recently from c200f32 to 63ae75a Compare May 22, 2026 12:09

greptile-apps Bot reviewed May 22, 2026

View reviewed changes

JanuszL force-pushed the remove_cpu_codecs branch from 63ae75a to 8428bfe Compare May 22, 2026 12:19

JanuszL force-pushed the remove_cpu_codecs branch from 8428bfe to 5c1c613 Compare May 22, 2026 13:20

jantonguirao approved these changes May 25, 2026

View reviewed changes

szalpal approved these changes May 25, 2026

View reviewed changes

Update DALI deps and extra version

3c37290

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

Typo fix

d54c8dd

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>

JanuszL merged commit a8d5ec5 into NVIDIA:main May 26, 2026
6 checks passed

JanuszL deleted the remove_cpu_codecs branch May 26, 2026 05:30

JanuszL mentioned this pull request May 29, 2026

Update video decoder & reader supported format list #6377

Merged

18 tasks

dmitry-tokarev-nv mentioned this pull request Aug 2, 2026

build(container): media-codec cleanup + build gate + NVDEC video decode (OPS-7665) ai-dynamo/dynamo#11836

Merged

		batch_size = 3
		pipe = test_pipeline(batch_size=batch_size, num_threads=3, device_id=0)

		if device == "gpu":
		filenames.append(f"{get_dali_extra_path()}/db/video/cfr_test.mp4")

		@@ -1 +1 @@
		b270f29e9d7655512e7e8eaf055cca4d19b55f55
		ToDo



		@params("cpu", "gpu")
		@params("gpu")

Uh oh!

Conversation

JanuszL commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

JanuszL commented May 18, 2026

Uh oh!

dali-automaton commented May 18, 2026

Uh oh!

greptile-apps Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

greptile-apps Bot May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JanuszL commented May 18, 2026

Uh oh!

JanuszL commented May 18, 2026

Uh oh!

JanuszL commented May 18, 2026

Uh oh!

greptile-apps Bot May 18, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot May 18, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dali-automaton commented May 18, 2026

Uh oh!

dali-automaton commented May 19, 2026

Uh oh!

dali-automaton commented May 19, 2026

Uh oh!

greptile-apps Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dali-automaton commented May 19, 2026

Uh oh!

greptile-apps Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dali-automaton commented May 19, 2026

Uh oh!

dali-automaton commented May 19, 2026

Uh oh!

dali-automaton commented May 20, 2026

Uh oh!

dali-automaton commented May 20, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JanuszL commented May 18, 2026 •

edited

Loading

greptile-apps Bot commented May 18, 2026 •

edited

Loading