Remove nvenc/decs plus update Xenna to 0.2.0

[abhinavg4]

Description

Update Xenna to 0.2.0

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR updates cosmos-xenna from 0.1.2 to 0.2.0, removing the NVENC/NVDEC hardware codec resource abstractions (nvdecs, nvencs, entire_gpu) in favor of a simpler fractional GPU allocation model (gpus = 1.0 / nb_streams_per_gpu). It also corrects a previously inverted RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES env var value and cleans up the import paths to align with the new Xenna 0.2.0 API.

Key changes:

• Resources simplified: nvdecs, nvencs, and entire_gpu fields removed; GPU scheduling now uses fractional gpus values exclusively.
• RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES corrected: Previously set to "0" (which would have let Ray manage CUDA_VISIBLE_DEVICES, contradicting the comment), now correctly set to "1" to fully cede GPU device management to Xenna.
• XENNA_RESPECT_CUDA_VISIBLE_DEVICES removed: No longer needed since Ray is now properly prevented from overriding CUDA_VISIBLE_DEVICES.
• ClipTranscodingStage simplified: Removed dependency on private Xenna GPU introspection calls (_get_local_gpu_info, _make_gpu_resources_from_gpu_name); fractional GPU allocation (1.0 / nb_streams_per_gpu) is now used directly.
• Tests updated: MockGpuResources removed; new focused tests split hwaccel-command assertions from resource-allocation assertions.
• New transitive dependencies: obstore and portpicker added to uv.lock as new Xenna 0.2.0 deps.

Confidence Score: 5/5

• This PR is safe to merge; it is a clean library version upgrade with well-scoped, consistent changes throughout.
• All resource-field removals are applied consistently across the codebase (Resources, adapter, executor, pipeline describe, tests, and docs). The env-var fix (RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES "0" → "1") corrects a prior inversion. The fractional GPU model correctly replaces NVENC-unit counting. Tests are updated and properly cover the new behavior.
• No files require special attention.

Important Files Changed

Filename	Overview
nemo_curator/stages/resources.py	Removes nvdecs, nvencs, and entire_gpu fields from Resources dataclass; cleans up __post_init__ and requires_gpu accordingly. The simplification is clean and correct.
nemo_curator/backends/xenna/executor.py	Fixes RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES from "0" to "1" (correcting the prior inverted value) and removes RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO which was a Xenna 0.1.x-specific env var no longer needed in 0.2.0.
nemo_curator/backends/xenna/adapter.py	Import path moved to cosmos_xenna.pipelines.private.resources and nvdecs/nvencs/entire_gpu removed from XennaResources construction. Private module path is flagged in prior review threads.
nemo_curator/stages/video/clipping/clip_extraction_stages.py	Replaces NVENC-unit-based GPU allocation with fractional GPU allocation (1.0 / nb_streams_per_gpu); removes private Xenna imports for GPU detection, simplifying the resource model considerably.
nemo_curator/core/utils.py	Removes XENNA_RESPECT_CUDA_VISIBLE_DEVICES=1 env var, which is no longer needed since RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1 now correctly hands full CUDA management to Xenna.
tests/stages/video/clipping/test_clip_transcoding_stage.py	Removes MockGpuResources class, adds focused test_add_hwaccel_options_enabled, test_add_hwaccel_options_enabled_multiple_streams, and test_resources_fractional_gpu_allocation tests that properly separate command-option and resource-allocation concerns.
nemo_curator/backends/experimental/ray_data/adapter.py	Removes the now-stale nvdecs/nvencs guard that raised a ValueError for Ray Data; the TODO is resolved by removing those resource fields entirely.
pyproject.toml	Bumps cosmos-xenna pin from ==0.1.2 to ==0.2.0; uv.lock updated accordingly with new transitive dependencies (obstore, portpicker).
nemo_curator/pipeline/pipeline.py	Removes NVDEC/NVENC lines from Pipeline.describe(), dropping it below the complexity threshold so # noqa: C901 is correctly removed.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["ClipTranscodingStage.__post_init__"] --> B{use_hwaccel or\nh264_nvenc encoder?}
    B -- Yes --> C{nb_streams_per_gpu > 0?}
    C -- Yes --> D["Resources(gpus = 1.0 / nb_streams_per_gpu)"]
    C -- No --> E["Resources(gpus = 1)"]
    B -- No --> F["Resources(cpus = num_cpus_per_worker)"]

    D --> G["XennaStageAdapter.required_resources"]
    E --> G
    F --> G
    G --> H["XennaResources(cpus=..., gpus=...)"]

    subgraph Ray Init
        I["ray.init(runtime_env)"] --> J["RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n(Ray will NOT set CUDA_VISIBLE_DEVICES)"]
    end

    H --> K["Xenna 0.2.0 manages\nCUDA_VISIBLE_DEVICES"]
    J --> K

Loading

_{Last reviewed commit: 5ad14d7}

[praateekmahajan]

@abhinavg4 Is the XENNA_RESPECT_CUDA_VISIBLE_DEVICES flag still relevant? If not we could probably remove that right?

[abhinavg4]

@abhinavg4 Is the XENNA_RESPECT_CUDA_VISIBLE_DEVICES flag still relevant? If not we could probably remove that right?

I think as we discussed, I'm just keeping everything same as Xenna. I removed the ACCL thing

[praateekmahajan]

@abhinavg4 I'm talking about a different env variable than what we discussed. We don't need XENNA_RESPECT_CUDA_VISIBLE_DEVICES environment flag to be set here

Curator/nemo_curator/core/utils.py

Line 178 in b795fe2

os.environ["XENNA_RESPECT_CUDA_VISIBLE_DEVICES"] = "1"

Since this is no longer used in Xenna 0.2.0 per my understanding https://github.com/search?q=repo%3Anvidia-cosmos%2Fcosmos-xenna%20XENNA_RESPECT_CUDA_VISIBLE_DEVICES&type=code

[abhinavg4]

/ok to test b3d93c2

[abhinavg4]

/ok to test 0978433

[praateekmahajan]

LGTM

[thomasdhc]

Skip codecov due to hang