feat(video): add Gym support for sync and async GRPO

[ehosseiniasl]

Summary

Add end-to-end video support for NeMo Gym GRPO with synchronous and asynchronous Megatron training.

• Carry video inputs through datasets, Gym rollouts, trajectory collection, batching, generation, and policy training.
• Preserve video metadata and cached frame tensors across distributed worker boundaries.
• Add Nemotron Omni dynamic-resolution video preprocessing with temporal tubelets and expanded visual placeholders.
• Use MBridges canonical NemotronOmniModelProvider instead of the historical LLaVA model path.
• Apply NeMo RL-owned runtime compatibility patches to stock vLLM 0.25.1, removing the custom-vLLM-fork dependency.
• Keep rollout and policy preprocessing aligned for frame sampling, resizing, temporal patching, token IDs, and log probabilities.
• Support video rollouts in synchronous and asynchronous GRPO.
• Preserve video data across async replay-buffer, refit, and checkpoint-resume paths.
• Fix fresh async runs waiting unnecessarily for a complete lookahead batch before starting training.
• Add cached-video dataset preparation, sync/async recipes, and reproducible 2-node and 16-node launchers.
• Keep GRPO training unlimited with raw TMPE reporting.

Motivation

Video GRPO requires the rollout and policy paths to consume the same frames, multimodal token layout, and token IDs. Differences in video preprocessing or tokenization can produce incorrect
policy log probabilities and elevated TMPE.

The implementation keeps video media and preprocessing metadata intact throughout Gym, vLLM, distributed trajectory handling, and Megatron training. It also supports asynchronous generation,
in-flight weight refits, and replay-buffer checkpoint recovery without losing multimodal data.

Validation

Focused tests

• Passed 52 focused video, Gym, vLLM, multimodal batching, and async tests against stock vLLM 0.25.1.
• Passed async startup-barrier and recipe-configuration tests.
• Passed Ruff and git diff --check.

2-node synchronous GRPO

• Job: 5811480
• Completed 40 optimizer steps in a four-hour allocation.
• Raw TMPE: 1.0142
• Average reward at step 40: 0.25
• Generation KL: 0.0008
• All samples remained unmasked.
• W&B: https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/t7wtboef

16-node asynchronous GRPO

• Job: 5822801
• Verified through 15 optimizer/refit cycles.
• Raw TMPE remained approximately 1.013-1.017.
• Rewards were nonzero and ranged from approximately 0.219-0.406 across the inspected steps.
• In-flight refits successfully advanced generation weight versions.
• All inspected samples remained unmasked.
• W&B: https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/19jwwuw1

Both runs used:

• grpo.max_num_steps=-1
• grpo.seq_logprob_error_threshold=null
• Stock vLLM 0.25.1
• Cached 32-frame video input
• Canonical MBridge Nemotron Omni expanded-sequence training

Dependency stack

Merge order:

1. feat(video): enable canonical Nemotron Omni training for V2 MoE checkpoints Megatron-Bridge#5304� feat(video): enable canonical Nemotron Omni training for V2 MoE checkpoints
2. feat(video): support video inputs in vLLM responses Gym#2324� feat(video): support video inputs in vLLM responses

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cspades]

@ehosseiniasl Do you have an ETA for when this PR will be merged?

[cspades]

@rohitrango Could you maybe answer?