Skip to content

fix(multimodal): align video pixel shuffle with inference - #6255

Open
ehosseiniasl wants to merge 1 commit into
NVIDIA:mainfrom
ehosseiniasl:ehsan/video-rl-gym-async-grpo
Open

fix(multimodal): align video pixel shuffle with inference#6255
ehosseiniasl wants to merge 1 commit into
NVIDIA:mainfrom
ehosseiniasl:ehsan/video-rl-gym-async-grpo

Conversation

@ehosseiniasl

@ehosseiniasl ehosseiniasl commented Aug 4, 2026

Copy link
Copy Markdown

Summary

Correct dynamic-resolution pixel shuffling for multimodal image and temporal-video inputs.

  • Perform spatial pixel shuffle using each input's real patch grid.
  • Apply pixel shuffle independently to dynamic-resolution video tubelets.
  • Use the actual post-shuffle token count when expanding multimodal placeholders.
  • Avoid applying pixel shuffle twice on packed inputs.
  • Add regression coverage for non-square inputs and 448x576 video frames.

Motivation

The policy training path must preserve the same spatial feature ordering used during inference. Flattened reshaping of non-square video frames grouped the wrong neighboring
patches, causing generation and training to consume differently ordered visual features.

This is required for consistent Nemotron Omni video GRPO training and rollout/training log-probability parity.

Validation

  • Added unit coverage for non-square pixel-shuffle ordering.
  • Added dynamic-resolution temporal-video coverage using 448x576 frames.
  • Exercised through synchronous NeMo RL video GRPO training.

Related work

Megatron-Bridge: NVIDIA-NeMo/Megatron-Bridge#5304
Gym: NVIDIA-NeMo/Gym#2324

Signed-off-by: Ehsan Hosseini Asl <ehsan.hosseiniasl@gmail.com>
@ehosseiniasl
ehosseiniasl requested review from a team as code owners August 4, 2026 18:59
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft August 4, 2026 18:59
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants