fix(grpo): group advantages by rollout identity - #3959
Merged
Conversation
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> (cherry picked from commit d5398b7)
ruff-format (v0.9.9, as pinned in .pre-commit-config.yaml) was not clean on four files touched by this PR; every offending hunk is a line this PR adds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> (cherry picked from commit 1508125)
input_ids was the GRPO group key before this PR; both consumers are now reward_group_ids, leaving the upstream flatten with no readers. Every generation batch built an unused [B, S_prompt] tensor, multiplied by dynamic_sampling_num_gen_batches under DAPO, and re-ran PackedTensor.flattened_concat over every image on VLM recipes. Removes the flatten, the NeMo-Gym branch assignment, and the trailing del input_ids (which would otherwise NameError). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> (cherry picked from commit fc1a3c1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fc1a3c193357ee7a64374141a7a0c5ec8e597ed0tosuper-v3.5-posttraining.The source PR is still open. This side-branch backport intentionally snapshots its current three-commit state while the remaining main-branch review discussion continues.
Backport notes
-x, preserving authorship and trailers.tests/unit/single_controller/test_single_controller.py; no unrelated main-only tests were brought over.Testing
pre-commit run --all-filespassed, including Ruff, Ruff format, Pyrefly, and recipe checks.Source: #3882