Skip to content

feat: grpo advantage clip overlong filter - #2686

Merged
yfw merged 5 commits into
mainfrom
feat/grpo-advantage-clip-overlong-filter
Jun 5, 2026
Merged

feat: grpo advantage clip overlong filter#2686
yfw merged 5 commits into
mainfrom
feat/grpo-advantage-clip-overlong-filter

Conversation

@arnavk-nvidia

@arnavk-nvidia arnavk-nvidia commented Jun 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do ?

add overlong filtering for gym rollout + configurable advantage clipping

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@arnavk-nvidia
arnavk-nvidia requested a review from a team as a code owner June 4, 2026 16:23
@copy-pr-bot

copy-pr-bot Bot commented Jun 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@arnavk-nvidia arnavk-nvidia added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Jun 4, 2026
@arnavk-nvidia
arnavk-nvidia requested a review from yfw June 4, 2026 16:28
@arnavk-nvidia arnavk-nvidia changed the title Feat/grpo advantage clip overlong filter Feat: grpo advantage clip overlong filter Jun 4, 2026
@arnavk-nvidia arnavk-nvidia changed the title Feat: grpo advantage clip overlong filter feat: grpo advantage clip overlong filter Jun 4, 2026
yfw and others added 4 commits June 4, 2026 14:07
@yfw @HeyyyyyyG @arnavk-nvidia
add overlong filtering for gym rollout + configurable advantage clipping
4b12319 
c1abc09b4b9fa1a75cbd10f5ba07c74442c5ff67

#1949

Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-Authored-By: Jiaqi Zeng <jiaqiz@nvidia.com>
Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
@arnavk-nvidia
fixed incompatible interfaces to Master Config
92838c0 
Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
@arnavk-nvidia
using the helper function to mask prompts
b824131 
Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
@arnavk-nvidia
ruff format
70fd5bd 
Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
@arnavk-nvidia
arnavk-nvidia force-pushed the feat/grpo-advantage-clip-overlong-filter branch from 8a58ad1 to 70fd5bd Compare June 4, 2026 21:10
@arnavk-nvidia

arnavk-nvidia commented Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 70fd5bd

yfw
yfw reviewed Jun 4, 2026
View reviewed changes
Comment thread nemo_rl/algorithms/grpo.py Outdated
Comment thread nemo_rl/algorithms/grpo.py Outdated
Comment thread nemo_rl/algorithms/grpo.py
Comment thread nemo_rl/algorithms/grpo.py Outdated
@arnavk-nvidia
adding changes to grpo_sync, update yaml configs, tests and documenta…
9217dac 
…tion

Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
@arnavk-nvidia
arnavk-nvidia requested review from a team as code owners June 5, 2026 05:30
@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Jun 5, 2026
@arnavk-nvidia
arnavk-nvidia requested a review from yfw June 5, 2026 05:35
@arnavk-nvidia

arnavk-nvidia commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 9217dac

yfw
yfw approved these changes Jun 5, 2026
View reviewed changes

@yfw yfw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks!

@yfw
yfw merged commit aeb55b5 into main Jun 5, 2026
75 of 76 checks passed
@yfw
yfw deleted the feat/grpo-advantage-clip-overlong-filter branch June 5, 2026 18:12
pengdurice pushed a commit to pengdurice/RL that referenced this pull request Jun 12, 2026
@arnavk-nvidia @yfw
@HeyyyyyyG @pengdurice
feat: grpo advantage clip overlong filter (NVIDIA-NeMo#2686)
887e265 
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
Signed-off-by: Arnav Kundu <arnavk@nvidia.com>
Co-authored-by: Yi-Fu Wu <yifu.wu@gmail.com>
Co-authored-by: Jiaqi Zeng <jiaqiz@nvidia.com>
@Alvorecer721 Alvorecer721 mentioned this pull request Aug 11, 2026
Merged
@yfw yfw mentioned this pull request Aug 31, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yfw yfw yfw approved these changes

Assignees

No one assigned

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arnavk-nvidia @yfw