feat(grpo): add val_start_at to delay periodic validation - #3400
Merged
Conversation
Adds grpo.val_start_at (NotRequired): the first training step eligible for periodic validation. Absent keeps today's behavior exactly. When set, periodic validation (val_period) only fires once step >= val_start_at; val_at_start and val_at_end are unaffected. Motivation: on long-horizon agentic RL (e.g. SWE agents), early-training validation is expensive (hours of rollouts) and uninformative because the policy cannot plausibly have reached a useful accuracy yet. A validation lower bound recovers that wall-clock without changing the cadence afterwards: with val_period=2 and val_start_at=3, validation runs at steps 4, 6, 8, ... Covered by test_periodic_validation_starts_at_configured_step for both grpo_train and async_grpo_train, including the val_at_end interaction. Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
yuki-97
reviewed
Jul 29, 2026
yuki-97
left a comment
yuki-97
left a comment
Contributor
There was a problem hiding this comment.
thanks @michal2409 , overall LGTM and left some minor comments.
25 tasks
Address review on NVIDIA-NeMo#3400: - val_start_at is now a required GRPOConfig int mirroring val_period's convention (-1 disables the delay). The default is recorded in every standalone GRPO-family root config (grpo_math_1B.yaml, the nemo_gym roots, and the nemotron-3-super stage configs — all other grpo YAMLs inherit one of these via defaults:) plus the v2 reference config. - Trainers direct-read master_config.grpo["val_start_at"] and the guards drop the None branch: (step + 1) >= val_start_at. - test_periodic_validation_starts_at_configured_step now also covers grpo_train_sync via a new mock_sync_grpo_infrastructure helper that stubs the SyncRolloutActor and the TQ data-plane policy surface so the sync driver loop runs for real. Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
yuki-97
previously approved these changes
Jul 30, 2026
yuki-97
left a comment
yuki-97
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks @michal2409
Contributor
|
/ok to test 0a4e31a |
yuki-97
enabled auto-merge (squash)
July 30, 2026 05:11
research/template_project/single_update.py validates the full pydantic MasterConfig, so its config copy needs every required grpo key too. Fixes the L1_Functional_Tests_Other_1 failure. Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
michal2409
added a commit
to michal2409/RL
that referenced
this pull request
Jul 30, 2026
research/template_project/single_update.py validates the full pydantic MasterConfig, so its config copy needs every required grpo key too (same failure mode NVIDIA-NeMo#3400 hit in L1_Functional_Tests_Other_1). Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
michal2409
added a commit
to michal2409/RL
that referenced
this pull request
Jul 30, 2026
research/template_project/single_update.py validates the full pydantic MasterConfig, so its config copy needs every required grpo key too (same failure mode NVIDIA-NeMo#3400 hit in L1_Functional_Tests_Other_1). Signed-off-by: Michal Futrega <mfutrega@nvidia.com>
Contributor
Author
|
The L1 failure was |
This was referenced Jul 30, 2026
Contributor
Author
|
/ok to test 127db4d |
yuki-97
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
grpo.val_start_at: the first training step eligible for periodic validation. It mirrorsval_period's required-int convention:-1(the exemplar default) disables the delay and keeps today's behavior exactly. When set, periodic validation (val_period) only fires oncestep >= val_start_at;val_at_startandval_at_endare unaffected.