fix: AUT-2228 route partial reruns to the current attempt - #4048
Merged
chtruong814 merged 1 commit intoSep 8, 2026
Merged
Conversation
Signed-off-by: svcnemo-autobot <svcnemo-autobot@nvidia.com>
chtruong814
approved these changes
Sep 8, 2026
chtruong814
enabled auto-merge (squash)
September 8, 2026 17:41
svcnvidia-nemo-ci
added
CI:L1
Contributor
|
/ok to test 136e53c |
rrs45
added a commit
that referenced
this pull request
Sep 8, 2026
Picks up periodic rollout checkpointing (#3924) and the partial-rerun routing fix (#4048), which landed while the previous merge was being verified. One conflict, in the critic training block: main now raises _optimizer_commit_in_progress before the value_training timer so a periodic snapshot cannot land mid-update. Kept, with our rl.sc.value_training span re-applied around the timer and the flag left where main set it -- outside the span, since it guards the whole irreversible update rather than the measured region. Signed-off-by: Raj Singh <rajsin@nvidia.com>
asolergi-nv
added a commit
that referenced
this pull request
Sep 9, 2026
Two conflicts, both pure adjacency -- independent additions landing at the same file position, so both sides are kept: - nemo_rl/algorithms/single_controller.py: this branch's _SUPERVISOR_DRAIN_TIMEOUT_S constant sits where #3924 (periodic rollout checkpointing) added _MAX_CONSECUTIVE_ROLLOUT_CHECKPOINT_FAILURES and the _RolloutCheckpointCut dataclass. Neither refers to the other. - tests/unit/environments/test_nemo_gym_utils.py: #4014 (L2 Gym-to-RL rollout acceptance) appended a spinup-cleanup test to the same end-of-file this branch appended TestUnresolvedAgentRefsAreDiagnosable to. main's function first, this branch's class after it. The other three commits (#3730 colocated MInf, #4002 actor venv list from a Python leaf module, #4048 partial rerun routing) merged clean. Verified after the merge: the restart keys are still in both SC exemplars, on_dead_shard is still absent from the config and both yamls, and 1287 tests pass across single_controller, fleet_health, weight_sync and the gym utils. Signed-off-by: asolergibert <asolergibert@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Partial reruns must target runners allocated for the current workflow attempt.
What changed
Updated the ephemeral router to v1.9.2, passed its attempt-independent base through organization-member preflight, and reconstructed attempt-qualified labels at 9 scheduling boundaries. Static and GB200 labels remain unchanged.
Details
Legacy labels remain unchanged.
Tested
Passed YAML parsing, routing/output wiring and boundary-count checks, whitespace, size, conflict-marker, and private-key scans. Full pre-commit could not start because the typed checkout does not expose Git metadata; CI remains authoritative.