Skip to content

feat(rollout): add stable partial-rollout lineage - #3506

Draft
macandro96 wants to merge 6 commits into
amahishi/sc-tq-token-capture-recoveryfrom
amahishi/partial-rollout-lineage
Draft

feat(rollout): add stable partial-rollout lineage#3506
macandro96 wants to merge 6 commits into
amahishi/sc-tq-token-capture-recoveryfrom
amahishi/partial-rollout-lineage

Conversation

@macandro96

@macandro96 macandro96 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds the durable identity and in-memory lineage foundation for recovering partially completed NeMo-Gym prompt groups.

  • Gives each prompt group and GRPO sibling a stable logical identity.
  • Tracks physical Gym/Gate attempts separately, so redispatch can create a new attempt without changing canonical training-row identity.
  • Records sibling receipts, rewards, and TQ staging keys as soon as Gym seals them.
  • Models ownership through generation, finalization, training claim, and uncheckpointed application.
  • Preserves sealed siblings when another sibling fails and abandons only unfinished attempts.
  • Stores compact attempt UUIDs and small PromptRef values while excluding full runtime prompts from serialized lineage.
  • Validates the versioned serialized state and rejects unsupported or invalid statuses.

This layer intentionally does not persist or restore the ledger. Persistence and selective redispatch are added by #3507.

Stack

Scope

This is internal plumbing for the existing token_capture.enabled=true Single Controller NeMo-Gym path. It adds no user-facing configuration.

Tensor payloads remain in TQ. The controller ledger contains identity, ownership state, compact prompt references, receipts, rewards, and storage references.

Validation

  • Unit coverage is included for identity, lineage transitions, retries, finalization ownership, training claims, serialization, and invalid persisted states.
  • The PR remains draft while the refreshed focused suite, recovery functionals, and full CI are rerun.

@copy-pr-bot

copy-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@macandro96 macandro96 changed the title amahishi/partial rollout lineage feat(rollout): add stable partial-rollout lineage Aug 5, 2026
@github-actions

Copy link
Copy Markdown

✅ Submodule Fast-Forward Check Results

Check based on commit: 95f0101 (PR #3506 from amahishi/partial-rollout-lineage)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of amahishi/sc-tq-token-capture-recovery branch (fast-forward)

All submodule changes look good! ✨

@macandro96
macandro96 force-pushed the amahishi/partial-rollout-lineage branch from 95f0101 to 9ff9088 Compare August 20, 2026 15:28
@github-actions github-actions Bot added Documentation Improvements or additions to documentation CI Relating to CI labels Aug 20, 2026
terrykong and others added 6 commits August 21, 2026 20:59
Signed-off-by: asolergibert <asolergibert@nvidia.com>
Co-authored-by: asolergibert <asolergibert@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit e983576)
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
(cherry picked from commit 52e2dbc)
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
(cherry picked from commit 0726648)
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
Signed-off-by: Anish Mahishi <amahishi@nvidia.com>
@macandro96
macandro96 force-pushed the amahishi/partial-rollout-lineage branch from 9ff9088 to 478c894 Compare August 24, 2026 21:03
@github-actions github-actions Bot removed the CI Relating to CI label Aug 24, 2026
@macandro96

Copy link
Copy Markdown
Contributor Author

Will be superseded by #3923

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants