feat(rollout): add periodic partial-rollout recovery snapshots

[macandro96]

What does this PR do?

Adds frequent partial-rollout snapshots and makes streamed optimizer steps recoverable from their last durable trainer anchor.

• Adds rollout_checkpointing.interval_s, keep_latest_k, and restore_mode.
• Captures native TQ state, replay metadata, rollout lineage, and the dataloader cursor as one checkpoint-barrier cut.
• Supports bootstrap snapshots before step 1 and anchors later snapshots to the latest durable trainer checkpoint.
• Publishes through a temporary directory, validation, COMMITTED, atomic rename, and an atomically updated LATEST pointer.
• Retains the previous committed snapshot when a periodic save fails.
• Allows periodic snapshots while a streamed optimizer step is active.
• Keeps training-claimed rows durable in TQ until the optimizer step finishes.
• Persists claimed groups as replayable FINALIZED ownership, so restart discards uncheckpointed gradients and replays the complete step from its durable trainer anchor.
• Preserves restored target-step dispatch and validates sampler capacity to avoid rollout-pump deadlocks.
• Includes crash/restart functional coverage for sibling reuse and streamed-step snapshots.

Stack

• Layer 3 of 4.
• Previous: feat(rollout): persist and recover partial rollout groups #3507.
• Next: feat(rollout): add partial recovery benchmark telemetry #3585.

Configuration

checkpointing:
  enabled: true
  metric_name: null

data_plane:
  checkpointing_enabled: true

token_capture:
  enabled: true

rollout_checkpointing:
  interval_s: 60
  keep_latest_k: 2
  restore_mode: latest

interval_s: null disables periodic rollout snapshots.

Bootstrap snapshots live under bootstrap/rollout_snapshots/; later snapshots live under the matching durable step_N/rollout_snapshots/ anchor.

Streamed-step recovery

If a snapshot is taken after 2 of 8 prompt groups have contributed uncommitted gradients, the snapshot retains those rows as replayable finalized ownership. Restart loads the durable trainer anchor, discards the uncheckpointed 2/8 gradients, and trains all eight groups exactly once.

Validation

• Unit coverage includes periodic scheduling, checkpoint cuts, atomic publication, retention, failure fallback, active streamed steps, restored dispatch state, and bootstrap recovery.
• Functional scripts cover partial sibling reuse and active streamed-step rollback/replay.
• The PR remains draft while the refreshed functionals and full CI are rerun.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

✅ Submodule Fast-Forward Check Results

Check based on commit: 18d67cd (PR #3508 from amahishi/partial-rollout-ledger)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of amahishi/sc-tq-token-capture-recovery branch (fast-forward)

All submodule changes look good! ✨

[macandro96]

Will be superseded by #3924