fix: clean stale rollouts and broadcasts in SLURM template - #2303
Closed
samsja wants to merge 1 commit into
Closed
Conversation
When restarting a multi-node RL run, stale rollouts and broadcasts from the previous run under the orchestrator output dir could cause issues. Remove them at job start.
samsja
marked this pull request as ready for review
April 17, 2026 15:24
Member
|
fixed in #2304 |
mikasenghaas
added a commit
that referenced
this pull request
Aug 10, 2026
deps/verifiers -> main 91d4f69e6 (#2303 merged); research-environments -> the rename branch head with the automationbench_env collision fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
added a commit
that referenced
this pull request
Aug 11, 2026
…ts parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
added a commit
that referenced
this pull request
Aug 11, 2026
* feat!: drop v0 env compat from the orchestrator and configs The verifiers legacy bridge is removed upstream, so every v0 code path goes: - configs: drop the [legacy] block, is_legacy, the legacy/env conflict validators, and resolve_legacy_env_kwargs from EnvConfig and EnvServerConfig; env-server TOMLs no longer carry a legacy section - env-server entrypoint: always serve a taskset (no bridge fork) - orchestrator: Env always loads its taskset client-side (info()/ run_group()/task_idx addressing removed); requires_group_scoring and its consumers (variable permit costs, group-scored partial drops, run_group fan-out) are gone — one permit is one episode everywhere - TrainSource rows always carry the task; the pending-draw head-of-line hold and per-env permit costs are gone (next_example takes no args) - GroupState.task is required; task_idx derives from task.data.idx - env taskset ids drop the -v1 suffix across configs, examples, docs, skills, and the workspace members (companion renames in verifiers and research-environments) - port the qwen3-vl features-payload test from the v0 RendererClient to renderers.client.generate (the path the v1 train client drives) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers + research-environments to the rename branches and relock deps/verifiers -> feat/legacy-package (6a8de86), deps/research-environments -> feat/drop-v1-suffix (27ed790). Re-pin to the merged commits before landing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin to the reworked branch head v0 code stays in place upstream; the branch now only removes the v1 legacy bridge and the -v1 suffixes (lock unchanged). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (v1 CLI simplifications) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (main merged, python floors) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (v1 drops on-demand Hub installs) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (v0 tree under verifiers.legacy, aliased) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (ID plain str, public alias names) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (ruff fixes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (static-tooling fixes) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (gepa schema id fix) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (docs/legacy move) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers pin (hub installs restored in v1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: pin verifiers to the merged main commit deps/verifiers -> main 91d4f69e6 (#2303 merged); research-environments -> the rename branch head with the automationbench_env collision fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: floor verifiers at 0.3.1.dev12 The first dev release carrying the v0 deprecation (verifiers#2303); submodule follows main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: verifiers submodule points at main head (the #2303 merge), not its parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: pin submodules to the merged mains and floor verifiers at 0.3.1.dev13 deps/verifiers -> f24b8ec0 (#2311), deps/research-environments -> c0a0d1d7a (#754); 0.3.1.dev13 is the release cut from that verifiers commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump verifiers to main d53ab56f (0.3.1.dev14) Picks up the model-catalog revert (verifiers#2314). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
$ORCHESTRATOR_OUTPUT_DIR/rolloutsand$ORCHESTRATOR_OUTPUT_DIR/broadcastsat the start of multi-node SLURM jobs to prevent stale data from previous runs causing issues.Note
Low Risk
Low risk: only adds a pre-run cleanup step in the SLURM batch template; main concern is accidental deletion if
ORCHESTRATOR_OUTPUT_DIRis misconfigured.Overview
Prevents stale orchestrator artifacts from previous runs by adding a startup cleanup in
multi_node_rl.sbatch.j2thatrm -rfs$ORCHESTRATOR_OUTPUT_DIR/rolloutsand$ORCHESTRATOR_OUTPUT_DIR/broadcastsbefore creating log directories and launching the job.Reviewed by Cursor Bugbot for commit 571a262. Bugbot is set up for automated code reviews on this repo. Configure here.