[mikasenghaas]

Summary

Part of the monitors stack: #4228 → #4316 → #4317 → #4244. Base of the stack.

• One prime monitor. PrimeMonitor and PrimeMonitorConfig replace the train and eval pair. [monitors.prime] and --no-monitors.prime are unchanged.
• One monitors config. MonitorsConfig (wandb, file, prime) replaces TrainMonitorsConfig and EvalMonitorsConfig. The online-eval process takes the trainer's monitors as they are. The RL trainer gets a monitors.prime field, which it doesn't set up yet.
• The process that supervises the launch registers the Prime run, and every component attaches to it. This is how a hosted launch works: the platform pre-creates the run, passes RUN_ID to each pod, and sets the final status from the JobSet.
  • Local: the rl or sft launcher.
  • SLURM, single node: the rl or sft launcher inside the job. The submitting process registers nothing.
  • SLURM, multi node: python -m prime_rl.entrypoints.run_owner {rl,sft} <config> srun ... wraps the job's launch step. It fails the run when the step exits non-zero, or when SLURM sends SIGTERM (cancel, time limit).
• The Prime run gets the whole launch config. An rl launch registers the full RL config (trainer, inference, orchestrator, deployment), not only the orchestrator's. Multi-node SLURM jobs write the full rl.json and sft.json for the run owner. The multi-node SFT trainer runs its own trainer.json, as the RL components already do.
  • The run's id becomes the run identity: PRL_RUN_ID, and with it the W&B run id and the episodes' run.id. The launcher also exports it as RUN_ID, which each component's prime monitor attaches to.
  • A launch that fails marks the run FAILED with the failed component's error, e.g. RuntimeError: Trainer failed with exit code -9. The SDK leaves an attached run's failure to the process that registered it.
  • A component started without a launcher still registers its own run.
• SFT online evals log to the same run, like RL. The online-eval process uploads its eval metrics and every eval episode to the sft run. Before, it opened one Prime evaluation per eval epoch.
  • The process that registered the run finishes it once every component exited, and fails it when a component fails. Attached components only flush, so no component can finish the run while another still logs to it. On hosted training the platform sets the final status from the JobSet.
  • A paused run no longer gets reported as crashed: the SDK's exit hook is dropped after the flush.
• Standalone uv run eval still opens one Prime evaluation per source and epoch.
  • Without a Prime API key, it fails at launch with the prime login hint, like rl and sft. The evaluations open later, when each epoch starts.
• Fewer helpers.
  • The links record is keyed by the monitor's config field (prime, wandb), and run_links takes the MonitorsConfig.
  • The platform record is written through update_record.
  • register takes the config directly and turns the SDK's missing-key error into the prime login hint.
• The orchestrator's monitor producer is orchestrator, the name every other place uses (orchestrator.log, the W&B labels).

Breaking

• SFT online evals no longer create Prime evaluations. Their results are on the sft run's Prime page, and an sft run's monitors/prime/run.json no longer lists evaluations.
• With the prime monitor on, PRL_RUN_ID (and so the W&B run id) of an rl or sft launch is the Prime run's id instead of a random uuid.
• A custom multi-node SLURM template (slurm.template_path) must wrap its launch srun in python -m prime_rl.entrypoints.run_owner {rl,sft} <config>, as the bundled templates do. Without it, nothing registers the shared run.
• A multi-node SFT job's sft.json is now the full config. The trainer reads trainer.json; a custom multi-node SFT template must pass {{ config_path }} (the trainer's) to the trainer and {{ run_config_path }} to the run owner.
• The Prime run's config of an rl launch is the full RL config instead of the orchestrator config.
• An RL config can no longer turn monitors.wandb or monitors.file on in one component only: RLConfig rejects it ("must match across components"). Set the shared [monitors.<name>] instead.
• A hosted orchestrator no longer finishes its attached run; the platform marks it COMPLETED from the JobSet, as it already did.
• The orchestrator's file-monitor producer is orchestrator instead of orch. Its annotation stream moves from monitors/file/traces/annotations/orch to .../annotations/orchestrator, and its metrics.jsonl rows carry "producer": "orchestrator".

Verification

Stack head (#4244 at 450707973), 2026-10-30:

Run	Prime	Result
Hosted RL, reverse-text, 10 steps (prime train --pr 4244 --image-tag v0.10.1.dev143)	u68lwuthm9f7e87fxb2pnq5f	COMPLETED by the platform; trainer and orchestrator attached via $RUN_ID
Hosted SFT, fake data, online evals every 5 steps	uvmvu17v6bks0kbip83xubnd	COMPLETED; online evals at steps 0, 5, 10 in the same run
Local rl, reverse-text, 5 steps	g4n7oc8hqhz2pfqfi5hj9nno	COMPLETED by the launcher; W&B run has the same id
Local sft, 10 steps, evals every 5	dxz2k1etkdbh3go026y96td4	COMPLETED by the launcher; eval metrics at steps 0, 5, 10
Local eval	evaluation	exit 0

On hosted runs the platform stores the submitted TOML as the run's config; the full resolved config is registered for local and SLURM launches only.

Run ownership:

• Local RL, 3 steps: qxnhsl4ngd9qocvlcqzbbquw COMPLETED. The orchestrator only flushed; the launcher finished the run.
• run_owner with a step that exits 0: sppd7r2l1kdposmws2tarp8g COMPLETED.
• SIGTERM to run_owner while the step runs: the step received the forwarded TERM and finished a 2 s teardown before the owner exited 143; rfev5w1qbxnxlko6dwd22apt FAILED with RuntimeError: SLURM terminated the job.
• uv run rl @ examples/basic/reverse-text/rl.toml --max-steps 3 --no-monitors.wandb, local: vtcuhcp16vigkskxu02d763l COMPLETED. Its run_config on the platform has trainer, inference, orchestrator and deployment.
• Multi-node dry runs: rl.json loads as a multi-node RLConfig and registers trainer, inference, orchestrator, deployment and slurm. sft.json loads as a multi-node SFTConfig, and the script's TRAIN_CONFIG_PATH is trainer.json.
• Multi-node RL and SFT dry runs (examples/advanced/qwen3-30b-a3b/math.toml, .../sft/h200/base.toml) render scripts that pass bash -n. run_owner loads the configs the scripts point to.
• run_owner rl <orchestrator.json> bash -s <<'STEP_SH' ... exit 3: the step saw RUN_ID=PRL_RUN_ID=<id>, the wrapper exited 3, and Prime shows run ekc8dcf8agutxcrizts1bkbe FAILED with RuntimeError: The SLURM job step exited with code 3.
• SIGTERM to a running run_owner: exit 143, and Prime shows run q4g9hv95qoqsl06yeacle2jd FAILED with RuntimeError: SLURM terminated the job.
• test_slurm_run_owner_fails_the_run_with_its_step: the owner registers the run (not attached), exports its id, and fails the run only when the step fails.

Latest head, local reverse-text:

Run	Prime run	Status
uv run rl @ examples/basic/reverse-text/rl.toml --max-steps 5 --no-monitors.wandb	kbv529xg6xoop15a0yf7ijwq	COMPLETED
uv run sft @ examples/basic/reverse-text/sft.toml --max-steps 10 --eval.interval 5 --no-ckpt	bwd98te1u76s82qqrloaty8i	COMPLETED; trainer and online eval attached via $RUN_ID
uv run eval @ configs/debug/eval/single-turn.toml	evaluation	banner shows the evaluation link

Earlier head, with episode counts and the failure path:

Run	Prime link	Local = Prime episodes	Arrival → ingested (p50 / max)	Status
uv run sft @ examples/basic/reverse-text/sft.toml --max-steps 10 --eval.interval 5 --no-ckpt	training run	96 eval (3 epochs × 32)	1.0 s / 1.9 s	COMPLETED
uv run rl @ examples/basic/reverse-text/rl.toml --max-steps 5 --no-monitors.wandb	training run	707 train + 512 eval	1.0 s / 1.8 s	COMPLETED
Failure path: the RL run above, with the trainer killed (SIGKILL) mid-run	training run	—	—	FAILED: RuntimeError: Trainer failed with exit code -9

• SFT run: the trainer and the online-eval process both logged Logging metrics and episodes to Prime run neakqq9sbr4s4o0942oeb2ys (...) (attached via $RUN_ID).
• RL run: the orchestrator attached through $RUN_ID. An episode read back from Prime Traces carries run.id = fnog09c4s2updmlso4omjoh7, so PRL_RUN_ID is the Prime id.
• Failure path: components attach to the launcher's run, and the SDK skips a failed finish from an attached run. So the launcher marks the run failed itself, with the component's error.
• Unit tests: pytest tests/unit --ignore=tests/unit/train tests/unit/train/sft/test_prime_monitor.py: 810 passed, with main merged.
• Hosted: not run on this head. fix: mark live config fields instead of freezing the rest #4318, which unblocked hosted RL on main, has merged.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes run identity, Prime lifecycle ownership, and multi-node SLURM wiring; custom templates or mixed monitor settings can break tracking or fail validation.

Overview
Unifies Prime and monitor configuration so training uses one platform run and one config shape everywhere.

Prime: PrimeTrainMonitor / PrimeEvalMonitor and separate train/eval monitor config types become PrimeMonitor and shared MonitorsConfig. The rl/sft launcher (or multi-node run_owner) registers the platform run via launch_run, sets PRL_RUN_ID / RUN_ID / W&B run id, and finishes or fails the run when the launch ends; worker processes attach and only flush. RL registration sends the full RL config, not orchestrator-only. SFT online evals now upload metrics and episodes to that same training run instead of opening per-epoch Prime evaluations; standalone uv run eval still opens one evaluation per env/epoch.

Launch / SLURM: Bundled multi-node templates wrap the job step in python -m prime_rl.entrypoints.run_owner. Multi-node SFT writes full sft.json for the owner and trainer.json for torchrun. run_links takes MonitorsConfig; link records use prime / wandb keys.

Other: RL validates monitors.wandb and monitors.file are on or off consistently on trainer and orchestrator. Orchestrator file-monitor producer is orchestrator (annotation path rename). Docs/skills updated accordingly.

^{Reviewed by Cursor Bugbot for commit 7e0c55f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7e0c55f. Configure here.}

[hallerite]

Reviewed the shared monitor configuration, run ownership, and local/SLURM launch wiring. The supervising process registering and finalizing the run, with components attaching and flushing, is a sound model.