Repository navigation
prime-runs - #3459
Merged
Merged
prime-runs#3459
Conversation
Replace the hand-rolled TrainRun (RFT lifecycle, per-step metrics POSTs, every-10th-step Parquet presign->PUT->confirm, atexit failure marking) with prime_runs' pr.init(kind="train"): the SDK owns the uploads on a background thread with retries and backpressure, reports crashed on a process that exits without finalizing, and adds a Prime Traces sink for allowlisted accounts. episodes_to_parquet_bytes and SAMPLE_SCHEMA moved into prime_runs.projection; the kind/subset cohort filter stays here, the step cadence moves into the SDK's training samples sink. TEMPORARY: prime-runs is not on PyPI yet; pinned to the prime repo's feature/prime-runs-train-backend branch (#856 + #873) via uv source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGeqWNdRB8ckrgch1tqVzJ
…tests - pyproject/uv.lock: the temporary git source points at the SDK branch head (aba26790) instead of a commit that only a merged PR ref still reaches; it picks up the per-upload sample id ranges, the training sink cooldown, replayable metrics and null handling for non-finite values. Dropped once 0.1.0 is on PyPI. - orchestrator: with an online platform run, episodes carry the platform's run id, so the run's traces can be queried by the id the dashboard shows. W&B keeps the launcher's PRL_RUN_ID either way. - monitor: log_metrics hops off the loop like log_episodes (a queue put can block under backpressure); the PRIME_RUNS_MODE value passes straight through; _base_url only strips /rft (the SDK strips /api/v1); a 60 s finish_timeout bounds the drain on finish and on the atexit crash path. - docs/training.md: the platform-monitoring section describes the SDK, the team requirement, PRIME_RUNS_MODE=disabled and RUN_ID attach. - tests/unit/monitors: first coverage of the monitor (init kwargs, attach, disabled switch, base URL). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
mikasenghaas
reviewed
Sep 2, 2026
Adopting the platform run id for episode provenance coupled the orchestrator's identity to which monitor came up online. prime-runs now sets `run.id`/`run.type` on the uploaded copy of every episode (keeping `work`, the dispatch step), so Prime Traces documents carry the dashboard's id without the orchestrator learning it, and `self.run_id` is back to the launcher's `PRL_RUN_ID` (or a local uuid) as on main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHMjWstGWd1oGKjoj5RgeT
kcoopermiller
added a commit
to PrimeIntellect-ai/prime
that referenced
this pull request
Sep 4, 2026
## Why
Eval and training jobs need one call that opens a run on the platform, streams records while they execute, and marks the run finished, failed, cancelled, or crashed when they stop. verifiers and prime-rl each hand-roll that lifecycle against the raw API today — two copies of the same thing — and nobody uploads the config a run was actually launched with, which is why the Config tab shows a v0 blob instead of anyone's `eval.toml`.
## What
A new leaf package, `prime-runs` (`httpx` + `prime-traces`; `pyarrow` behind a `train` extra), where the run is an object:
```python
import prime_runs as pr
run = pr.init(name="gsm8k-qwen3-8b", environments=["gsm8k"],
model=config.model, framework="verifiers",
config="eval.toml") # stored byte-for-byte
print(run.url)
for episode in rollouts:
run.log_episodes([episode]) # a queue put; streams on a background thread
run.finish(summary=pr.metrics.from_episodes(episodes))
```
Training runs use the same handle over the RFT external-runs API (folded in from #873):
```python
run = pr.init(kind="train", model=base_model, environments=[env_id],
training=pr.TrainingSpec(max_steps=1000, batch_size=64),
config=cfg.model_dump(), team_id=team) # or id=os.environ["RUN_ID"] to attach
run.log_episodes(step_episodes) # every 10th step -> Parquet presign / PUT / confirm
run.log_metrics(metrics, step=step) # POST /rft/metrics, on its own uploader
run.finish() # idempotent finalize; failed/cancelled/crashed -> status PUT
```
`Run` exposes `id`, `url`, `kind`, `log_episodes()` / `log_traces()` / `log_metrics()`, `update_summary()`, `flush()`, `finish()`, `fail()`, a context manager (an exception → `failed`, Ctrl-C → `cancelled`, an exit without `finish()` → `crashed` via atexit), and loss counters (`dropped_records`, `failed_records`, `errors`).
Records are keyed to the run at upload: the traces sink sets `run.id` / `run.type` on the uploaded copy of every trace and episode (member traces included) and keeps the producer's `name` / `work`, so verifiers and prime-rl keep their own local run ids and never learn the platform's.
Every record is written to two places during the transition: Prime Traces (the system of record; an account outside the beta has that sink retire quietly) and the table today's viewer reads — the v0 sample table for evals, per-step Parquet objects for training. Once the viewers read traces natively, the legacy sink is one line to drop and no producer changes.
How it degrades: queued batches coalesce so the request rate tracks upload latency; a full queue drops (counted) rather than stalling the run; a transient failure costs its batch and three in a row retire the sink — a training run pauses it for five minutes instead, since it outlives any platform deploy; non-finite numbers become `null` in the sample tables rather than losing the row. `POST /evaluations/`, `/rft/external-runs` and `/samples` are never replayed after an ambiguous failure (a retry could duplicate); presign, confirm, `/rft/metrics` (one row per step server-side) and `/rft/finalize` are.
Integrations: [verifiers#2415](PrimeIntellect-ai/verifiers#2415) (eval), [prime-rl#3459](PrimeIntellect-ai/prime-rl#3459) (training).
## Known gaps
- **No producer-facing way to mark an evaluation FAILED** in the API as deployed today: `finalize` only moves PROCESSING → COMPLETED, so a crashed or interrupted eval stays RUNNING on the dashboard. The SDK records status/error under `metadata.prime_runs` and warns. platform#4740 adds `PUT /evaluations/{id} {status: FAILED|CANCELLED, error_message}` with a 409 guard; once deployed, `EvalsBackend.finalize` should use it and keep the metadata fallback for older backends (`RunStatus.CANCELLED` already exists for that).
- **`config_source` is stored, not yet shown.** `init(config=<path>)` keeps the launched file byte-for-byte under `metadata.config_source` (`run_config.config_source` for training). The eval Config tab renders it once platform#4740 lands; verifiers#2415 sends only the three v0 keys for now and adds the file in a follow-up, and the training Config tab needs the same branch.
- **Attach targets external runs.** `init(kind="train", id=...)` works for runs created through `POST /rft/external-runs`; the v1 monitoring endpoints answer 400 for a hosted run's id (the same endpoints prime-rl's old monitor used, so nothing regresses).
- **Training samples are keyed by dispatch step.** Objects are additive per step on the platform, so an off-policy episode landing in a later batch adds an object for its dispatch step; each object numbers its `sample_id`s in its own range (a forked child in a random band). "Step N" in the sample viewer means dispatched at N, while the metrics at N describe the batch trained at N.
## Before merge
1. ~~prime-traces 0.0.3 on PyPI (#872), then `prime-traces>=0.0.3` here.~~ Done.
2. A PyPI **pending publisher** for `prime-runs` (owner `PrimeIntellect-ai`, repo `prime`, workflow `release-runs.yml`, environment `pypi-prod`). The name is unclaimed; the first run of the release workflow creates the project. The workflow publishes before it tags, so a failed first publish leaves no tag and re-runs from `main` via `workflow_dispatch`.
3. ~~Coalesce queued sample batches, add `RunStatus.CANCELLED`, fold #873 in so 0.1.0 carries the training backend.~~ Done.
4. Squash-merge (`main` requires linear history); `release-runs.yml` publishes 0.1.0 with eval + train. verifiers#2415 and prime-rl#3459 then drop their `[tool.uv.sources]` git pins and relock.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> New client code talks to production eval/RFT/traces APIs with nuanced retry and upload semantics; mistakes could duplicate runs, drop samples, or mis-report terminal status, though behavior is heavily tested and defaults to warn-not-raise.
>
> **Overview**
> Introduces **`prime-runs`**, a new leaf Python package (`httpx` + `prime-traces`, optional `pyarrow` for training) that centralizes platform run lifecycle: **`init()`** registers or attaches to a run, **`log_episodes` / `log_traces` / `log_metrics`** stream records on background uploaders, and **`finish()`** drains queues and closes the run (with context-manager, atexit crash reporting, and fork-safe HTTP/traces clients).
>
> **Eval** runs use **`EvalsBackend`** (`/evaluations`, hub environment resolve) and dual sinks to Prime Traces plus the legacy v0 sample table; **training** runs use **`RftBackend`** (`/rft/external-runs`, attach via `id=`), Parquet sample uploads on a step cadence, and a separate metrics uploader. The SDK stamps platform **`run.id` / `run.type`** on records, stores launch **`config_source`** verbatim, and implements deliberate retry/idempotency (no replay on ambiguous `POST` creates or append-only samples).
>
> **Repo wiring:** CI gains **`test-runs`** (Python 3.11–3.13) and version-bump enforcement for `packages/prime-runs`; **`release-runs.yml`** publishes to PyPI and tags `prime-runs-v*`. Hermetic tests cover HTTP, backends, sinks, projection, and end-to-end `init()` flows.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b65b9a. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
prime-runs 0.1.0 is published (prime#856 → prime-runs-v0.1.0), so the TEMPORARY git source is gone; `prime-runs[train]>=0.1.0` now resolves from PyPI and the lock carries prime-runs 0.1.0 + prime-traces 0.0.3 from the registry. The relock also picks up the SDK's run re-keying in the traces sink (prime#856 1b65b9a1), which the git pin predated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
kcoopermiller
marked this pull request as ready for review
September 4, 2026 03:58
This was referenced Sep 4, 2026
Merged
0.1.1 (prime#895 via prime#896) teaches the SDK's platform client the internal RFT root a hosted run's launcher injects as $PRIME_API_BASE (`…/api/internal/rft`): requests go under /api/internal and the run's token also travels in x-api-key, which is what that router authenticates on. The monitor's `_base_url()` already passed that root minus `/rft`, so nothing here changes besides the floor; the relock moves prime-runs only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
samsja
reviewed
Sep 8, 2026
Requested in review: the monitor is a thin adapter over prime-runs and the SDK covers the lifecycle itself. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HpQPfQPmow2Vaq6qHZZZjZ
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 163cd60. Configure here.
samsja
previously approved these changes
Sep 8, 2026
Move the verifiers submodule from 828488fff (Sep 3) to 392d596de, which carries verifiers #2415 (eval runs streamed through prime-runs), and refresh uv.lock: verifiers now declares prime-runs>=0.1.1, already a direct dependency here. No prime-rl code change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0119wW9JQxDuDrLaUBwJB3Bk
samsja
approved these changes
Sep 9, 2026
mikasenghaas
added a commit
that referenced
this pull request
Sep 14, 2026
Main moved the prime monitor onto prime-runs (#3459) and renamed the admin clients to AdminPlane (#3472). The eval monitor now opens one prime-runs eval run per finished epoch instead of posting to /evaluations by hand, which retires prime_rl.monitors.platform: the SDK's projection and metrics modules own the sample format and the run aggregates. The eval runner takes the AdminPlane rename and the error-rate line from #3462. The swe example stays under examples/eval (main had moved it to configs/evals for `uv run evals`), and the verifiers submodule follows feat/remove-eval-cli's merge of main. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Replaces
monitors/prime.py'sTrainRunwithpr.init(kind="train", ...).What moved where
TrainRun.create/$RUN_IDattachpr.init(kind="train", model=, environments=, training=pr.TrainingSpec(...), config=, ...)/pr.init(kind="train", id=$RUN_ID)TrainRun.log_metrics(fire-and-forget task per POST)run.log_metrics(values, step=)— a queue put; the SDK's uploader owns retries,Retry-After, backpressureepisodes_to_parquet_bytes+SAMPLE_SCHEMA+ presign→PUT→confirmprime_runs.projection+ the SDK's training samples sink (same schema, same flow)step % 10cadence inlog_episodesstep_interval=10), keyed on each episode'srun.work.step— theTrainRunInfothe dispatcher stampsfinalize→POST /finalize, fallbackset_statusrun.finish()— drain, then an idempotent finalize; 409 on the failure path is treated as already-closed_mark_failedThe
kind == "train" and subset == "effective"cohort filter stays here. The monitor keepssanitize()so dropped non-finite metric paths are still named in the log (the SDK would drop them silently).$PRIME_API_BASEstill works — it historically points at…/api/v1/rft, so the monitor strips what the SDK re-adds.Decisions
run.work.step), not the batch step; the platform keeps one Parquet object per upload and the viewer unions every object under a step, so an off-policy episode landing in a later batch adds an object rather than replacing one, and each object numbers itssample_ids in its own range (SDK side,aba26790). "Step N" in the sample viewer therefore means dispatched at N, while the metrics at N describe the batch trained at N.PRL_RUN_ID(or a local uuid) as the run identity for episodes, W&B and sandbox labels, exactly as onmain. The SDK keys every uploaded record to the platform run (run.id/run.typeon the uploaded copy,workkept), so Prime Traces documents carry the id the dashboard shows without the orchestrator learning it.Dependency
prime-runs[train]>=0.1.1from PyPI (0.1.1, PrimeIntellect-ai/prime#896); the lock carriesprime-runs 0.1.1+prime-traces 0.0.3from the registry. The monitor's own unit tests run in CI; there is no GPU-free end-to-end path for the training seam, so the RFT contract was verified by reading the platform's/api/v1/rft/*and/api/internal/rft/*routes against the SDK's calls.Hosted managed runs
Since PrimeIntellect-ai/platform#4885, hosted managed runs start the orchestrator with
--monitors.prime.name,RUN_ID, a per-run token inPRIME_API_KEYandPRIME_API_BASE=<platform>/api/internal/rft. prime-runs 0.1.1 (PrimeIntellect-ai/prime#895) addresses that internal router and sends the run's token inx-api-key, the header it authenticates on; 0.1.0 only spoke/api/v1with a Bearer header and would have left those runs silently unmonitored. This monitor's_base_url()passes$PRIME_API_BASEminus the trailing/rft, which 0.1.1 accepts, so the floor bump is the whole change. External runs (prime-rl launched outside the platform) were never affected.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
Note
Medium Risk
Changes how training runs register and upload to the Prime platform (including hosted
$RUN_ID/ internal RFT paths); misconfiguration now fails launch instead of silently skipping monitoring, but orchestrator run identity for W&B stays unchanged.Overview
Replaces the in-repo
TrainRunHTTP/Parquet implementation inPrimeMonitorwithprime-runs[train]>=0.1.1, so run registration (or$RUN_IDattach), metrics streaming, episode sample uploads, finalize, and crash reporting are owned by the SDK uploader instead of customhttpx+ fire-and-forget tasks.The monitor layer is now a thin adapter:
pr.init(kind="train", …)on startup,run.log_metrics/run.log_episodesviaasyncio.to_thread, andrun.finish()with a 60s drain timeout. Local behavior kept where it matters: non-finite metrics are still sanitized with named warnings, only train / effective episodes are forwarded, and$PRIME_API_BASEis normalized for the SDK. Episode every-10th-step upload cadence moves into the SDK sink (keyed on dispatch step onrun.work).Docs in
training.mddescribe the SDK, team allowlisting,PRIME_RUNS_MODE=disabled, andRUN_IDattach. Lock/pyproject addprime-runsand transitiveprime-traces, with fast-track resolver entries.Reviewed by Cursor Bugbot for commit aed9bbf. Bugbot is set up for automated code reviews on this repo. Configure here.