Skip to content

prime-runs - #3459

Merged
kcoopermiller merged 13 commits into
mainfrom
cooper/prime-runs
Sep 9, 2026
Merged

kcoopermiller merged 13 commits into
mainfrom
cooper/prime-runs

Conversation

@kcoopermiller

@kcoopermiller kcoopermiller commented Sep 1, 2026 •

Copy link
Copy Markdown
Member

Replaces monitors/prime.py'sTrainRun with pr.init(kind="train", ...).

What moved where

was in prime-rl now
TrainRun.create / $RUN_ID attach pr.init(kind="train", model=, environments=, training=pr.TrainingSpec(...), config=, ...) / pr.init(kind="train", id=$RUN_ID)
TrainRun.log_metrics (fire-and-forget task per POST) run.log_metrics(values, step=) — a queue put; the SDK's uploader owns retries, Retry-After, backpressure
episodes_to_parquet_bytes + SAMPLE_SCHEMA + presign→PUT→confirm prime_runs.projection + the SDK's training samples sink (same schema, same flow)
the step % 10 cadence in log_episodes the sink (step_interval=10), keyed on each episode's run.work.step — the TrainRunInfo the dispatcher stamps
finalize → POST /finalize, fallback set_status run.finish() — drain, then an idempotent finalize; 409 on the failure path is treated as already-closed
atexit _mark_failed the SDK's atexit hook reports crashed (with a message); an attached run's failure marking stays with the launcher, as before

The kind == "train" and subset == "effective" cohort filter stays here. The monitor keeps sanitize() so dropped non-finite metric paths are still named in the log (the SDK would drop them silently). $PRIME_API_BASE still works — it historically points at …/api/v1/rft, so the monitor strips what the SDK re-adds.

Decisions

  • Step keying. The samples sink keys each episode by the step it was dispatched at (run.work.step), not the batch step; the platform keeps one Parquet object per upload and the viewer unions every object under a step, so an off-policy episode landing in a later batch adds an object rather than replacing one, and each object numbers its sample_ids in its own range (SDK side, aba26790). "Step N" in the sample viewer therefore means dispatched at N, while the metrics at N describe the batch trained at N.
  • Run identity. The orchestrator keeps the launcher's PRL_RUN_ID (or a local uuid) as the run identity for episodes, W&B and sandbox labels, exactly as on main. The SDK keys every uploaded record to the platform run (run.id / run.type on the uploaded copy, work kept), so Prime Traces documents carry the id the dashboard shows without the orchestrator learning it.

Dependency

prime-runs[train]>=0.1.1 from PyPI (0.1.1, PrimeIntellect-ai/prime#896); the lock carries prime-runs 0.1.1 + prime-traces 0.0.3 from the registry. The monitor's own unit tests run in CI; there is no GPU-free end-to-end path for the training seam, so the RFT contract was verified by reading the platform's /api/v1/rft/* and /api/internal/rft/* routes against the SDK's calls.

Hosted managed runs

Since PrimeIntellect-ai/platform#4885, hosted managed runs start the orchestrator with --monitors.prime.name, RUN_ID, a per-run token in PRIME_API_KEY and PRIME_API_BASE=<platform>/api/internal/rft. prime-runs 0.1.1 (PrimeIntellect-ai/prime#895) addresses that internal router and sends the run's token in x-api-key, the header it authenticates on; 0.1.0 only spoke /api/v1 with a Bearer header and would have left those runs silently unmonitored. This monitor's _base_url() passes $PRIME_API_BASE minus the trailing /rft, which 0.1.1 accepts, so the floor bump is the whole change. External runs (prime-rl launched outside the platform) were never affected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR


Note

Medium Risk
Changes how training runs register and upload to the Prime platform (including hosted $RUN_ID / internal RFT paths); misconfiguration now fails launch instead of silently skipping monitoring, but orchestrator run identity for W&B stays unchanged.

Overview
Replaces the in-repo TrainRun HTTP/Parquet implementation in PrimeMonitor with prime-runs[train]>=0.1.1, so run registration (or $RUN_ID attach), metrics streaming, episode sample uploads, finalize, and crash reporting are owned by the SDK uploader instead of custom httpx + fire-and-forget tasks.

The monitor layer is now a thin adapter: pr.init(kind="train", …) on startup, run.log_metrics / run.log_episodes via asyncio.to_thread, and run.finish() with a 60s drain timeout. Local behavior kept where it matters: non-finite metrics are still sanitized with named warnings, only train / effective episodes are forwarded, and $PRIME_API_BASE is normalized for the SDK. Episode every-10th-step upload cadence moves into the SDK sink (keyed on dispatch step on run.work).

Docs in training.md describe the SDK, team allowlisting, PRIME_RUNS_MODE=disabled, and RUN_ID attach. Lock/pyproject add prime-runs and transitive prime-traces, with fast-track resolver entries.

Reviewed by Cursor Bugbot for commit aed9bbf. Bugbot is set up for automated code reviews on this repo. Configure here.

kcoopermiller and others added 3 commits September 1, 2026 10:09
Replace the hand-rolled TrainRun (RFT lifecycle, per-step metrics POSTs,
every-10th-step Parquet presign->PUT->confirm, atexit failure marking)
with prime_runs' pr.init(kind="train"): the SDK owns the uploads on a
background thread with retries and backpressure, reports crashed on a
process that exits without finalizing, and adds a Prime Traces sink for
allowlisted accounts. episodes_to_parquet_bytes and SAMPLE_SCHEMA moved
into prime_runs.projection; the kind/subset cohort filter stays here,
the step cadence moves into the SDK's training samples sink.

TEMPORARY: prime-runs is not on PyPI yet; pinned to the prime repo's
feature/prime-runs-train-backend branch (#856 + #873) via uv source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AGeqWNdRB8ckrgch1tqVzJ
kcoopermiller and others added 3 commits September 1, 2026 17:31
…tests

- pyproject/uv.lock: the temporary git source points at the SDK branch head
  (aba26790) instead of a commit that only a merged PR ref still reaches;
  it picks up the per-upload sample id ranges, the training sink cooldown,
  replayable metrics and null handling for non-finite values. Dropped once
  0.1.0 is on PyPI.
- orchestrator: with an online platform run, episodes carry the platform's
  run id, so the run's traces can be queried by the id the dashboard shows.
  W&B keeps the launcher's PRL_RUN_ID either way.
- monitor: log_metrics hops off the loop like log_episodes (a queue put can
  block under backpressure); the PRIME_RUNS_MODE value passes straight
  through; _base_url only strips /rft (the SDK strips /api/v1); a 60 s
  finish_timeout bounds the drain on finish and on the atexit crash path.
- docs/training.md: the platform-monitoring section describes the SDK,
  the team requirement, PRIME_RUNS_MODE=disabled and RUN_ID attach.
- tests/unit/monitors: first coverage of the monitor (init kwargs, attach,
  disabled switch, base URL).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
Adopting the platform run id for episode provenance coupled the
orchestrator's identity to which monitor came up online. prime-runs now sets
`run.id`/`run.type` on the uploaded copy of every episode (keeping `work`,
the dispatch step), so Prime Traces documents carry the dashboard's id
without the orchestrator learning it, and `self.run_id` is back to the
launcher's `PRL_RUN_ID` (or a local uuid) as on main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FHMjWstGWd1oGKjoj5RgeT
kcoopermiller added a commit to PrimeIntellect-ai/prime that referenced this pull request Sep 4, 2026
## Why

Eval and training jobs need one call that opens a run on the platform, streams records while they execute, and marks the run finished, failed, cancelled, or crashed when they stop. verifiers and prime-rl each hand-roll that lifecycle against the raw API today — two copies of the same thing — and nobody uploads the config a run was actually launched with, which is why the Config tab shows a v0 blob instead of anyone's `eval.toml`.

## What

A new leaf package, `prime-runs` (`httpx` + `prime-traces`; `pyarrow` behind a `train` extra), where the run is an object:

```python
import prime_runs as pr

run = pr.init(name="gsm8k-qwen3-8b", environments=["gsm8k"],
              model=config.model, framework="verifiers",
              config="eval.toml")              # stored byte-for-byte
print(run.url)

for episode in rollouts:
    run.log_episodes([episode])               # a queue put; streams on a background thread

run.finish(summary=pr.metrics.from_episodes(episodes))
```

Training runs use the same handle over the RFT external-runs API (folded in from #873):

```python
run = pr.init(kind="train", model=base_model, environments=[env_id],
              training=pr.TrainingSpec(max_steps=1000, batch_size=64),
              config=cfg.model_dump(), team_id=team)   # or id=os.environ["RUN_ID"] to attach
run.log_episodes(step_episodes)        # every 10th step -> Parquet presign / PUT / confirm
run.log_metrics(metrics, step=step)    # POST /rft/metrics, on its own uploader
run.finish()                           # idempotent finalize; failed/cancelled/crashed -> status PUT
```

`Run` exposes `id`, `url`, `kind`, `log_episodes()` / `log_traces()` / `log_metrics()`, `update_summary()`, `flush()`, `finish()`, `fail()`, a context manager (an exception → `failed`, Ctrl-C → `cancelled`, an exit without `finish()` → `crashed` via atexit), and loss counters (`dropped_records`, `failed_records`, `errors`).

Records are keyed to the run at upload: the traces sink sets `run.id` / `run.type` on the uploaded copy of every trace and episode (member traces included) and keeps the producer's `name` / `work`, so verifiers and prime-rl keep their own local run ids and never learn the platform's.

Every record is written to two places during the transition: Prime Traces (the system of record; an account outside the beta has that sink retire quietly) and the table today's viewer reads — the v0 sample table for evals, per-step Parquet objects for training. Once the viewers read traces natively, the legacy sink is one line to drop and no producer changes.

How it degrades: queued batches coalesce so the request rate tracks upload latency; a full queue drops (counted) rather than stalling the run; a transient failure costs its batch and three in a row retire the sink — a training run pauses it for five minutes instead, since it outlives any platform deploy; non-finite numbers become `null` in the sample tables rather than losing the row. `POST /evaluations/`, `/rft/external-runs` and `/samples` are never replayed after an ambiguous failure (a retry could duplicate); presign, confirm, `/rft/metrics` (one row per step server-side) and `/rft/finalize` are.

Integrations: [verifiers#2415](PrimeIntellect-ai/verifiers#2415) (eval), [prime-rl#3459](PrimeIntellect-ai/prime-rl#3459) (training).

## Known gaps

- **No producer-facing way to mark an evaluation FAILED** in the API as deployed today: `finalize` only moves PROCESSING → COMPLETED, so a crashed or interrupted eval stays RUNNING on the dashboard. The SDK records status/error under `metadata.prime_runs` and warns. platform#4740 adds `PUT /evaluations/{id} {status: FAILED|CANCELLED, error_message}` with a 409 guard; once deployed, `EvalsBackend.finalize` should use it and keep the metadata fallback for older backends (`RunStatus.CANCELLED` already exists for that).
- **`config_source` is stored, not yet shown.** `init(config=<path>)` keeps the launched file byte-for-byte under `metadata.config_source` (`run_config.config_source` for training). The eval Config tab renders it once platform#4740 lands; verifiers#2415 sends only the three v0 keys for now and adds the file in a follow-up, and the training Config tab needs the same branch.
- **Attach targets external runs.** `init(kind="train", id=...)` works for runs created through `POST /rft/external-runs`; the v1 monitoring endpoints answer 400 for a hosted run's id (the same endpoints prime-rl's old monitor used, so nothing regresses).
- **Training samples are keyed by dispatch step.** Objects are additive per step on the platform, so an off-policy episode landing in a later batch adds an object for its dispatch step; each object numbers its `sample_id`s in its own range (a forked child in a random band). "Step N" in the sample viewer means dispatched at N, while the metrics at N describe the batch trained at N.

## Before merge

1. ~~prime-traces 0.0.3 on PyPI (#872), then `prime-traces>=0.0.3` here.~~ Done.
2. A PyPI **pending publisher** for `prime-runs` (owner `PrimeIntellect-ai`, repo `prime`, workflow `release-runs.yml`, environment `pypi-prod`). The name is unclaimed; the first run of the release workflow creates the project. The workflow publishes before it tags, so a failed first publish leaves no tag and re-runs from `main` via `workflow_dispatch`.
3. ~~Coalesce queued sample batches, add `RunStatus.CANCELLED`, fold #873 in so 0.1.0 carries the training backend.~~ Done.
4. Squash-merge (`main` requires linear history); `release-runs.yml` publishes 0.1.0 with eval + train. verifiers#2415 and prime-rl#3459 then drop their `[tool.uv.sources]` git pins and relock.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> New client code talks to production eval/RFT/traces APIs with nuanced retry and upload semantics; mistakes could duplicate runs, drop samples, or mis-report terminal status, though behavior is heavily tested and defaults to warn-not-raise.
> 
> **Overview**
> Introduces **`prime-runs`**, a new leaf Python package (`httpx` + `prime-traces`, optional `pyarrow` for training) that centralizes platform run lifecycle: **`init()`** registers or attaches to a run, **`log_episodes` / `log_traces` / `log_metrics`** stream records on background uploaders, and **`finish()`** drains queues and closes the run (with context-manager, atexit crash reporting, and fork-safe HTTP/traces clients).
> 
> **Eval** runs use **`EvalsBackend`** (`/evaluations`, hub environment resolve) and dual sinks to Prime Traces plus the legacy v0 sample table; **training** runs use **`RftBackend`** (`/rft/external-runs`, attach via `id=`), Parquet sample uploads on a step cadence, and a separate metrics uploader. The SDK stamps platform **`run.id` / `run.type`** on records, stores launch **`config_source`** verbatim, and implements deliberate retry/idempotency (no replay on ambiguous `POST` creates or append-only samples).
> 
> **Repo wiring:** CI gains **`test-runs`** (Python 3.11–3.13) and version-bump enforcement for `packages/prime-runs`; **`release-runs.yml`** publishes to PyPI and tags `prime-runs-v*`. Hermetic tests cover HTTP, backends, sinks, projection, and end-to-end `init()` flows.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b65b9a. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
prime-runs 0.1.0 is published (prime#856 → prime-runs-v0.1.0), so the
TEMPORARY git source is gone; `prime-runs[train]>=0.1.0` now resolves from
PyPI and the lock carries prime-runs 0.1.0 + prime-traces 0.0.3 from the
registry. The relock also picks up the SDK's run re-keying in the traces
sink (prime#856 1b65b9a1), which the git pin predated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
@kcoopermiller
kcoopermiller marked this pull request as ready for review September 4, 2026 03:58

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread src/prime_rl/monitors/prime.py Resolved
kcoopermiller and others added 2 commits September 4, 2026 10:04
0.1.1 (prime#895 via prime#896) teaches the SDK's platform client the
internal RFT root a hosted run's launcher injects as $PRIME_API_BASE
(`…/api/internal/rft`): requests go under /api/internal and the run's token
also travels in x-api-key, which is what that router authenticates on. The
monitor's `_base_url()` already passed that root minus `/rft`, so nothing
here changes besides the floor; the relock moves prime-runs only.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
Comment thread tests/unit/monitors/test_prime.py Resolved
kcoopermiller and others added 2 commits September 8, 2026 15:27
Requested in review: the monitor is a thin adapter over prime-runs and the
SDK covers the lifecycle itself.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HpQPfQPmow2Vaq6qHZZZjZ

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 163cd60. Configure here.

Comment thread src/prime_rl/monitors/prime.py Resolved
@kcoopermiller
kcoopermiller requested a review from samsja September 8, 2026 22:40
samsja
samsja previously approved these changes Sep 8, 2026
Move the verifiers submodule from 828488fff (Sep 3) to 392d596de, which
carries verifiers #2415 (eval runs streamed through prime-runs), and refresh
uv.lock: verifiers now declares prime-runs>=0.1.1, already a direct
dependency here. No prime-rl code change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0119wW9JQxDuDrLaUBwJB3Bk
@kcoopermiller
kcoopermiller merged commit dd2dd54 into main Sep 9, 2026
17 of 19 checks passed
@kcoopermiller
kcoopermiller deleted the cooper/prime-runs branch September 9, 2026 03:06
mikasenghaas added a commit that referenced this pull request Sep 14, 2026
Main moved the prime monitor onto prime-runs (#3459) and renamed the
admin clients to AdminPlane (#3472). The eval monitor now opens one
prime-runs eval run per finished epoch instead of posting to
/evaluations by hand, which retires prime_rl.monitors.platform: the
SDK's projection and metrics modules own the sample format and the run
aggregates. The eval runner takes the AdminPlane rename and the
error-rate line from #3462. The swe example stays under examples/eval
(main had moved it to configs/evals for `uv run evals`), and the
verifiers submodule follows feat/remove-eval-cli's merge of main.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants