add the prime-runs SDK by kcoopermiller · Pull Request #856 · PrimeIntellect-ai/prime

kcoopermiller · 2026-08-19T20:53:26Z

Why

Eval and training jobs need one call that opens a run on the platform, streams records while they execute, and marks the run finished, failed, cancelled, or crashed when they stop. verifiers and prime-rl each hand-roll that lifecycle against the raw API today — two copies of the same thing — and nobody uploads the config a run was actually launched with, which is why the Config tab shows a v0 blob instead of anyone's eval.toml.

What

A new leaf package, prime-runs (httpx + prime-traces; pyarrow behind a train extra), where the run is an object:

import prime_runs as pr

run = pr.init(name="gsm8k-qwen3-8b", environments=["gsm8k"],
              model=config.model, framework="verifiers",
              config="eval.toml")              # stored byte-for-byte
print(run.url)

for episode in rollouts:
    run.log_episodes([episode])               # a queue put; streams on a background thread

run.finish(summary=pr.metrics.from_episodes(episodes))

Training runs use the same handle over the RFT external-runs API (folded in from #873):

run = pr.init(kind="train", model=base_model, environments=[env_id],
              training=pr.TrainingSpec(max_steps=1000, batch_size=64),
              config=cfg.model_dump(), team_id=team)   # or id=os.environ["RUN_ID"] to attach
run.log_episodes(step_episodes)        # every 10th step -> Parquet presign / PUT / confirm
run.log_metrics(metrics, step=step)    # POST /rft/metrics, on its own uploader
run.finish()                           # idempotent finalize; failed/cancelled/crashed -> status PUT

Run exposes id, url, kind, log_episodes() / log_traces() / log_metrics(), update_summary(), flush(), finish(), fail(), a context manager (an exception → failed, Ctrl-C → cancelled, an exit without finish() → crashed via atexit), and loss counters (dropped_records, failed_records, errors).

Records are keyed to the run at upload: the traces sink sets run.id / run.type on the uploaded copy of every trace and episode (member traces included) and keeps the producer's name / work, so verifiers and prime-rl keep their own local run ids and never learn the platform's.

Every record is written to two places during the transition: Prime Traces (the system of record; an account outside the beta has that sink retire quietly) and the table today's viewer reads — the v0 sample table for evals, per-step Parquet objects for training. Once the viewers read traces natively, the legacy sink is one line to drop and no producer changes.

How it degrades: queued batches coalesce so the request rate tracks upload latency; a full queue drops (counted) rather than stalling the run; a transient failure costs its batch and three in a row retire the sink — a training run pauses it for five minutes instead, since it outlives any platform deploy; non-finite numbers become null in the sample tables rather than losing the row. POST /evaluations/, /rft/external-runs and /samples are never replayed after an ambiguous failure (a retry could duplicate); presign, confirm, /rft/metrics (one row per step server-side) and /rft/finalize are.

Integrations: verifiers#2415 (eval), prime-rl#3459 (training).

Known gaps

No producer-facing way to mark an evaluation FAILED in the API as deployed today: finalize only moves PROCESSING → COMPLETED, so a crashed or interrupted eval stays RUNNING on the dashboard. The SDK records status/error under metadata.prime_runs and warns. platform#4740 adds PUT /evaluations/{id} {status: FAILED|CANCELLED, error_message} with a 409 guard; once deployed, EvalsBackend.finalize should use it and keep the metadata fallback for older backends (RunStatus.CANCELLED already exists for that).
config_source is stored, not yet shown. init(config=<path>) keeps the launched file byte-for-byte under metadata.config_source (run_config.config_source for training). The eval Config tab renders it once platform#4740 lands; verifiers#2415 sends only the three v0 keys for now and adds the file in a follow-up, and the training Config tab needs the same branch.
Attach targets external runs. init(kind="train", id=...) works for runs created through POST /rft/external-runs; the v1 monitoring endpoints answer 400 for a hosted run's id (the same endpoints prime-rl's old monitor used, so nothing regresses).
Training samples are keyed by dispatch step. Objects are additive per step on the platform, so an off-policy episode landing in a later batch adds an object for its dispatch step; each object numbers its sample_ids in its own range (a forked child in a random band). "Step N" in the sample viewer means dispatched at N, while the metrics at N describe the batch trained at N.

Before merge

~~prime-traces 0.0.3 on PyPI (release: prime-traces 0.0.3 #872), then prime-traces>=0.0.3 here.~~ Done.
A PyPI pending publisher for prime-runs (owner PrimeIntellect-ai, repo prime, workflow release-runs.yml, environment pypi-prod). The name is unclaimed; the first run of the release workflow creates the project. The workflow publishes before it tags, so a failed first publish leaves no tag and re-runs from main via workflow_dispatch.
~~Coalesce queued sample batches, add RunStatus.CANCELLED, fold prime-runs: RftBackend for training runs #873 in so 0.1.0 carries the training backend.~~ Done.
Squash-merge (main requires linear history); release-runs.yml publishes 0.1.0 with eval + train. verifiers#2415 and prime-rl#3459 then drop their [tool.uv.sources] git pins and relock.

Note

Medium Risk
New client code talks to production eval/RFT/traces APIs with nuanced retry and upload semantics; mistakes could duplicate runs, drop samples, or mis-report terminal status, though behavior is heavily tested and defaults to warn-not-raise.

Overview
Introduces prime-runs, a new leaf Python package (httpx + prime-traces, optional pyarrow for training) that centralizes platform run lifecycle: init() registers or attaches to a run, log_episodes / log_traces / log_metrics stream records on background uploaders, and finish() drains queues and closes the run (with context-manager, atexit crash reporting, and fork-safe HTTP/traces clients).

Eval runs use EvalsBackend (/evaluations, hub environment resolve) and dual sinks to Prime Traces plus the legacy v0 sample table; training runs use RftBackend (/rft/external-runs, attach via id=), Parquet sample uploads on a step cadence, and a separate metrics uploader. The SDK stamps platform run.id / run.type on records, stores launch config_source verbatim, and implements deliberate retry/idempotency (no replay on ambiguous POST creates or append-only samples).

Repo wiring: CI gains test-runs (Python 3.11–3.13) and version-bump enforcement for packages/prime-runs; release-runs.yml publishes to PyPI and tags prime-runs-v*. Hermetic tests cover HTTP, backends, sinks, projection, and end-to-end init() flows.

^{Reviewed by Cursor Bugbot for commit 1b65b9a. Bugbot is set up for automated code reviews on this repo. Configure here.}

prime-evals models a run as three stateless calls over Dict[str, Any], so neither of its intended consumers uses it on their main path: verifiers reimplemented create -> batch -> finalize inline in v1/utils/platform.py, and prime-rl hits a different API family entirely from utils/monitor/prime.py. Producers have objects, a long-running loop, steps, ranks, forks and crashes; each of them solved that privately, twice. prime-runs makes the run an object instead: run = pr.init(name=..., environments=["gsm8k"], model=..., framework=...) run.log_traces([episode]); run.log({"reward": r}, step=step) run.finish(summary=pr.metrics.from_episodes(episodes)) Identity: init() is called before rollouts start and the ID it returns is the run ID everywhere, local archive included. Producers already stamp the run onto their traces, so nothing is re-stamped and no producer record is rewritten. The join key is run.id inside the trace document -- an indexed ClickHouse column with a delete-by-run path -- not an upload-scoped context key. Backends and sinks are independent axes. Backends own lifecycle (EvalsBackend, OfflineBackend); sinks own transport (TracesSink, plus EvalSamplesSink for the viewer's flat table). Both sinks run during the transition, because Prime Traces is gated to an account allowlist and a traces-only client would leave everyone else with an empty dashboard. When the Viewer API reads traces natively, the default sink list drops one entry and no producer changes. The SDK owns the operational work: streaming instead of buffering, a bounded upload queue that drops and counts rather than stalling a training run, fork safety via register_at_fork, contained errors (on_error="warn" by default), terminal status through the context manager / atexit / signals, rank awareness, and an offline mode that is a real run -- which is what lets producers delete their --no-push branching. trace_to_sample / build_samples move here from verifiers: it is knowledge about a platform wire format, and prime-rl currently reaches across a repo boundary to import it from a module path that has already drifted. Known platform gap: there is no producer-facing way to mark an evaluation failed (finalize only goes PROCESSING -> COMPLETED, UpdateEvaluationRequest has no status). EvalsBackend calls the status endpoint it needs, latches on 404 so it probes once, and falls back to recording the terminal state in metadata while warning that the run will keep showing as running. The fallback stops firing on its own once the endpoint ships. Leaf package by construction -- httpx, pydantic, tenacity, prime-traces and nothing else -- because the prime CLI depends on verifiers, so verifiers can never depend on prime. verifiers already takes prime-tunnel and prime-sandboxes on the same terms. 107 tests, hermetic (httpx.MockTransport + tmp dirs). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0dcb62a816

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

All six were real. Verified against the backend where the claim depended on service behaviour. Run identity (high). init() exported PRIME_RUN_ID and then read it back, so a second init() in the same process silently attached to the first run and never created or finalized one of its own. Exports now record the exporting PID, and only an ID from a *different* PID counts as inherited — a forked child sees the parent's PID and joins, a re-init sees its own and opens a fresh run. Lifecycle ownership follows intent rather than "was an ID present": an explicit id= is a deliberate resume and finalizes, an ID picked up from the environment belongs to whoever exported it. finish() also stops advertising the run it owned. Fork safety (high). The child inherited the parent's httpx pools and buffered file handles. Two processes writing one socket interleave into a single HTTP stream, and a duplicated write buffer gets flushed twice. Connection and file holders now reset in the child through a single process-wide hook in _fork.py: dropped, never closed (closing sends close_notify down a socket the parent is still reading) and never flushed (the buffer holds records the parent will write itself). The hook is registered once rather than per instance, because os.register_at_fork cannot be undone — the old per-worker registration pinned every run the process ever opened. Metadata replacement (medium). The service writes metadata with {"$set": {"metadata": ...}}, a document-level replace. The failure fallback PUT carried only {"prime_runs": ...}, erasing the config finish() had just written. finalize() now receives the run's full config and merges into it. Abandoned uploads (medium). finish() waited 60s for a flush, ignored the result, then close() joined 30s — while a single sample POST is allowed 300s. The budget is now derived from the upload timeout, a flush that does not drain warns instead of passing silently, and close() leaves sinks open when the thread is still alive rather than pulling a client out from under a live request. Signal status (medium). The handler reported FAILED while atexit, RunStatus and the README all said CRASHED. Signals and Ctrl-C now report CRASHED: the producer never said the run failed, it was stopped from outside its control flow. FAILED stays for what the producer itself reports. Environment version pinning (medium). EnvironmentRef accepted version_id and dropped it, even though the API's EnvironmentReference carries it — the run attached to whatever version the hub resolved that day. Found while testing the signal fix: _handle_signal read the displaced handler *after* finish(), which restores and clears that table, so chaining always fell back to SIG_DFL — re-raising the signal at default disposition and killing the process instead of running the handler the application installed. Captured before finish() now. 122 tests (up from 107), including a real os.fork() end-to-end check that the child joins the parent's run and no record is written twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three findings the Bugbot pass did not cover. The other two Codex raised (environment version pinning, PRIME_RUN_ID re-init) were already fixed in 14bbac5, including its note that `setdefault` left descendants pointing at a stale ID when an explicit `id=` was supplied — `_announce` now assigns unconditionally. Non-idempotent retries. The client retried every method through ambiguous failures, so a lost response to POST /evaluations/ would create a second evaluation that the SDK never tracked, leaving an orphaned duplicate run. The module docstring even asserted this was safe "because run creation happens once per init()", which confuses the call site with the retry loop inside it. Retry safety is now decided per call. A failure is ambiguous when the request may already have been processed (gateway 502/504, read timeout, stream broken after the bytes went out); unambiguous when nothing reached the server (connect failure, 429 refused before any work). Unambiguous failures replay for every method. Ambiguous ones replay only when the caller declares `idempotent=True`, which defaults to `method != "POST"`. Run creation and sample appends keep the default; get-or-create, finalize and status writes declare themselves safe. Same classification prime-traces' client already uses. This also stops the samples sink duplicating rows on a lost response — duplicates silently skew every average on the dashboard, where a lost batch is at least recoverable. Uploader failures under on_error="raise". A sink fails on the uploader thread, where the raise went straight into the worker's own except and was discarded — so flush() and finish() returned success while records were being dropped, in exactly the mode documented as being for "tests and CI, where a silent upload failure is the bug". The failure is now held and re-raised at the next synchronization point the caller controls: flush(), or the very end of finish() so the run is still closed out properly first. The atexit and signal paths swallow it, since neither is a place to surface an exception. Signal handlers were never restored. `_restore_signal_handlers` compared `signal.getsignal(signum) is self._handle_signal`, but every access to a bound method builds a new object, so the identity check could never match. Handlers stayed installed for the life of the process: the finished Run was pinned, and the next run in that process saw a non-default handler and declined to install its own, leaving it unable to report signal termination. The bound method is created once in __init__ and compared against. 133 tests (up from 122). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Two follow-up findings, both on the previous rounds' fixes. Offline records were buffered. reset_after_fork() dropped the inherited file handles, but on CPython the last reference going away closes them, and close() flushes — writing out the child's copy of the parent's buffer and duplicating every record still sitting in it. Dropping without closing is not expressible for a buffered writer, so the buffer is gone instead: records are written to an unbuffered append-mode handle, encoded here. Nothing is ever held in process memory, so a fork has nothing to copy and flush() has nothing to do. The earlier fork test passed on this path by luck. It forked before the uploader thread had opened the file, so no handle and no buffer existed in the child. Replaced with an assertion that records are readable through a separate handle with no flush and no close, which is what actually pins the property. Transient sample failures retired the sink. Making sample POSTs non-replayable (332fc21) was right on its own, but combined with the worker disabling a sink on any raise it meant a single 502 stopped every later batch — so one gateway blip could leave the rest of a run missing from the dashboard of exactly the accounts the v0 sample table exists to serve. That trade is worse than the duplicates it avoids. The worker now separates "this batch failed" from "this sink is finished". A permanent failure (gated account, rejected credential) will fail identically forever and still retires the sink immediately. A transient one gets three consecutive strikes, reset by any success, so a blip costs one batch and a sustained outage still stops the SDK re-attempting for hours. Dropped records are counted either way, so run.dropped_records reflects the loss. This applies to the traces sink too, which had the same all-or-nothing behaviour. 138 tests (up from 133). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous commit folded failed sink writes into `dropped`, which is documented and warned about as queue backpressure. Two things went wrong with that. A default online run writes to both the traces sink and the sample table, so one failed batch was counted twice. And a failure on one sink was counted at all even when the other sink stored the records, so `finish()` could warn about data missing from a run that has all of it. They are different losses and stay separate now. `dropped` counts records that reached no sink because the queue was full — the producer outran the uploader, and those records are stored nowhere. `failed_records` counts per sink, exposed as a mapping rather than a total, because summing it would recreate exactly the overstatement above. The finish warnings are phrased to match: one about records that reached nothing, one per sink about what that sink could not store. 139 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Five points where prime-runs diverged from prime-traces, prime-evals, prime-sandboxes and prime-tunnel without meaning to: - Normalize an explicitly passed `base_url`. `Config` strips a trailing `/api/v1`; the `PlatformClient` constructor did not, so `pr.init( base_url=".../api/v1")` requested `/api/v1/api/v1` while the identical value in `PRIME_API_BASE_URL` worked. prime-traces carries the same helper for the same reason. - Map 403 to a typed `ForbiddenError`, matching `prime_traces.ForbiddenError` — which the traces sink already branches on for beta gating, while a 403 from the platform API collapsed into a generic `RunAPIError`. Behavior is unchanged (it was already classified permanent); callers can now branch. - Drop `pydantic` and `tenacity`. Neither was imported: this package models no response bodies and hand-rolls its retry loop. They do not belong in the dependency tree of a leaf package that lands inside verifiers. - Add the LICENSE file the other packages ship. - Declare 3.13, which CI has been testing all along. Also documents the three departures that are deliberate — private client instead of `core/`, dataclasses instead of pydantic models, no async client — and what to do about the three blocking calls when driving a run from async code, since every sibling SDK ships an async client and this one does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

0.0.2 resolves the traces service URL to the platform API, which does not route /api/v1/traces, so every traces upload 404s and the traces sink retires with a warning on the first batch. 0.0.3 (#871, released in #872) defaults to https://prime-traces.pintel.dev. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyW4j57juAg5cheTX8PmUR

The uploader dispatched one queue item per request. verifiers hands over one episode per rollout, so that was one POST /samples per episode against the platform's 300/min limit: any eval finishing more than five episodes a second got 429s, the retries backed the 256-item queue up, submit() blocked the rollout slot for 5s per episode, and records were then dropped from both sinks. The old one-shot upload never lost rows to rate limits. UploadWorker._run now drains everything already queued behind the item it took into one dispatch, so the request rate tracks upload latency rather than rollout throughput. Sinks already split by size (25 MB for samples, the traces client's own batching). Coalescing stops at a flush barrier (flush() promises everything before it has been written), at the stop sentinel, and at a batch of the other record kind (the traces sink infers the line format from the first record). A worker with no sinks — every disabled run — no longer copies batches into a queue or starts a thread for nothing. RunStatus gains CANCELLED: an interrupt is a decision, not a fault, and the platform's producer statuses are FAILED | CANCELLED. The context manager maps KeyboardInterrupt and asyncio.CancelledError to it (was CRASHED); the atexit hook still reports CRASHED. Added now so the enum does not change under consumers when the SDK adopts the status endpoint. init(finish_timeout=) and finish(timeout=) bound how long queued uploads get to drain before the run is closed out. An abort path can pass a few seconds so an interrupted producer is not pinned behind a slow platform for the full 300s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBkJRDCFihXMjU1YpJTt14

* feat(runs): RftBackend — training runs over /api/v1/rft/external-runs What prime-rl's monitors/prime.py::TrainRun did by hand, in the SDK, so the monitor can become a thin adapter over pr.init(kind="train"). Additive: the eval path is unchanged apart from run.type now following the run's kind. init(kind="train", model=<base model>, environments=[hub ids], training=TrainingSpec(max_steps, batch_size, rollouts_per_example, seq_len, wandb_*), config=<run_config>, team_id=...) registers an external run. A team is required: the platform creates external runs for a team and enables them per team, so a team outside the allowlist gets the platform's 403 reason as a ForbiddenError from init() and the producer decides what that means. init(kind="train", id=$RUN_ID) attaches to a run a managed launch created: nothing is registered, the platform keeps the run's failure marking (a non-completed finish is logged and left to the launcher), and a clean finish() still completes it — prime-rl's contract. Terminal status: completed -> POST /rft/finalize {exit_code: 0}, which is a compare-and-set on the server with "already finalized" as success, so it is replayed on an ambiguous failure. failed/cancelled/crashed -> PUT /external-runs/{id}/status {status: failed, error_message} with the finer reason in the message (the RFT vocabulary is completed | failed); a 409 means someone closed it first and is not an error. update() is a no-op: run_config travels with create and there is no summary document. Run.log_metrics(values, step=) queues one row per call on a second uploader (a metrics dict is not a record, and a slow sample upload must not hold a step's metrics back); rows are stamped with _timestamp, step is set unless present, non-finite values dropped. RftMetricsSink posts one row per request (the endpoint takes a single step; 60/min per token, Retry-After honoured). RftSamplesSink is the training viewer's Parquet table: episodes whose run.work is training work (verifiers TrainRunInfo, which prime-rl stamps on every dispatched episode) are grouped by step, encoded on every 10th step (prime-rl's cadence, configurable) and sent through presign -> PUT -> confirm. Presign is replayable, the storage PUT is retried, confirm is not replayed. The encoder is prime-rl's episodes_to_parquet_bytes moved into prime_runs.projection, duck-typed like the rest; pyarrow is an optional extra (prime-runs[train]) and without it the sink turns itself off with a warning while traces still flow. Eval-work episodes, bare traces and JSON episodes have no row here and reach Prime Traces only. stamp_run takes the run type from the sink context, so bare dicts logged to a training run carry run.type="train" — the traces service's vocabulary. 64 new tests (RFT backend, both sinks, init wiring, attach semantics, the real Parquet encoder under importorskip). Version unchanged: this ships in the release after 0.1.0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBkJRDCFihXMjU1YpJTt14 * fix(runs): make the training samples sink fork-safe RftSamplesSink lazily opens its own httpx.Client for the presigned PUTs and did not register with _fork, so a forked child inherited the parent's socket and its atexit finish() would have closed it (sending close_notify on the parent's connection). Register the sink; reset_after_fork drops an owned client (rebuilt on the next upload) and marks an injected one unusable, the way TracesSink and PlatformClient already do. Also spell out the one-object-per-step contract: a step's episodes must be logged in one call (prime-rl does), since the platform keeps one object key per (run, step). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GBkJRDCFihXMjU1YpJTt14 * fix(runs): harden training finalization and metrics * fix(runs): number training samples per upload and let training sinks recover - RftSamplesSink hands the encoder a per-upload sample_id offset (stride 2**32) so a step uploaded as several objects keeps (step, sample_id) unique; the confirm POST is replayable; the module docstring and README describe the additive per-step contract (dispatched-at-N vs trained-at-N) - RftMetricsSink posts are replayable: the platform merges rows by step - UploadWorker(retire_cooldown=): a sink that struck out on transient failures is paused and tried again after the cooldown; init() sets a 300 s cooldown for kind="train", evals keep the permanent retirement - attach() targets external runs (the v1 monitoring endpoints answer 400 for a hosted run): documented on init(), RftBackend.attach and the README Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR * fix(runs): keep a forked child's sample_id ranges apart from its parent's A forked child inherits the training samples sink's per-step upload counters, so it would hand out the same sample_id range as its parent for the next object of a step. After reset_after_fork the child draws its ranges at random from the upper half of the range space, which the parent's counter never reaches; every id stays below 2**53 since the viewer parses sample_id as a JSON number. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

- docstrings and comments cut to the contract or the non-obvious reason; section banners removed - one send_each write loop in sinks/base.py replaces three copies; _finite replaces the two metric-cleaning functions; the worker's deadline helpers are shared with run.py - unused surface removed: RftBackend.log_metrics, the PlatformClient context manager, the mapping form of ConfigSource.coerce - tests: docstrings stripped, 28 redundant tests removed, the after-finish checks parametrized - README rewritten in the prime-traces shape - projection: NaN and infinity become null in projected samples and in the Parquet JSON columns, so a non-finite reward or metric no longer loses the eval batch or the training step it sits in Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e4ca656. Configure here.}

- _finite compared every cleaned value with `!= {}` to drop empty nested mappings; an array-like metric value makes that comparison ambiguous and would raise on the producer thread. Only identity and mapping checks now. - EvalSamplesSink attributes a record whose rows straddle two HTTP batches to the first batch, so a lost record is counted once, as before the send_each refactor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR

`stamp_run` kept a record's own `run` block, and verifiers and prime-rl always record one with their local run id, so the only way to get the platform's id into the trace document was for the producers to adopt it. The sink now sets `run.id`/`run.type` on the uploaded copy (envelope and every member trace) and keeps the rest of the block (`name`, `work`); producers keep their own ids and never learn the platform's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FHMjWstGWd1oGKjoj5RgeT

## Why The eval CLI hand-rolled its own platform client: `verifiers/v1/utils/platform.py` was 317 lines of resolve env → create → post samples → finalize over raw httpx, run once after the last rollout. [prime#856](PrimeIntellect-ai/prime#856) moves that lifecycle into `prime-runs`, a leaf SDK (`httpx` + `prime-traces`) where a run is an object held for the length of the eval rather than three stateless calls at the end of it. ## What `platform.py` is now ~120 lines of wiring; the SDK owns transport, retries, backpressure and the terminal status: ```python run = open_run(config, push_state, num_examples=len(tasks)) # before the first rollout; assigns config.run.id async def on_complete(episode): episode.record_run(EvalRunInfo(id=config.run.id, name=config.run.name)) await append_episode(out, episode, write_lock) await asyncio.to_thread(run.log_episodes, [episode]) # streams as rollouts land await asyncio.to_thread(finish_run, run, episodes, push_state) ``` Gone: `push_traces`, `credentials`, `json_bytes`, `run_metrics`, the batching loop, the payload ceiling, the in-memory buffer. `trace_to_sample` and `build_samples` moved to `prime_runs.projection` (knowledge about a platform wire format, not about an eval framework). `build_samples` stays importable from here because prime-rl's released monitor imports it across the repo boundary; nothing imports `trace_to_sample`. | file | what changed | | --- | --- | | `utils/platform.py` | `open_run` / `finish_run` / `abort_run` + `PushState`, the dashboard's read-through view of the run | | `cli/eval/runner.py` | both runners open the run before the first rollout, stream to it, and close it out on every exit path; `gather_rollouts` cancels sibling rollouts on the first failure so nothing keeps uploading into a run that is being closed | | `cli/eval/main.py` | drops the end-of-run push | | `configs/cli/eval.py` | `RunConfig` mints no id of its own: `open_run` assigns the prime-runs handle's id in every mode (the platform's evaluation id online, the SDK's local id otherwise), so the run dir, every trace on disk and the dashboard share one id. Linking uploads to the platform run is the SDK's job regardless (it re-keys every record it sends) | | `cli/dashboard/eval.py` | the footer reads the live run: dim `Pushing traces (<url>)` while it streams, white `Traces pushed (<url>)`, red on failure | | `tests/v1/conftest.py` | the E2E fixture sets `push=False` | | `serve/client.py` | `EnvClient.close()` gathers only the still-pending cancel senders: on Python 3.12+ a gather over finished tasks completes without yielding, so the loop spun at 100% CPU on Ctrl-C once the runner started waiting for rollouts to unwind before teardown |    > [!NOTE] > ### Replace manual HTTP trace pushing with `prime-runs` SDK in `platform.py` > - Replaces the hand-built HTTP upload implementation in [platform.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2415/files#diff-8644072f827be448db5b0010e0969c6c86d9bbf07191ca00a993f416d123235f) with the `prime-runs` SDK for opening runs, logging episodes, and closing out runs. > - Moves platform run lifecycle management into `run_eval` in [runner.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2415/files#diff-65be4a576330571f2729a3bbcdbe92ac81bbcba450c313597c1bdca3a1c3f2e4), removing the legacy post-run `push_traces` call from [main.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2415/files#diff-1335625d1379d7f1a4bf6a2f961d4b5fde5f3a9bfb8eb4cc736859f08698a8c8). > - Adds `gather_rollouts` helper to cancel and await all concurrent rollout tasks when any task fails or is cancelled before re-raising. > - Changes `cli.RunConfig.id` to be assigned by `open_run` during execution instead of being eagerly generated as a UUID. > - Risk: `cli.RunConfig.id` property now raises if accessed before `open_run` assigns an ID. > >  > #### Changes since #2415 opened > > - Added a `closing` flag to `verifiers.v1.cli.eval.runner.run_eval` that prevents `abort_run` from being called once `finish_run` has been initiated [10fea3a] > - Updated documentation and logging in `verifiers.v1.utils.platform` to clarify `finish_run` semantics and report actual run status [10fea3a] > - Modified `run_eval` function in the evaluation runner to shield `finish_run` execution from cancellation using `run_shielded` wrapping `asyncio.to_thread`, removed the `closing` boolean guard, and updated exception handling to unconditionally invoke `abort_run` for any exception in the try block [9706c82] > - Modified `abort_run` function to return early when `run.status` is not `pr.RunStatus.RUNNING` in addition to checking if `run.finished` is `True`, and updated the docstring to document that `finish()` sets `run.status` as it starts [9706c82] >  > >  > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized e06c307.</sup> >  >   --- > [!NOTE] > **Medium Risk** > Changes eval lifecycle, run identity, and platform upload timing across runner and config; failures are mostly best-effort, but code that reads `run.id` before `open_run` will now raise. > > **Overview** > Replaces the eval CLI’s hand-rolled **httpx** upload (create eval → batch samples → finalize after the last rollout) with **`prime-runs`**, which owns the run for the whole eval. > > **`run_eval`** now calls **`open_run`** before rollouts, **`log_episodes`** as each episode finishes (and for resumed episodes), and **`finish_run`** / **`abort_run`** on success or failure. The post-run **`push_traces`** path in **`main`** is removed. **`gather_rollouts`** cancels sibling rollouts when one fails so uploads don’t continue into a closing run; close-out runs off the event loop behind **`run_shielded`**. > > **`RunConfig.id`** is no longer a client UUID—it is assigned once by **`open_run`** from the SDK handle (platform id online, local id when push is off). **`PushState`** and the rich dashboard footer reflect streaming status, partial upload loss, and close-out errors. > > Adds **`prime-runs>=0.1.1`** (and lock entries for **`prime-traces`**); E2E tests set **`push=False`**. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9706c82. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

kcoopermiller requested review from DamianB-BitFlipper, JannikSt, JohannesHa, burnpiro, kennethnym and willccbb as code owners August 19, 2026 20:53