Repository navigation
release: prime-traces 0.0.3 - #872
Merged
Merged
Conversation
Publishes #871 (default `traces_url` = https://prime-traces.pintel.dev instead of the platform API, which does not route /api/v1/traces). PyPI 0.0.2 still carries the old fallback, so every prime-traces / prime-runs upload from a PyPI install 404s and retires the traces sink loudly. No other changes since 0.0.2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6LbnUUfTfEAUj74hXsjh6
kcoopermiller
requested review from
DamianB-BitFlipper,
JannikSt,
JohannesHa,
burnpiro,
kennethnym and
willccbb
as code owners
August 27, 2026 18:51
kennethnym
approved these changes
Aug 27, 2026
kcoopermiller
added a commit
that referenced
this pull request
Aug 27, 2026
0.0.2 resolves the traces service URL to the platform API, which does not route /api/v1/traces, so every traces upload 404s and the traces sink retires with a warning on the first batch. 0.0.3 (#871, released in #872) defaults to https://prime-traces.pintel.dev. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CyW4j57juAg5cheTX8PmUR
Merged
kcoopermiller
added a commit
that referenced
this pull request
Sep 4, 2026
## Why
Eval and training jobs need one call that opens a run on the platform, streams records while they execute, and marks the run finished, failed, cancelled, or crashed when they stop. verifiers and prime-rl each hand-roll that lifecycle against the raw API today — two copies of the same thing — and nobody uploads the config a run was actually launched with, which is why the Config tab shows a v0 blob instead of anyone's `eval.toml`.
## What
A new leaf package, `prime-runs` (`httpx` + `prime-traces`; `pyarrow` behind a `train` extra), where the run is an object:
```python
import prime_runs as pr
run = pr.init(name="gsm8k-qwen3-8b", environments=["gsm8k"],
model=config.model, framework="verifiers",
config="eval.toml") # stored byte-for-byte
print(run.url)
for episode in rollouts:
run.log_episodes([episode]) # a queue put; streams on a background thread
run.finish(summary=pr.metrics.from_episodes(episodes))
```
Training runs use the same handle over the RFT external-runs API (folded in from #873):
```python
run = pr.init(kind="train", model=base_model, environments=[env_id],
training=pr.TrainingSpec(max_steps=1000, batch_size=64),
config=cfg.model_dump(), team_id=team) # or id=os.environ["RUN_ID"] to attach
run.log_episodes(step_episodes) # every 10th step -> Parquet presign / PUT / confirm
run.log_metrics(metrics, step=step) # POST /rft/metrics, on its own uploader
run.finish() # idempotent finalize; failed/cancelled/crashed -> status PUT
```
`Run` exposes `id`, `url`, `kind`, `log_episodes()` / `log_traces()` / `log_metrics()`, `update_summary()`, `flush()`, `finish()`, `fail()`, a context manager (an exception → `failed`, Ctrl-C → `cancelled`, an exit without `finish()` → `crashed` via atexit), and loss counters (`dropped_records`, `failed_records`, `errors`).
Records are keyed to the run at upload: the traces sink sets `run.id` / `run.type` on the uploaded copy of every trace and episode (member traces included) and keeps the producer's `name` / `work`, so verifiers and prime-rl keep their own local run ids and never learn the platform's.
Every record is written to two places during the transition: Prime Traces (the system of record; an account outside the beta has that sink retire quietly) and the table today's viewer reads — the v0 sample table for evals, per-step Parquet objects for training. Once the viewers read traces natively, the legacy sink is one line to drop and no producer changes.
How it degrades: queued batches coalesce so the request rate tracks upload latency; a full queue drops (counted) rather than stalling the run; a transient failure costs its batch and three in a row retire the sink — a training run pauses it for five minutes instead, since it outlives any platform deploy; non-finite numbers become `null` in the sample tables rather than losing the row. `POST /evaluations/`, `/rft/external-runs` and `/samples` are never replayed after an ambiguous failure (a retry could duplicate); presign, confirm, `/rft/metrics` (one row per step server-side) and `/rft/finalize` are.
Integrations: [verifiers#2415](PrimeIntellect-ai/verifiers#2415) (eval), [prime-rl#3459](PrimeIntellect-ai/prime-rl#3459) (training).
## Known gaps
- **No producer-facing way to mark an evaluation FAILED** in the API as deployed today: `finalize` only moves PROCESSING → COMPLETED, so a crashed or interrupted eval stays RUNNING on the dashboard. The SDK records status/error under `metadata.prime_runs` and warns. platform#4740 adds `PUT /evaluations/{id} {status: FAILED|CANCELLED, error_message}` with a 409 guard; once deployed, `EvalsBackend.finalize` should use it and keep the metadata fallback for older backends (`RunStatus.CANCELLED` already exists for that).
- **`config_source` is stored, not yet shown.** `init(config=<path>)` keeps the launched file byte-for-byte under `metadata.config_source` (`run_config.config_source` for training). The eval Config tab renders it once platform#4740 lands; verifiers#2415 sends only the three v0 keys for now and adds the file in a follow-up, and the training Config tab needs the same branch.
- **Attach targets external runs.** `init(kind="train", id=...)` works for runs created through `POST /rft/external-runs`; the v1 monitoring endpoints answer 400 for a hosted run's id (the same endpoints prime-rl's old monitor used, so nothing regresses).
- **Training samples are keyed by dispatch step.** Objects are additive per step on the platform, so an off-policy episode landing in a later batch adds an object for its dispatch step; each object numbers its `sample_id`s in its own range (a forked child in a random band). "Step N" in the sample viewer means dispatched at N, while the metrics at N describe the batch trained at N.
## Before merge
1. ~~prime-traces 0.0.3 on PyPI (#872), then `prime-traces>=0.0.3` here.~~ Done.
2. A PyPI **pending publisher** for `prime-runs` (owner `PrimeIntellect-ai`, repo `prime`, workflow `release-runs.yml`, environment `pypi-prod`). The name is unclaimed; the first run of the release workflow creates the project. The workflow publishes before it tags, so a failed first publish leaves no tag and re-runs from `main` via `workflow_dispatch`.
3. ~~Coalesce queued sample batches, add `RunStatus.CANCELLED`, fold #873 in so 0.1.0 carries the training backend.~~ Done.
4. Squash-merge (`main` requires linear history); `release-runs.yml` publishes 0.1.0 with eval + train. verifiers#2415 and prime-rl#3459 then drop their `[tool.uv.sources]` git pins and relock.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> New client code talks to production eval/RFT/traces APIs with nuanced retry and upload semantics; mistakes could duplicate runs, drop samples, or mis-report terminal status, though behavior is heavily tested and defaults to warn-not-raise.
>
> **Overview**
> Introduces **`prime-runs`**, a new leaf Python package (`httpx` + `prime-traces`, optional `pyarrow` for training) that centralizes platform run lifecycle: **`init()`** registers or attaches to a run, **`log_episodes` / `log_traces` / `log_metrics`** stream records on background uploaders, and **`finish()`** drains queues and closes the run (with context-manager, atexit crash reporting, and fork-safe HTTP/traces clients).
>
> **Eval** runs use **`EvalsBackend`** (`/evaluations`, hub environment resolve) and dual sinks to Prime Traces plus the legacy v0 sample table; **training** runs use **`RftBackend`** (`/rft/external-runs`, attach via `id=`), Parquet sample uploads on a step cadence, and a separate metrics uploader. The SDK stamps platform **`run.id` / `run.type`** on records, stores launch **`config_source`** verbatim, and implements deliberate retry/idempotency (no replay on ambiguous `POST` creates or append-only samples).
>
> **Repo wiring:** CI gains **`test-runs`** (Python 3.11–3.13) and version-bump enforcement for `packages/prime-runs`; **`release-runs.yml`** publishes to PyPI and tags `prime-runs-v*`. Hermetic tests cover HTTP, backends, sinks, projection, and end-to-end `init()` flows.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b65b9a. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EYdQmKAVnGteT1tkimJb3g
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Low Risk
Version-only change with no runtime logic edits; risk is limited to release/tag/PyPI process if publish fails.
Overview
Bumps
prime-tracesfrom 0.0.2 to 0.0.3 inpackages/prime-traces/src/prime_traces/__init__.pyso the traces release workflow can publish a PyPI build that includes the #871 fix (Config.traces_urldefaulting to the traces service instead ofapi.primeintellect.ai).There are no functional code changes in this PR—only the version string—so downstream installs from PyPI can pick up the corrected default URL and avoid trace upload 404s that still affect 0.0.2.
Reviewed by Cursor Bugbot for commit 4c86ea8. Bugbot is set up for automated code reviews on this repo. Configure here.