Repository navigation
feat(runs): attach eval runs to an evaluation a launcher already created - #913
Merged
Merged
Conversation
`init(id=...)` previously attached only training runs; an eval run was always created through `POST /evaluations/`. A hosted evaluation creates its evaluation record before the sandbox exists and hands the sandbox its id, so a sandbox running `uv run eval` produced a second, unlinked run. - `EvalsBackend.attach(run_id)`: a handle with no request; the sinks key every sample and trace to the launcher's id, a clean finish still finalizes the run, and a non-completed finish is left for the launcher to mark from the process exit (as for attached training runs). - An attached run no longer replaces the evaluation's `metadata` document on finish: the launcher created the run from its own config and reads it back, so only `metrics` go up. - `Backend` protocol and `DisabledBackend` gain `attach`; `init()` picks attach over create for either kind when `id` is given. Consumed by verifiers' `run.attach` and the platform's v1 hosted-eval runner (ENG-5781). Version bump follows in a release PR. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XrcqANSZ24BALuWS9MciYv
kcoopermiller
requested review from
DamianB-BitFlipper,
JannikSt,
JohannesHa,
akirillo,
burnpiro,
kennethnym,
mrmoxon and
willccbb
as code owners
September 9, 2026 00:18
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
kennethnym
approved these changes
Sep 9, 2026
kcoopermiller
added a commit
that referenced
this pull request
Sep 9, 2026
Bumps prime-runs to 0.1.2 so release-runs.yml publishes the one change since 0.1.1: #913 (`init(id=...)` attaches eval runs to an evaluation a launcher already created; consumed by verifiers `run.attach` and the platform's v1 hosted-eval runner). Merge after #913. Claude-Session: https://claude.ai/code/session_01XrcqANSZ24BALuWS9MciYv Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
kcoopermiller
added a commit
to PrimeIntellect-ai/verifiers
that referenced
this pull request
Sep 9, 2026
…dy created (#2569) ## Summary A hosted evaluation creates its evaluation record before the sandbox exists and hands the sandbox that id as `$EVALUATION_ID`. Until now the sandbox's `uv run eval` opened a run of its own through prime-runs, so a hosted v1 eval would produce a second, unlinked evaluation while the platform's record sat at running. This adds the one knob the platform needs to run the same command a researcher runs locally. - **`run.attach <evaluation-id>`** on `EvalConfig`: `open_run` passes it to `prime_runs.init(id=...)`. prime-runs 0.1.2 (PrimeIntellect-ai/prime#913, released) attaches eval runs the way it already attached training runs: no create, sinks keyed to that id, `/finalize` on a clean finish, failure marking left to the launcher (the platform reads the sandbox exit code), and the launcher's `metadata` document left alone. Requires `push`, validated on the config. - **No local fallback when attaching.** Without `run.attach` the behaviour is unchanged: a run that cannot be opened is logged and the eval continues locally. With it, a failed attach or the SDK's `PRIME_RUNS_MODE=disabled` kill switch raises: a local run nobody reads is not a substitute while the launcher's run waits until it times out. - Docs bullet under "Common config values", `prime-runs>=0.1.2` and the relock (the lock diff is the one package). Nothing changes for local runs; `run.attach` is unset unless a launcher sets it. ## Verified against prod With this branch and prime-runs from `main` (== 0.1.2): an evaluation created launcher-style through the evals API (`metadata={"launcher": "hosted-smoke"}`), then ``` eval alphabet-sort -n 1 -r 1 --no-serve --no-rich --run.attach <that id> ``` exited 0 with `--push: completed -> .../evaluations/<that id>`. Afterwards the evaluation had `total_samples` 1, the run's metrics (`avg_reward`, `avg_metrics.alphabet_sort`, `avg_error`), its original metadata byte for byte, and status PROCESSING; no second evaluation was created. The same command without `--run.attach` still creates its own run as before (checked earlier on the SDK side). Smoke evaluations deleted. ## Test plan - New `tests/v1/test_platform_run.py` (mocked `pr.init`): attach needs push; no attach → `id=None` and the created run's id is assigned; attach → `id=<attach>` and the config carries that id; failed attach raises; failed open without attach still falls back to a local run; kill switch refuses an attach. - `tests/v1/test_configs.py` still parses every checked-in config. ruff check + format clean. ## Follow-up (not here) `EvalsBackend.attach` makes no request, so a wrong id only surfaces on the first upload (404 in the run's footer) and the eval still exits 0. For hosted evals the id is always the platform's own; the platform's v1 runner will check `total_samples` on exit rather than trusting the exit code alone. Relates to #2512 (if the `eval` console script moves, the hosted runner's command moves with it). Linear: ENG-5781. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01XrcqANSZ24BALuWS9MciYv <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes platform run lifecycle for hosted evals (hard failures on attach); local-only evals are unchanged when `run.attach` is unset. > > **Overview** > Adds **`run.attach <evaluation-id>`** so hosted sandboxes can upload into an evaluation the launcher already created instead of opening a second run. `open_run` passes that id to **`prime_runs.init(id=...)`** (bump to **0.1.2**); the config gets the same id for traces and the run dir. > > **Attach is strict:** it requires **`push`**, rejects **`PRIME_RUNS_MODE=disabled`**, and **does not fall back** to a local run on attach failure—unlike a normal push open, which still continues locally. Docs cover **`push`** and **`run.attach`** under common eval config. > > New **`tests/v1/test_platform_run.py`** mocks `pr.init` for attach validation, success paths, fallback without attach, and kill-switch behavior. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2a989a8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Add `run.attach` to stream hosted evals into existing platform runs in v1 > - `RunConfig.attach` identifies an existing platform run to attach a hosted sandbox evaluation to. The `attach_needs_push` validator rejects configurations that set `run.attach` with push disabled. > - [platform.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2569/files#diff-8644072f827be448db5b0010e0969c6c86d9bbf07191ca00a993f416d123235f) `open_run` passes the attach id to online SDK initialization. Attached evaluations fail with a runtime error if attachment cannot be performed, instead of falling back to a disabled local run. Unattached runs keep the existing fallback behavior. > - The SDK kill switch rejects attach requests before initialization, preventing a silent conversion to a local run. > - Raises the minimum `prime-runs` dependency from 0.1.1 to 0.1.2. > - Risk: `open_run` now raises an attachment-specific runtime error on SDK failures for attached runs (no local fallback); reviewers should verify callers in [platform.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2569/files#diff-8644072f827be448db5b0010e0969c6c86d9bbf07191ca00a993f416d123235f) handle this error path. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 2a989a8.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prime_runs.init(id=...)attached only training runs; an eval run was always created throughPOST /evaluations/. A hosted evaluation creates its evaluation record before the sandbox exists and hands the sandbox that id as$EVALUATION_ID, so a sandbox runninguv run evalwould produce a second, unlinked run. This makesid=attach for evals too.Note
Medium Risk
Changes eval run registration and teardown for the attach path, including what gets written to platform evaluation records on finish and failure; behavior for SDK-created runs is unchanged.
Overview
prime_runs.init(id=...)now attaches to an existing evaluation forkind="eval", not only training runs. Hosted sandboxes that receive$EVALUATION_IDcan stream samples and finalize the launcher’s record instead of creating a duplicate viaPOST /evaluations/.EvalsBackend.attachreturns aRunHandlewith no API call (same idea asRftBackend.attach).init()callsbackend.attach(id)for eval or train whenidis set and drops the oldConfigurationErrorfor eval + id.On
Run.finish, attached runs omit config on update so the launcher’s metadata document is not overwritten; only metrics/summary are sent. Non-success exits on attached runs skip finalize and failure metadata, leaving status to the launcher (aligned with attached training).Tests cover no-create attach, samples/finalize on the given id, metrics-only PUT, failure deferral, and attach without environments.
Reviewed by Cursor Bugbot for commit a767245. Bugbot is set up for automated code reviews on this repo. Configure here.