[kcoopermiller]

Summary

A hosted evaluation creates its evaluation record before the sandbox exists and hands the sandbox that id as $EVALUATION_ID. Until now the sandbox's uv run eval opened a run of its own through prime-runs, so a hosted v1 eval would produce a second, unlinked evaluation while the platform's record sat at running. This adds the one knob the platform needs to run the same command a researcher runs locally.

• run.attach <evaluation-id> on EvalConfig: open_run passes it to prime_runs.init(id=...). prime-runs 0.1.2 (feat(runs): attach eval runs to an evaluation a launcher already created prime#913, released) attaches eval runs the way it already attached training runs: no create, sinks keyed to that id, /finalize on a clean finish, failure marking left to the launcher (the platform reads the sandbox exit code), and the launcher's metadata document left alone. Requires push, validated on the config.
• No local fallback when attaching. Without run.attach the behaviour is unchanged: a run that cannot be opened is logged and the eval continues locally. With it, a failed attach or the SDK's PRIME_RUNS_MODE=disabled kill switch raises: a local run nobody reads is not a substitute while the launcher's run waits until it times out.
• Docs bullet under "Common config values", prime-runs>=0.1.2 and the relock (the lock diff is the one package).

Nothing changes for local runs; run.attach is unset unless a launcher sets it.

Verified against prod

With this branch and prime-runs from main (== 0.1.2): an evaluation created launcher-style through the evals API (metadata={"launcher": "hosted-smoke"}), then

eval alphabet-sort -n 1 -r 1 --no-serve --no-rich --run.attach <that id>

exited 0 with --push: completed -> .../evaluations/<that id>. Afterwards the evaluation had total_samples 1, the run's metrics (avg_reward, avg_metrics.alphabet_sort, avg_error), its original metadata byte for byte, and status PROCESSING; no second evaluation was created. The same command without --run.attach still creates its own run as before (checked earlier on the SDK side). Smoke evaluations deleted.

Test plan

• New tests/v1/test_platform_run.py (mocked pr.init): attach needs push; no attach → id=None and the created run's id is assigned; attach → id=<attach> and the config carries that id; failed attach raises; failed open without attach still falls back to a local run; kill switch refuses an attach.
• tests/v1/test_configs.py still parses every checked-in config. ruff check + format clean.

Follow-up (not here)

EvalsBackend.attach makes no request, so a wrong id only surfaces on the first upload (404 in the run's footer) and the eval still exits 0. For hosted evals the id is always the platform's own; the platform's v1 runner will check total_samples on exit rather than trusting the exit code alone. Relates to #2512 (if the eval console script moves, the hosted runner's command moves with it).

Linear: ENG-5781.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XrcqANSZ24BALuWS9MciYv

---

Note

Medium Risk
Changes platform run lifecycle for hosted evals (hard failures on attach); local-only evals are unchanged when run.attach is unset.

Overview
Adds run.attach <evaluation-id> so hosted sandboxes can upload into an evaluation the launcher already created instead of opening a second run. open_run passes that id to prime_runs.init(id=...) (bump to 0.1.2); the config gets the same id for traces and the run dir.

Attach is strict: it requires push, rejects PRIME_RUNS_MODE=disabled, and does not fall back to a local run on attach failure—unlike a normal push open, which still continues locally. Docs cover push and run.attach under common eval config.

New tests/v1/test_platform_run.py mocks pr.init for attach validation, success paths, fallback without attach, and kill-switch behavior.

^{Reviewed by Cursor Bugbot for commit 2a989a8. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add run.attach to stream hosted evals into existing platform runs in v1

• RunConfig.attach identifies an existing platform run to attach a hosted sandbox evaluation to. The attach_needs_push validator rejects configurations that set run.attach with push disabled.
• platform.py open_run passes the attach id to online SDK initialization. Attached evaluations fail with a runtime error if attachment cannot be performed, instead of falling back to a disabled local run. Unattached runs keep the existing fallback behavior.
• The SDK kill switch rejects attach requests before initialization, preventing a silent conversion to a local run.
• Raises the minimum prime-runs dependency from 0.1.1 to 0.1.2.
• Risk: open_run now raises an attachment-specific runtime error on SDK failures for attached runs (no local fallback); reviewers should verify callers in platform.py handle this error path.

^{Macroscope summarized 2a989a8.}

[macroscopeapp]

Approvability

Verdict: Approved at 2a989a8

Macroscope's review found this PR approvable — This adds a small, opt-in run.attach path for hosted evaluations while preserving existing local and ordinary push behavior. The dependency, validation, failure handling, and attach/fallback cases are covered in the focused tests, with no schema or deployment impact.

^{You can add or adjust custom eligibility rules. Learn more.}