feat(v1): add prime-agent harness over native ACP - #2254
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review This PR introduces a new prime-agent harness with substantial new runtime logic (~450+ lines of new code). New features of this scope require human review. Additionally, an unresolved review comment identifies a potential bug in the resume path where the system prompt may be incorrectly dropped. You can customize Macroscope's approvability policy. Learn more. |
|
Pushed b59b806 with four fixes found by running a real eval in the docker runtime. Each was verified in a container, not inferred. The eval now completes end to endFrom the trace: What was brokenProgress was measurable at each step, which is how I knew each fix was real:
Two are worth calling out because they come from copying the Env prefix. The published release derives its prefix from its own package The Now inlined. DependencyThe silent- I verified this harness against a locally patched 0.6.0 tarball containing that fix, using the
|
Drives prime-agent's own ACP mode through the existing ACP helper rather than the pi harness's third-party pi-acp adapter. That adapter spawns `pi --mode rpc` and hard-codes pi's RPC command and event union, so prime-agent's IPython-only tool model, subagents, autonomous gates, goals, and heartbeats either degrade to a generic tool call or disappear. prime-agent speaks ACP natively as of its ACP mode, and carries the concepts ACP has no field for in a namespaced `ai.primeintellect.prime-agent` `_meta` envelope, so a rollout can observe them without the harness parsing a private protocol. Targets the current launch() contract. HOME is pinned per trace because prime-agent writes session and kernel state beneath it and concurrent rollouts must not share either.
0.6.0 is the first Prime Agent release that ships native ACP mode, so the harness can now install a published tarball that actually has --mode acp. Verified the derived tarball URL returns 200.
… key Four fixes found by running an actual eval in the docker runtime, each verified in a container rather than inferred: - Containers ship no Node and prime-agent requires >=22.8, so install failed with "npm: not found". Bootstraps Node the way the pi harness does, and exports it on PATH for the launch wrapper too, since the bundled Node is not on the container PATH. - The published release derives its env prefix from its own package piConfig, so it reads PRIME_AGENT_CODING_AGENT_DIR, not the upstream PI_ prefix. With the wrong name models.json was ignored and the run failed with "Unknown provider intercept". - prime-agent does not expand "$VAR" in models.json: it sends the literal string as the bearer token, which produced "401 unauthorized". Confirmed against a capturing HTTP server, which received `Bearer $PRIME_AGENT_INTERCEPT_KEY`. Inlines the secret instead of the pi-style indirection. With these, an eval completes end to end: 4 model calls, err 0.00, and a scored alphabet_sort reward.
…he prompt Review follow-up on two real problems. The bearer token was written in plaintext to the per-trace models.json. Concurrent rollouts share a runtime, so model-executed code in one rollout could read another rollout's credential and issue authenticated requests against its interception endpoint. This was a consequence of inlining the secret to work around prime-agent not expanding "$VAR": the pi harness's indirection had kept it out of the file. models.json now carries a placeholder, and the launch wrapper substitutes the real value from the environment at exec time under a 0700 dir and a 0600 file. The system prompt was applied twice: once via --append-system-prompt and again by the ACP runner, which seeds it into the conversation for a new session. Dropped the flag and left a note so it does not come back. Verified by rerunning the eval: ok=true, 3 model calls, 0 errors, and a scored alphabet_sort reward.
…Alpine The install guard short-circuited on the binary alone, so a runtime that already held one build reused it after `version` or `tarball_url` changed. Key the guard on the requested tarball like the pi harness keys on its versions, and give the Alpine branch the same repo-bump retry, since the official Node build is glibc-only and an older Alpine's own nodejs-current is below the 22.8 prime-agent needs.
…have prime-agent's ACP mode ignores the `mcpServers` of `session/new` entirely, and its own MCP integrations are authored Python skills the model imports in its kernel, so tool servers handed to the harness never reached the model: the repo's own echo-acp-resume-v1 fixture ran to completion with the agent reporting the tool did not exist and the reward at 0. Declare SUPPORTS_MCP false so `validate_pairing` rejects that pairing instead of degrading it.
…t segment A resumed segment replays the accreted conversation, and the ACP runner renders that transcript into the prompt with the `[system]` block the first segment already rendered into it, so passing `system_prompt` again delivered the task instructions twice. Observed on echo-user-sim-v1: the second segment's prompt carried two copies of the system prompt, one after this change.
…arly
Three install/launch edges from review, none reachable on the images this runs
on today but all of them failing obscurely when they are hit:
- the wrapper substituted the bearer token with `sed`, so a token containing
`|`, `&`, or a backslash would corrupt the key or abort the wrapper before
exec; node now rewrites the parsed models.json instead. Today's secret is
`secrets.token_urlsafe(16)`, which cannot contain those, so this is about not
depending on the generator's alphabet.
- a curl-less non-Debian image ran `apt-get` regardless and failed three steps
later inside `tar` ("tar: invalid magic"); it now says what it needs.
- an unrecognized machine fell back to the x64 Node archive and failed as
"prime-agent requires Node.js 22.8 or newer"; it is now rejected by name,
like the OS check already does.
The bucket URL reads like a stale internal endpoint next to the user-facing installer at app.primeintellect.ai/prime-agent/install.sh, so note why the harness uses it. That script is a thin front end: it defaults its own prime_agent_base_url to this bucket and downloads $base/releases/v$version/$package-$version.tgz, the same shape _tarball() builds. The value is also the prime-agent repo's R2_PUBLIC_BASE_URL variable, which its release workflow publishes to. This harness installs the npm tarball directly instead of running the script, so it needs the artifact base rather than the installer URL. Comment only; no behavior change. Verified the derived URL for the pinned version returns 200, and the docker eval still reports ok=true with 4 model calls, no errors, and a scored reward.
a0681ae to
f9513af
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f9513af. Configure here.
| # it rendered on the first segment. Re-emitting the system prompt here | ||
| # would hand the model the same instructions twice. | ||
| if trace.branches and trace.branches[-1].messages: | ||
| system_prompt = None |
There was a problem hiding this comment.
Resume drops system prompt
Medium Severity
On the non-live fallback path, launch clears system_prompt whenever the branch already has messages. Default resume already strips system messages from the replayed transcript so resolve_prompt can re-emit them, so those instructions never reach the new ACP session.
Reviewed by Cursor Bugbot for commit f9513af. Configure here.
2 participants
Problem
Verifiers v1 has no native Prime Agent harness. Routing through the
piadapter loses Prime Agent-specific behavior and does not preserve its live ACP session state.Change
prime-agentv1 harness using Prime Agent’s native ACP mode.Verification
This PR is the base of #2260, which adds the production hardening needed for long-lived concurrent sessions.
Note
Medium Risk
New harness depends on external npm tarballs and Node bootstrap in containers; API key handling and concurrent install locking are security-sensitive but scoped per trace with explicit MCP rejection for tool tasksets.
Overview
Adds a
prime-agentv1 harness that runs Prime Agent in native ACP mode via the sharedACPhelper, instead of going through thepiadapter.PrimeAgentHarnessbootstraps Node (≥22.8) and installs a pinned npm release (default 0.6.0) under a locked shared path, wires model calls through the interception endpoint as an OpenAI-compatible provider, and launchesprime-agent --mode acpwith per-traceHOME(session + IPython kernel isolation). It setsSUPPORTS_MCP = falseso tool-bearing tasksets fail pairing validation, supports resume/skills/autonomous gates, injects API keys at exec time through a wrapper script, and removes.vf-prime-agent-{trace.id}on cleanup.Tests: new
prime-agent-persistence-v1fixture and docker e2etest_prime_agent_persists_native_acp_sessioncheck IPython state across two interaction turns, rollout dir cleanup, and native (non–transcript-replay) message shape; pytestprime_agentmark and harness list docs are updated.Reviewed by Cursor Bugbot for commit f9513af. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add prime-agent harness over native ACP for v1 e2e evaluation
PrimeAgentHarnessin harness.py, which installs the prime-agent npm tarball into/tmp/vf-prime-agenton first use and launches it in ACP mode with per-trace isolated working directories._preparewrites amodels.jsonrouting through an interception endpoint and generates a wrapper script that injects the API key at launch time, with restricted file permissions (700/600) to limit secret exposure.SUPPORTS_RESUME=True), skipping the system prompt on resumed segments, and optional autonomous mode with configurable gates.PrimeAgentPersistenceEnvand matching taskset in prime_agent_persistence_v1.py that validates IPython kernel state persists across two interaction turns without reloading from conversation text.flock/lockfand a Node one-liner during setup and launch; failures in those steps will surface as non-zero exit codes with no structured error wrapping.📊 Macroscope summarized a0681ae. 1 file reviewed, 5 issues evaluated, 5 issues filtered, 0 comments posted
🗂️ Filtered Issues
verifiers/v1/harnesses/prime_agent/harness.py — 0 comments posted, 5 evaluated, 5 filtered
Harness.resumeremoves existing system messages from the branch wheneverdata.system_promptis set (line 203 ofharness.py), then callslaunch; because that branch is nonempty, this line sets the otherwise re-emittedsystem_prompttoNone. The ACP runner consequently receives neither a system message in the replayed prompt nor its separate system prompt, so resumed segments run without the task instructions. [ Already posted ]launchcreates per-trace agent directories containing ACP sessions, kernel state, wrapper, and (after execution) the bearer-keymodels.json, butPrimeAgentHarnessnever overridescleanup. The baseHarness.cleanupis a no-op, so on a reused runtime every completed rollout leaves this state behind indefinitely, consuming disk and retaining sensitive conversation/session data. [ Already posted ]chmod 700/chmod 600calls do not isolate concurrent rollouts: every agent and its model-executed IPython code runs as the same runtime user, so it can list another trace's.vf-prime-agent-*directory and read that directory'smodels.jsonafter its wrapper has replaced the placeholder withPRIME_AGENT_INTERCEPT_KEY. This exposes another rollout's bearer credential despite the intended per-trace isolation. [ Already posted ]chmod 700on eachagent_dirdoes not isolate concurrent rollouts because all agent subprocesses in a shared runtime run as the same Unix user. After the wrapper replaces the placeholder, any model-executed code can enumerate sibling.vf-prime-agent-*directories and read theirmodels.json(mode600is also readable by that same user), exposing another rollout's interception credential. Per-rollout OS users or another isolation mechanism are needed; permissions alone do not provide the claimed credential isolation. [ Already posted ]PRIME_AGENT_ACP.runomitssession_path. ConsequentlyACP.runsuppliesNone, and its runner treats every segment as new and callsconnection.new_sessionrather thanresume_session/load_session. Resumed interactions therefore replay the transcript instead of continuing the native ACP session, losing any session state that is not representable in the messages. [ Previously rejected ]