Repository navigation
Conversation
c0faec4 to
8734f52
Compare
An existing but malformed reward.json currently falls back to reward.txt, which can turn an invalid verifier result into a passing score. Honor JSON precedence whenever the file exists: malformed, unreadable, oversized, non-object, or non-finite JSON fails grading. Text fallback is allowed only when JSON is absent. Stale reward files are cleared before verification, and a valid score remains authoritative even if test.sh exits nonzero. Add task.reward_key (default: reward) to select the scalar from named verifier outputs. Missing selected keys fail explicitly instead of implicitly summing dimensions. Unselected named values remain trace metrics. Shared and separate grading use the same selection path; reward.txt defines the reward key. Stack: #2806 → #2807 → this PR. This PR targets codex/harbor-verifier-resolution and remains independent of the deployment refactor (#2803). Validation before removing the PR-added e2e tests: full suite with Harbor installed: 152 passed, 131 skipped. All 18 Harbor-focused checks passed, including real Docker verifier scripts covering JSON/text precedence, missing files, malformed/empty/scalar/string/bool/non-finite JSON, oversized and unreadable files, named selection, stale-file cleanup, nonzero exits, and separate image-owned grading. Documented TOML validated. Ruff, all pre-commit hooks, and pre-push CI-parity type checking passed. Additional validation on real Prime VMs at combined-stack commit 4d82a7d: all 16 verifier/reward checks and 5 complete HarborEnv episodes passed their assertions. Episodes cover shared and fresh grading, valid named rewards, deliberate invalid-JSON failures with reward 0, and Compose with sidecar artifact restoration into a fresh grader. Cleanup audit found zero remaining labeled VMs. Tests used a deterministic harness without model calls; the dedicated custom-image/bundled-tests case was validated on Docker only. These Prime results apply to the combined stack, not independent runs of each intermediate branch. No test changes are included in this stack. The integration validation above was completed against identical production code. Current combined-stack contributor checks: 134 passed, 131 skipped; Ruff and all pre-commit hooks passed. <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Enforce Harbor reward precedence and add explicit `reward_key` selection > - Makes an existing `reward.json` authoritative: invalid, unreadable, or malformed JSON now fails grading, and `reward.txt` fallback only runs when the JSON file is absent. See [taskset.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2808/files#diff-2bff5dabcb9a41f4d830e8bbb890e8cc6f5b8331ef2ac4c3b19ffad0e046fe00). > - Adds `HarborTaskConfig.reward_key` (default `reward`) to select which named verifier value becomes the training reward; all other named values are kept as trace metrics. > - `reward.json` now accepts only a string-to-float mapping; scalar text rewards from `reward.txt` are validated separately as finite floats. > - A nonzero verifier exit no longer overrides a valid reward file; a missing selected key fails grading with a `TaskError`. > - Risk: scalar JSON rewards that previously passed validation now raise `TaskError` in `HarborTask.run_verifier`; reward files read through `read_reward_json` no longer fall back to text on validation failure. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 4c2f4da.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
76edaa1 to
83bd375
Compare
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — Split execution places the harness on a separate Prime VM while tools run in the workspace. Unresolved findings concern credential exposure, processes surviving into scoring or evidence collection, and workspace reclamation during startup; these require resolution before relying on the new isolation boundary. Needs human approval because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
83bd375 to
8e2eeb5
Compare
|
Macroscope has since reviewed this pull request. An earlier review was skipped by a cost limit; a review has now completed, so that notice no longer applies. |
8e2eeb5 to
96ece3f
Compare
96ece3f to
4a8cf93
Compare
4a8cf93 to
7546c1b
Compare
With deployment.mode = "split", the harness runs on a Prime VM of its own that joins the rollout's private LAN, and its tools execute in the workspace. The workspace is placed exactly as in colocated mode, so plain images, ServicesSpec services, native Compose services and tool server VMs all split the same way. - The harness VM may connect to the workspace and to tool server VMs; nothing may connect to it. It takes the task's network policy plus the model and tool routes, and the workspace keeps no route to the model. - ExecutionConnection.serve starts a worker in the workspace that listens on the LAN, in a cgroup of its own; closing the harness session kills the worker and every process its tools started. - RLM runs its native tools and builtin skills through nano-rlm's TCP worker. - Harness and worker setup run concurrently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A workspace service that already listened on the worker's random port made the readiness probe succeed for the wrong process. The worker is now ready only once a process in its cgroup listens on the port; a port another process holds before the worker is released is skipped for a fresh one, up to three times. Setting deployment.harness.creates_per_min to 0 no longer inherits the workspace's rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split deployment accepted tool servers placed in a ServicesSpec or Compose service, but the LAN let the harness connect only to the workspace and to tool server VMs, so those tools failed after provisioning. The rollout now passes the services that host tool servers to the deployment, and the harness may connect to those services, not to the others. Pin nano-rlm to its split-execution head, which stops kernel commands with the kernel and keeps worker interrupts off the supervisor's event loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root read the worker's PID from a file the worker's user can write, moved that process into the worker's cgroup, and could kill it by that PID on cleanup, so a forged PID file could have aimed root at any process. Root now acts only on a PID it has checked to belong to the worker's user and to carry this launch's release flag; cleanup kills the cgroup, or that checked PID after checking it again, and never reads the file. The release flag lives in a root-owned directory, and the worker's log is read as the worker's user. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split deployment is a grid capability: the Split Execution job runs one Prime cell per bundled harness, and a harness without SUPPORTS_SPLIT_EXECUTION is collected as a skip with its reason, so a harness that opts in is tested without a test change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker's launch had grown into root-run shell that held the worker, moved it into a cgroup, matched its listening socket through /proc, and checked its identity against a PID file the worker's user could write. Two choices caused all of it: picking the worker's port, and cleaning up through a cgroup, which needs root. The worker now binds a port the kernel picks and announces it, so there is no port to choose or collide on. It runs with a VF_EXEC_ID marker in its environment, which every process its tools start inherits through detaching and reparenting; close() kills the processes carrying it as the worker's own user, so the cleanup reaches nothing that user could not signal itself. A process that empties its environment escapes it; the workspace VM still ends with the rollout. Pin nano-rlm to the head that announces its bound port. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The head stops native shell commands with their turn and session, and names the worker's history in compaction notices under split execution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker that crashes right after announcing its port leaves both its listen line and the exit marker in the log by the first read; serve returned the port. Check for the exit marker first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The head syncs the session ledger to the execution worker incrementally, so long split sessions no longer outgrow one 16 MiB frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| backend.prepare_execution(own(name, backend)) | ||
| for name, backend in d.target_backends.items() | ||
| ), | ||
| *([self._harness.prepare_execution(routes)] if self._harness else []), |
There was a problem hiding this comment.
Harness VM stays up during evidence collection
Medium Severity
stop_execution still only stops the workspace, so a split harness remains on the LAN while service evidence is collected. That VM keeps its harness → tool service rules, so leftover harness processes can still reach those services after the agent session has closed.
Reviewed by Cursor Bugbot for commit 2e25eb1. Configure here.
Closing the execution connection killed every process that inherited the worker's marker, including daemons the agent started, while a colocated harness's exit leaves them running. The harness closes before grading when the agent did not finish, so a daemon the task needs was gone only in split mode. nano-rlm stops what is still attached to a session as the session closes, so an RLM worker's close now stops just the worker and its own process group. Codex's exec-server keeps the full stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| ) -> None: | ||
| # The same pinned nano-rlm runs the tools in the workspace; the agent loop and | ||
| # the model credentials stay on the harness's VM. | ||
| await self._install(workspace) |
There was a problem hiding this comment.
🔴 Critical rlm/harness.py:200
setup_execution() passes self.config.resolved_env into the install process on the task VM, exposing forwarded harness credentials such as SERPER_API_KEY to commands there. Use an allowlisted environment for the workspace installation so harness secrets stay on the harness VM.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/rlm/harness.py around line 200:
`setup_execution()` passes `self.config.resolved_env` into the install process on the task VM, exposing forwarded harness credentials such as `SERPER_API_KEY` to commands there. Use an allowlisted environment for the workspace installation so harness secrets stay on the harness VM.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 038e388. Configure here.
| asyncio.create_task( | ||
| _keepalive(workspace.bind(), max(10.0, idle_timeout / 3)) | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Workspace keepalive starts after sibling boot
Medium Severity
The workspace keepalive in split mode starts only after the harness and every member VM finish booting, even though the workspace's idle clock starts as soon as its own start() returns. Sibling image pulls or first-use builds can outlast idle_timeout, so the platform can reclaim the workspace before setup runs. Member VMs already start their keepalive as soon as each one is up.
Reviewed by Cursor Bugbot for commit 038e388. Configure here.
nano-rlm#218 runs the supervisor's shell jobs, services and path watches in the workspace, so split execution refuses none of them and keeps the runtime prompt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


Based on main, which now includes the #2910 native-services stack (#2910, #2918, #2999); followed by #2831 and #2850. Companion: PrimeIntellect-ai/nano-rlm#218.
Tool execution shares the harness's VM, so the agent loop and the model credentials live where the agent's commands run. With
deployment.mode = "split", the harness runs on a Prime VM of its own that joins the rollout's private LAN, and its tools execute in the workspace.ServicesSpecservices, native Compose services and tool server VMs all split the same way.deployment.harnesssizes the harness VM (1 CPU and 2 GB by default); it shares the workspace's labels, region and creation rate.ToolsetConfig.service), and nothing may open one to it. It takes the task's network policy plus the model and tool routes; the workspace keeps no route to the model.ServicesSpecinterconnect rule from Run Prime task services as native VMs and fix Harbor setup #2910 changes from* → *totag:vf-task → tag:vf-task, and every member except the harness,mainincluded, carriesvf-task. Without a harness VM this is the same connectivity; with one, it keeps task services from reaching the harness. The native Compose per-network rules are unchanged, and the harness rule is added beside them.ExecutionConnection.servestarts a worker in the workspace, as the workspace's agent user. The worker binds a port the kernel picks and announces it, so there is no port to choose or collide on. It carries aVF_EXEC_IDmarker in its environment that every process its tools start inherits, through detaching and reparenting. Closing the harness session kills every process carrying the marker, as the worker's own user, so the cleanup needs no root and reaches nothing that user could not signal itself. Withkeep_detached, which RLM uses, it kills only the worker and its own process group: nano-rlm stops what is still attached to a session as the session closes, so a daemon the agent started keeps running, as after a colocated harness exits. That matters because the harness closes before grading when the agent did not finish. Codex's exec-server keeps the full cleanup.rlm.shell,rlm.services,rlm.watch.path): nothing is refused, and the agent gets the same runtime prompt as in colocated mode. The harness and worker installs run concurrently.This replaces the earlier nested-Docker design (a Docker workspace inside the harness VM). That design depended on the Docker-in-VM support that #2910 removes, and it pulled the task image inside every rollout's VM.
Validation
CI: a new Split Execution integration job runs
tests/integration/test_split_execution.pyover the harness grid. Split deployment is a grid capability, so every harness withSUPPORTS_SPLIT_EXECUTIONgets a Prime cell (RLM here; Codex from Support Codex native tools in split deployments #2850), and the others are collected as skips with their reason.Unit suite, ruff and ty pass.
Live Prime runs with a scripted tool-calling model: RLM's tools ran in the workspace (the workspace's files, working directory and environment; no credentials in the workspace's environment) and scored 1.0, including with the workspace's egress denied.
LAN reachability on real VMs for split plus
ServicesSpec, split plus a tool server VM, and split plus native Compose: the harness reaches the worker in the workspace; the workspace, services and tool VMs cannot reach the harness; the harness cannot reach task services; the workspace still reaches its Compose peers by name.Worker lifecycle on real VMs: a worker as a non-root user comes up in 0.3 s on its announced port, and after close the worker and a detached
setsidchild are gone; a root process carrying a copy of the marker survives that user's cleanup; a crashing worker fails in 0.3 s with its log. Withkeep_detached, close stops the worker and a child in its group and leaves a child in a group of its own, asetsiddaemon and a job in a session of its own running. End to end, an agent that never finishes starts a daemon that does the task once the harness's process is gone: it scores 1.0 in split and colocated, and 0 before this change.Each tool call crosses the LAN: about 25 ms round trip in these runs, and up to about 200 ms between distant regions.
nano-rlm: nano-rlm#218 has merged, and the RLM pin is its merge commit (
208aea9).🤖 Generated with Claude Code
Note
Medium Risk
Introduces a new rollout deployment path that changes VM placement, LAN firewall rules, and where model credentials and tool workers run; mitigated by opt-in mode, upfront validation, and integration tests.
Overview
Adds
deployment.mode = "split"so the agent loop runs on a dedicated Prime harness VM while tool execution stays in the task workspace.DeploymentConfiggains aharnesssizing block; split mode requires a Prime workspace and is rejected for borrowed runtimes, colocated task tools, and non-colocated agentic judges.Rollout and networking: provisioning spins up a second VM on the rollout LAN, records it as
trace.agent.harness_runtime, routes model interception and harness sessions through the harness host, and moves model/tool egress routes off the workspace onto the harness VM.ExecutionConnectionstarts LAN workers in the workspace (kernel-chosen ports,VF_EXEC_IDmarker-based teardown) for harnesses that implementsetup_execution().Harness contract: opt-in via
SUPPORTS_SPLIT_EXECUTIONwith preflight inAgent._rollout_params; RLM is the bundled implementer (pinned nano-rlm ref, TCP execution worker, builtin skills only in split). Private-network rules gain harness-specific LAN tags so the harness can reach the workspace and tool VMs but not vice versa.Tests/docs: new integration grid behavior
split,test_split_execution.pyfixture, CI job, anddocs/runtimes.mdsection for split deployment.Reviewed by Cursor Bugbot for commit e4b0ab1. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add split deployment mode running harness on a separate Prime VM beside the workspace
splitdeployment mode where the harness runs on its own Prime VM next to the workspace VM.DeploymentConfiggains asplitmode plus aharnessPrime runtime field, and the private-network strategy starts a second backend whose config merges workspace labels, region, and network policy (strategies.py, deployment.py).Rollout.openruns harness setup andsetup_execution()concurrently on the harness runtime, records the harness runtime inAgentInfo.harness_runtime, and cleans up the harness against the harness VM (rollout.py, trace.py).ExecutionConnection(exported viaverifiers.v1) that starts a marked workspace execution worker, discovers its port from logs within 60s, and kills marked processes on close (deployment.py). RLM harness implementssetup_execution()for split mode; the baseHarnessaddsSUPPORTS_SPLIT_EXECUTIONand rejects unsupported split setups.AgentConfigwith split mode requires a Prime workspace runtime;Agent.provision()raisesValueErrorfor split agents; agentic judge agents must stay colocated;RLMHarnessConfigdefaults to a new nano-rlm version pin; split RLM setup rejects external skills.Macroscope summarized e4b0ab1.