Skip to content

Run split harnesses on Prime VMs beside the workspace - #2803

Open
hallerite wants to merge 12 commits into
mainfrom
codex/split-execution
Open

hallerite wants to merge 12 commits into
mainfrom
codex/split-execution

Conversation

@hallerite

@hallerite hallerite commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Based on main, which now includes the #2910 native-services stack (#2910, #2918, #2999); followed by #2831 and #2850. Companion: PrimeIntellect-ai/nano-rlm#218.

Tool execution shares the harness's VM, so the agent loop and the model credentials live where the agent's commands run. With deployment.mode = "split", the harness runs on a Prime VM of its own that joins the rollout's private LAN, and its tools execute in the workspace.

  • Placement: the workspace is placed exactly as in colocated mode, so plain images, ServicesSpec services, native Compose services and tool server VMs all split the same way. deployment.harness sizes the harness VM (1 CPU and 2 GB by default); it shares the workspace's labels, region and creation rate.
  • Network: the harness VM may open connections to the workspace, to tool server VMs and to task services that host tool servers (ToolsetConfig.service), and nothing may open one to it. It takes the task's network policy plus the model and tool routes; the workspace keeps no route to the model.
  • Shared LAN rules (changes Run Prime task services as native VMs and fix Harbor setup #2910's code): the ServicesSpec interconnect rule from Run Prime task services as native VMs and fix Harbor setup #2910 changes from * → * to tag:vf-task → tag:vf-task, and every member except the harness, main included, carries vf-task. Without a harness VM this is the same connectivity; with one, it keeps task services from reaching the harness. The native Compose per-network rules are unchanged, and the harness rule is added beside them.
  • Execution: ExecutionConnection.serve starts a worker in the workspace, as the workspace's agent user. The worker binds a port the kernel picks and announces it, so there is no port to choose or collide on. It carries a VF_EXEC_ID marker in its environment that every process its tools start inherits, through detaching and reparenting. Closing the harness session kills every process carrying the marker, as the worker's own user, so the cleanup needs no root and reaches nothing that user could not signal itself. With keep_detached, which RLM uses, it kills only the worker and its own process group: nano-rlm stops what is still attached to a session as the session closes, so a daemon the agent started keeps running, as after a colocated harness exits. That matters because the harness closes before grading when the agent did not finish. Codex's exec-server keeps the full cleanup.
  • RLM: native tools and builtin skills run through nano-rlm's token-authenticated TCP worker, and so do the supervisor's shell jobs, services and path watches (rlm.shell, rlm.services, rlm.watch.path): nothing is refused, and the agent gets the same runtime prompt as in colocated mode. The harness and worker installs run concurrently.
  • Rejected before provisioning: harnesses without split support, custom skills, colocated task tools, borrowed runtimes and non-Prime workspaces.

This replaces the earlier nested-Docker design (a Docker workspace inside the harness VM). That design depended on the Docker-in-VM support that #2910 removes, and it pulled the task image inside every rollout's VM.

Validation

  • CI: a new Split Execution integration job runs tests/integration/test_split_execution.py over the harness grid. Split deployment is a grid capability, so every harness with SUPPORTS_SPLIT_EXECUTION gets a Prime cell (RLM here; Codex from Support Codex native tools in split deployments #2850), and the others are collected as skips with their reason.

  • Unit suite, ruff and ty pass.

  • Live Prime runs with a scripted tool-calling model: RLM's tools ran in the workspace (the workspace's files, working directory and environment; no credentials in the workspace's environment) and scored 1.0, including with the workspace's egress denied.

  • LAN reachability on real VMs for split plus ServicesSpec, split plus a tool server VM, and split plus native Compose: the harness reaches the worker in the workspace; the workspace, services and tool VMs cannot reach the harness; the harness cannot reach task services; the workspace still reaches its Compose peers by name.

  • Worker lifecycle on real VMs: a worker as a non-root user comes up in 0.3 s on its announced port, and after close the worker and a detached setsid child are gone; a root process carrying a copy of the marker survives that user's cleanup; a crashing worker fails in 0.3 s with its log. With keep_detached, close stops the worker and a child in its group and leaves a child in a group of its own, a setsid daemon and a job in a session of its own running. End to end, an agent that never finishes starts a daemon that does the task once the harness's process is gone: it scores 1.0 in split and colocated, and 0 before this change.

  • Each tool call crosses the LAN: about 25 ms round trip in these runs, and up to about 200 ms between distant regions.

nano-rlm: nano-rlm#218 has merged, and the RLM pin is its merge commit (208aea9).

🤖 Generated with Claude Code


Note

Medium Risk
Introduces a new rollout deployment path that changes VM placement, LAN firewall rules, and where model credentials and tool workers run; mitigated by opt-in mode, upfront validation, and integration tests.

Overview
Adds deployment.mode = "split" so the agent loop runs on a dedicated Prime harness VM while tool execution stays in the task workspace. DeploymentConfig gains a harness sizing block; split mode requires a Prime workspace and is rejected for borrowed runtimes, colocated task tools, and non-colocated agentic judges.

Rollout and networking: provisioning spins up a second VM on the rollout LAN, records it as trace.agent.harness_runtime, routes model interception and harness sessions through the harness host, and moves model/tool egress routes off the workspace onto the harness VM. ExecutionConnection starts LAN workers in the workspace (kernel-chosen ports, VF_EXEC_ID marker-based teardown) for harnesses that implement setup_execution().

Harness contract: opt-in via SUPPORTS_SPLIT_EXECUTION with preflight in Agent._rollout_params; RLM is the bundled implementer (pinned nano-rlm ref, TCP execution worker, builtin skills only in split). Private-network rules gain harness-specific LAN tags so the harness can reach the workspace and tool VMs but not vice versa.

Tests/docs: new integration grid behavior split, test_split_execution.py fixture, CI job, and docs/runtimes.md section for split deployment.

Reviewed by Cursor Bugbot for commit e4b0ab1. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add split deployment mode running harness on a separate Prime VM beside the workspace

  • Adds a split deployment mode where the harness runs on its own Prime VM next to the workspace VM. DeploymentConfig gains a split mode plus a harness Prime runtime field, and the private-network strategy starts a second backend whose config merges workspace labels, region, and network policy (strategies.py, deployment.py).
  • Split network rules allow harness-initiated connections to the workspace, tool VMs, and task services, but no inbound connections to the harness (private_network.py).
  • Rollout.open runs harness setup and setup_execution() concurrently on the harness runtime, records the harness runtime in AgentInfo.harness_runtime, and cleans up the harness against the harness VM (rollout.py, trace.py).
  • Adds ExecutionConnection (exported via verifiers.v1) that starts a marked workspace execution worker, discovers its port from logs within 60s, and kills marked processes on close (deployment.py). RLM harness implements setup_execution() for split mode; the base Harness adds SUPPORTS_SPLIT_EXECUTION and rejects unsupported split setups.
  • Adds integration tests for split execution and preflight rejection, plus a split fixture taskset and docs (test_split_execution.py, runtimes.md).
  • Behavioral Change: AgentConfig with split mode requires a Prime workspace runtime; Agent.provision() raises ValueError for split agents; agentic judge agents must stay colocated; RLMHarnessConfig defaults to a new nano-rlm version pin; split RLM setup rejects external skills.

Macroscope summarized e4b0ab1.

@hallerite hallerite changed the title feat: separate RLM harness and task execution on Prime VMs Unify deployments, split tool execution, and Harbor task integration Oct 8, 2026
@hallerite
hallerite force-pushed the codex/split-execution branch from c0faec4 to 8734f52 Compare October 8, 2026 02:20
xeophon added a commit that referenced this pull request Oct 8, 2026
An existing but malformed reward.json currently falls back to
reward.txt, which can turn an invalid verifier result into a passing
score. Honor JSON precedence whenever the file exists: malformed,
unreadable, oversized, non-object, or non-finite JSON fails grading.
Text fallback is allowed only when JSON is absent. Stale reward files
are cleared before verification, and a valid score remains authoritative
even if test.sh exits nonzero.

Add task.reward_key (default: reward) to select the scalar from named
verifier outputs. Missing selected keys fail explicitly instead of
implicitly summing dimensions. Unselected named values remain trace
metrics. Shared and separate grading use the same selection path;
reward.txt defines the reward key.

Stack: #2806 → #2807 → this PR. This PR targets
codex/harbor-verifier-resolution and remains independent of the
deployment refactor (#2803).

Validation before removing the PR-added e2e tests: full suite with
Harbor installed: 152 passed, 131 skipped. All 18 Harbor-focused checks
passed, including real Docker verifier scripts covering JSON/text
precedence, missing files, malformed/empty/scalar/string/bool/non-finite
JSON, oversized and unreadable files, named selection, stale-file
cleanup, nonzero exits, and separate image-owned grading. Documented
TOML validated. Ruff, all pre-commit hooks, and pre-push CI-parity type
checking passed.

Additional validation on real Prime VMs at combined-stack commit
4d82a7d: all 16 verifier/reward checks
and 5 complete HarborEnv episodes passed their assertions. Episodes
cover shared and fresh grading, valid named rewards, deliberate
invalid-JSON failures with reward 0, and Compose with sidecar artifact
restoration into a fresh grader. Cleanup audit found zero remaining
labeled VMs. Tests used a deterministic harness without model calls; the
dedicated custom-image/bundled-tests case was validated on Docker only.
These Prime results apply to the combined stack, not independent runs of
each intermediate branch.

No test changes are included in this stack. The integration validation
above was completed against identical production code. Current
combined-stack contributor checks: 134 passed, 131 skipped; Ruff and all
pre-commit hooks passed.

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Enforce Harbor reward precedence and add explicit `reward_key`
selection
> - Makes an existing `reward.json` authoritative: invalid, unreadable,
or malformed JSON now fails grading, and `reward.txt` fallback only runs
when the JSON file is absent. See
[taskset.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2808/files#diff-2bff5dabcb9a41f4d830e8bbb890e8cc6f5b8331ef2ac4c3b19ffad0e046fe00).
> - Adds `HarborTaskConfig.reward_key` (default `reward`) to select
which named verifier value becomes the training reward; all other named
values are kept as trace metrics.
> - `reward.json` now accepts only a string-to-float mapping; scalar
text rewards from `reward.txt` are validated separately as finite
floats.
> - A nonzero verifier exit no longer overrides a valid reward file; a
missing selected key fails grading with a `TaskError`.
> - Risk: scalar JSON rewards that previously passed validation now
raise `TaskError` in `HarborTask.run_verifier`; reward files read
through `read_reward_json` no longer fall back to text on validation
failure.
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
4c2f4da.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- Macroscope's pull request summary ends here -->
@hallerite
hallerite force-pushed the codex/split-execution branch 3 times, most recently from 76edaa1 to 83bd375 Compare October 9, 2026 08:05
@hallerite
hallerite marked this pull request as ready for review October 12, 2026 07:46
Comment thread verifiers/v1/grading.py Resolved
Comment thread verifiers/v1/runtimes/compose.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/runtimes/compose.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/tasksets/harbor/taskset.py Resolved
@macroscopeapp

macroscopeapp Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — Split execution places the harness on a separate Prime VM while tools run in the workspace. Unresolved findings concern credential exposure, processes surviving into scoring or evidence collection, and workspace reclamation during startup; these require resolution before relying on the new isolation boundary.

Needs human approval because:

  • 1 blocking correctness issue found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

@hallerite
hallerite force-pushed the codex/split-execution branch from 83bd375 to 8e2eeb5 Compare October 12, 2026 09:18
@macroscopeapp

macroscopeapp Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Macroscope has since reviewed this pull request. An earlier review was skipped by a cost limit; a review has now completed, so that notice no longer applies.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
@hallerite
hallerite force-pushed the codex/split-execution branch from 8e2eeb5 to 96ece3f Compare October 12, 2026 09:29
@hallerite
hallerite changed the base branch from main to codex/runtime-handles October 12, 2026 09:29
Comment thread verifiers/v1/runtimes/compose.py Resolved
@hallerite
hallerite force-pushed the codex/split-execution branch from 96ece3f to 4a8cf93 Compare October 12, 2026 09:42

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/runtimes/deployment.py Resolved
@hallerite
hallerite marked this pull request as draft October 12, 2026 09:58
@hallerite
hallerite force-pushed the codex/split-execution branch from 4a8cf93 to 7546c1b Compare October 12, 2026 13:07
@hallerite
hallerite marked this pull request as ready for review October 12, 2026 14:29

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/grading.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/runtimes/strategies.py Resolved
Comment thread verifiers/v1/runtimes/deployment.py Resolved
Comment thread verifiers/v1/runtimes/strategies.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/agent.py Resolved
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 30, 2026
Comment thread verifiers/v1/runtimes/deployment.py Resolved
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 30, 2026
mikasenghaas
mikasenghaas previously approved these changes Oct 31, 2026
hallerite and others added 6 commits October 31, 2026 01:05
With deployment.mode = "split", the harness runs on a Prime VM of its own that
joins the rollout's private LAN, and its tools execute in the workspace. The
workspace is placed exactly as in colocated mode, so plain images, ServicesSpec
services, native Compose services and tool server VMs all split the same way.

- The harness VM may connect to the workspace and to tool server VMs; nothing
  may connect to it. It takes the task's network policy plus the model and tool
  routes, and the workspace keeps no route to the model.
- ExecutionConnection.serve starts a worker in the workspace that listens on
  the LAN, in a cgroup of its own; closing the harness session kills the worker
  and every process its tools started.
- RLM runs its native tools and builtin skills through nano-rlm's TCP worker.
- Harness and worker setup run concurrently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A workspace service that already listened on the worker's random port made
the readiness probe succeed for the wrong process. The worker is now ready
only once a process in its cgroup listens on the port; a port another process
holds before the worker is released is skipped for a fresh one, up to three
times. Setting deployment.harness.creates_per_min to 0 no longer inherits the
workspace's rate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split deployment accepted tool servers placed in a ServicesSpec or Compose
service, but the LAN let the harness connect only to the workspace and to tool
server VMs, so those tools failed after provisioning. The rollout now passes
the services that host tool servers to the deployment, and the harness may
connect to those services, not to the others.

Pin nano-rlm to its split-execution head, which stops kernel commands with the
kernel and keeps worker interrupts off the supervisor's event loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root read the worker's PID from a file the worker's user can write, moved that
process into the worker's cgroup, and could kill it by that PID on cleanup, so
a forged PID file could have aimed root at any process. Root now acts only on a
PID it has checked to belong to the worker's user and to carry this launch's
release flag; cleanup kills the cgroup, or that checked PID after checking it
again, and never reads the file. The release flag lives in a root-owned
directory, and the worker's log is read as the worker's user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split deployment is a grid capability: the Split Execution job runs one Prime
cell per bundled harness, and a harness without SUPPORTS_SPLIT_EXECUTION is
collected as a skip with its reason, so a harness that opts in is tested
without a test change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The worker's launch had grown into root-run shell that held the worker, moved
it into a cgroup, matched its listening socket through /proc, and checked its
identity against a PID file the worker's user could write. Two choices caused
all of it: picking the worker's port, and cleaning up through a cgroup, which
needs root.

The worker now binds a port the kernel picks and announces it, so there is no
port to choose or collide on. It runs with a VF_EXEC_ID marker in its
environment, which every process its tools start inherits through detaching
and reparenting; close() kills the processes carrying it as the worker's own
user, so the cleanup reaches nothing that user could not signal itself. A
process that empties its environment escapes it; the workspace VM still ends
with the rollout.

Pin nano-rlm to the head that announces its bound port.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/deployment.py Resolved
The head stops native shell commands with their turn and session, and names
the worker's history in compaction notices under split execution.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved
hallerite and others added 3 commits October 31, 2026 01:38
A worker that crashes right after announcing its port leaves both its listen
line and the exit marker in the log by the first read; serve returned the
port. Check for the exit marker first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The head syncs the session ledger to the execution worker incrementally, so
long split sessions no longer outgrow one 16 MiB frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 31, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 31, 2026
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Oct 31, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/deployment.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

backend.prepare_execution(own(name, backend))
for name, backend in d.target_backends.items()
),
*([self._harness.prepare_execution(routes)] if self._harness else []),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Harness VM stays up during evidence collection

Medium Severity

stop_execution still only stops the workspace, so a split harness remains on the LAN while service evidence is collected. That VM keeps its harness → tool service rules, so leftover harness processes can still reach those services after the agent session has closed.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2e25eb1. Configure here.

Closing the execution connection killed every process that inherited the
worker's marker, including daemons the agent started, while a colocated
harness's exit leaves them running. The harness closes before grading when the
agent did not finish, so a daemon the task needs was gone only in split mode.

nano-rlm stops what is still attached to a session as the session closes, so an
RLM worker's close now stops just the worker and its own process group. Codex's
exec-server keeps the full stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
) -> None:
# The same pinned nano-rlm runs the tools in the workspace; the agent loop and
# the model credentials stay on the harness's VM.
await self._install(workspace)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Critical rlm/harness.py:200

setup_execution() passes self.config.resolved_env into the install process on the task VM, exposing forwarded harness credentials such as SERPER_API_KEY to commands there. Use an allowlisted environment for the workspace installation so harness secrets stay on the harness VM.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harnesses/rlm/harness.py around line 200:

`setup_execution()` passes `self.config.resolved_env` into the install process on the task VM, exposing forwarded harness credentials such as `SERPER_API_KEY` to commands there. Use an allowlisted environment for the workspace installation so harness secrets stay on the harness VM.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 038e388. Configure here.

asyncio.create_task(
_keepalive(workspace.bind(), max(10.0, idle_timeout / 3))
),
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Workspace keepalive starts after sibling boot

Medium Severity

The workspace keepalive in split mode starts only after the harness and every member VM finish booting, even though the workspace's idle clock starts as soon as its own start() returns. Sibling image pulls or first-use builds can outlast idle_timeout, so the platform can reclaim the workspace before setup runs. Member VMs already start their keepalive as soon as each one is up.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 038e388. Configure here.

nano-rlm#218 runs the supervisor's shell jobs, services and path watches in the
workspace, so split execution refuses none of them and keeps the runtime prompt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants