fix: blame the runtime, not the harness, when the box dies mid-rollout

[mikasenghaas]

Summary

• Not every runtime raises when the box is gone — some surface it as exec's own non-zero result, indistinguishable from the harness program failing. A sandbox that died mid-rollout was therefore recorded as HarnessError ("harness exited 137"), blaming the agent program for an infrastructure death.
• Adds Runtime.alive(): one trivial probe exec (true) that tells the two apart, and uses it at the single shared blame site — Harness.run's non-zero-exit path. Probe fails → SandboxError ("runtime died under harness ..."); probe succeeds → the program genuinely failed and the existing HarnessError stands.

Fixes RES-1195.

Verification

Real eval rollouts (bash harness, sleeper task whose agent runs sleep 63), killing the box manually mid-rollout:

• docker (docker rm -f <container> mid-rollout)
  • before: trace.error = HarnessError: harness 'bash' exited 137
  • after: trace.error = SandboxError: runtime died under harness 'bash' (exit 137)
• subprocess (workdir removed + harness process group SIGKILLed)
  • before: trace.error = HarnessError: harness 'bash' exited -9
  • after: trace.error = SandboxError: runtime died under harness 'bash' (exit -9)

Controls: a program exiting non-zero in a live box still records HarnessError on both runtimes. uv run pytest tests/v1 -n auto -m "not e2e" passes.

🤖 Generated with Claude Code

---

Note

Low Risk
Narrow failure-path change with an extra probe exec; genuine program failures in a live runtime still raise HarnessError unchanged.

Overview
When a harness segment exits non-zero, the framework now probes the runtime with Runtime.alive() (true exec) before assigning blame. If the box no longer runs commands, the trace gets SandboxError (“runtime died under harness …”) instead of HarnessError (“harness exited …”), so infra kills (e.g. exit 137 / -9) are not attributed to the agent program.

alive() is a default on the Runtime base class (success on exit 0, False on exec failure). The only call site is the shared non-zero exit path in Harness.run.

^{Reviewed by Cursor Bugbot for commit de4ada9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Raise SandboxError instead of HarnessError when the runtime dies mid-rollout

• Adds Runtime.alive() in base.py, an async method that runs true and returns False if any exception occurs.
• When a program exits with a non-zero code, the rollout handler in harness.py now calls runtime.alive() to distinguish a dead sandbox from a program failure, raising SandboxError if the runtime is gone and HarnessError otherwise.
• Behavioral Change: callers that previously caught HarnessError for all non-zero exits must now also handle SandboxError when the runtime has died.

📊 Macroscope summarized de4ada9. 2 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit de4ada9. Configure here.}

[macroscopeapp]

Approvability

Verdict: Approved

This is a focused error-attribution fix that adds a runtime health check to distinguish between command failures and runtime deaths, raising the appropriate error type. The change is small, self-contained to diagnostic logic, and authored by the primary maintainer of these files.

^{You can customize Macroscope's approvability policy. Learn more.}