Autonomous Eval

[sethkarten]

Summary

Combines the open Seth Karten draft/review PRs needed for the Autonomous Eval stack into one integration branch:

• Add autonomous continuation mode #222 Add autonomous continuation mode
• Add user orchestrator for agent messaging and observation #207 Add user orchestrator for agent messaging and observation
• Add auto refine review hook #201 Add auto refine review hook
• Default thinking level to xhigh #266 Default thinking level to xhigh
• Defer heartbeat delivery while sessions are active #265 Defer heartbeat delivery while sessions are active

Also updates the single autonomous mode semantics for evaluator-controlled runs:

• autonomous mode does not trust the model's prose or a git diff as terminal evidence
• failed verifier/quality gates feed back into the agent so it keeps repairing while budget remains
• passing verifier/quality gates are the host/evaluator completion condition for autonomous evals
• repeated unchanged failing-gate feedback is compacted to avoid context blow-up

This is intended for EmulatorBench autonomous evals where the verifier/eval harness, not the model, decides when the run is complete.

Validation

• npm install
• npm --prefix packages/coding-agent test -- suite/agent-session-autonomous.test.ts args.test.ts print-mode.test.ts suite/agent-session-compaction.test.ts daemon-mode.test.ts model-resolver.test.ts refinement.test.ts agent-observe.test.ts agent-session-bus.test.ts agent-session-services.test.ts — 215 passed
• ./node_modules/.bin/tsgo --noEmit
• git diff --check
• pre-commit full check passed: biome check --write --error-on-warnings . && tsgo --noEmit && npm run check:installer && npm run check:browser-smoke

Notes

The EmulatorBench verifier hardening (fixed feedback mode, no agent --mode override) lives in the research-environments-private side; this PR covers Prime Agent.

---

Note

High Risk
Changes the core agent turn/stop and compaction continuation paths (now async) and executes configurable shell gate commands with process-tree kills on timeout—security- and stability-sensitive for eval harnesses.

Overview
Autonomous eval mode lets a host or harness keep the agent running without human input until configured quality gates pass or budget limits (continuations, turns, tokens, wall time) are hit. Completion is driven by gate commands and limits—not assistant prose or a bare git diff.

New --autonomous and related CLI flags wire into session runtime config. A new autonomous.ts module runs gate commands (with timeouts, output caps, and git worktree snapshots to skip reruns when the workspace is unchanged), tracks usage (cache-read tokens excluded from the token budget), and builds continuation user messages.

AgentSession merges autonomous config, exposes /autonomous on|off|status, chains autonomous continuations after goal continuations via getContinuationMessages, and supports internalPrompt / suppressAutonomousContinuation so host-driven gate follow-ups are not double-injected. Threshold compaction can queue autonomous follow-ups before stopping the tool loop; bookkeeping restores or clears those queues when compaction is skipped or autonomous mode is turned off. _shouldStopAfterTurn is now async.

Print mode acts as the host loop: after idle, it may feed gate failures back as internal follow-up prompts, refresh gates, and exit 1 if gates still fail or limits are reached.

Autonomous mode is forced off when rlmDepth > 0. Docs now list defaultThinkingLevel default as xhigh; agent-observe documents get_agent returning { agent }; orchestration heartbeat sanitizes label fields.

waitForChildProcess normalizes signal exits and handles already-exited children—used by gate subprocess handling.

^{Reviewed by Cursor Bugbot for commit 5ea8cdb. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add autonomous eval mode with quality gates and continuation limits to the coding agent

• Introduces a new autonomous mode (autonomous.ts) that injects follow-up user prompts when quality gates fail or terminal evidence is missing, with configurable limits on continuations, turns, tokens, and timeout.
• Adds CLI flags (--autonomous, --autonomous-gate, --autonomous-max-turns, etc.) parsed in args.ts and wired through to session config via main.ts.
• Extends AgentSession to track autonomous runtime state, handle /autonomous on|off|status slash commands, and suppress autonomous continuations for host-driven or internal prompts.
• Integrates autonomous gate evaluation into print mode (print-mode.ts), exiting with code 1 when gates still fail after retries or autonomous limits are reached.
• Autonomous mode is automatically disabled for recursive (subagent) sessions where rlmDepth > 0.
• Risk: _shouldStopAfterTurn is now async, and threshold compaction may queue autonomous continuations that alter post-compaction session flow.

Changes since #278 opened

• Refactored runChildProcess function to spawn child processes with detached=true on non-Windows platforms, track PIDs using trackDetachedChildPid and untrackDetachedChildPid, replace manual event handling with waitForChildProcess, and terminate entire process trees via killProcessTree on timeout instead of the previous SIGTERM-then-SIGKILL sequence [5ea8cdb]
• Added test helper utilities isProcessRunning and waitForProcessExit and implemented test case 'terminates the autonomous gate process tree when the timeout expires' to verify process tree termination behavior on timeout [5ea8cdb]

^{Macroscope summarized 1705450.}

[sethkarten]

@kevinjosethomas PR 278 is rebased on current main. CI, Macroscope, and Cursor Bugbot are green, and all review threads are resolved. Remaining blocker is the required approval review.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 3af140d. Configure here.}