Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/experimental/native-supervisor-board.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
151 changes: 151 additions & 0 deletions docs/experimental/native-supervisor-herdr.md

Large diffs are not rendered by default.

66 changes: 66 additions & 0 deletions docs/experimental/native-supervisor-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Native supervisor: ShuvBro parity

The native implementation covers the orchestration and runtime portion of the intended ShuvBro workflow through Shuvcode Sessions, a durable supervisor, and native tools. Full parity also requires first-class Herdr support for the lead, workers, and secondmates. The [three-repository contract and local evidence](native-supervisor-herdr.md) cover the new managed-home integration; companion review and two-host qualification remain open. Compatibility with other agent harnesses and multiplexers is outside this native path.

This is an experimental implementation with qualified local and isolated-box behavior. Herdr qualification, production migration, and live external integrations require the additional checks below.

The comparison is pinned to ShuvBro `b952627653ee2e0a9df5224b4678e7d2d7a5a70d` and Shuvcode base `28350a1f141e1fe7fef440d72ca791d31f595349`. Development is on `native-supervisor`. Verification below was completed October 5, 2026, PDT. No package has been published.

## Capability and evidence

| Capability | Implemented behavior | Evidence and limits |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Projects | Register, clone, create, configure, choose defaults, archive, and restore. Existing work retains its pinned project and policy. | Registry and CLI tests; a real lead registered a second project on the evaluation VM. |
| Backlog | Queue before creating Sessions or worktrees; dependencies, landing dependencies, holds, dates, priorities, and exclusive resources gate dispatch. Independent work has no default worker cap. | SQLite and runtime tests; live hold/release and dependency workflow across two projects. No fleet-scale throughput claim. |
| Authority | Capture delivery mode, merge authority, and worker permissions at intake. Lead operations inherit approved policy; only the operator can change permissions or increase authority. | Runtime regression tests cover prohibited escalation, permitted inheritance, and immutable captured policy. |
| Lifecycle | Separate interrupt, hold, cancel, resume, retry, completion, and cleanup. Retain previous attempts and adopt pending work on lead transfer without changing admission identities. | Native integration and crash tests, including a held item, answered decision, active execution, receipts, and stable message IDs across manager death. |
| Decisions | Routine questions may be answered by the lead; user-only decisions require operator provenance. Keyed answers cannot target a different execution. | Native permission and decision tests; the live board supplied the user-only answer for `inspect-input`. |
| Delivery | Support local-only landing, direct origin PRs, and bound no-mistakes runs. Verify source/target commits, forge identity, checks, and merge approval. Preserve uncertain outcomes for explicit reconciliation. | Disposable Git, forge-contract, and validation-contract tests; a real ship task landed by fast-forward on the VM. No live PR, release, or no-mistakes pipeline was created by this evaluation. |
| Reports and cleanup | Verify ship/scout evidence after native execution becomes idle; preserve accepted report bytes independently of worktree lifetime. Cleanup requires current clean, terminal evidence. | Local landing and cleanup; source and delegated scout report persistence; lost-reply and cancellation fixtures. |
| Fleet view | Status, Bearings, and the board share a bounded snapshot with explicit queued, held, running, waiting, unknown, ready, and landed states. | CLI/board tests and actual board use for the two-project workflow. |
| Supervision | Durable outbox/notices, per-task serialization, deduplication, native admission uncertainty, and managed epoch fencing. One slow admission cannot block unrelated work. | A 1,000-obligation persistence test; real delayed native HTTP admission; managed SIGKILL recovery. The burst is a durability test, not a model throughput benchmark. |
| Persistent delegates | Independent homes, leads, projects, backlog, Sessions, and workers. Dependency-closed handoffs preserve source ownership and accepted destination evidence. | Two-home fixtures and two completed real-model scout handoffs. Source work never acquired a local execution. |
| SSH placement | Typed SSH bridge to a destination home; unavailable placement remains unknown; retry reuses the exact handoff and destination work identities. | Actual SSH refusal and recovery through a dedicated loopback SSH daemon on the VM. This proves SSH transport and home separation, not two-host network behavior. |
| Relay and inbox | Durable untrusted intake, keyed acknowledgment/dismissal, origin/approval tracking, promised replies, outcome reconciliation, validated durable reply images, and bounded code-aware reply splitting. | Local adapter and restart tests. Credentials, public Relay posts, and live external delivery were not exercised. |
| Voice | Optional push-to-talk client/relay, PCM framing, credential renewal, durable interaction IDs, bounded disclosure, and native intake. | Offline Python regressions and installed asset resolution. AWS speech calls and audio devices remain unverified. |
| Away mode | Confirmed exact-word contracts, escalation classification, coverage gaps, deferred user input, and a return brief before ordinary intake resumes. | Real native prompt-hook tests across supervisor restart and fresh-lead replacement; the deferred input reaches the model exactly once after catch-up. |
| Knowledge | Private/shared/fleet startup context; on-demand project/task notes; evidence-based aging, perishable retention, and prior-value archives. Primary stow synchronizes shared records before destination curation; persistent UUIDs bind ownership. | Knowledge tests, native first-request proof, and compiled two-home SSH/model/restart proof. Temporary evaluation markers were archived after verification. |
| Startup budget | Operator-set persistent per-home limit; bounded remediation remains available while intake and dispatch wait. Curation or a budget change releases queued work on the next attempt. | Operator authorization, over-budget dispatch, curation, and recovery tests. Token count is an estimate from UTF-8 bytes, not provider accounting. |
| Installed CLI | Linux Bun package carries the supervisor plugin and voice scripts next to the executable; managed entrypoints resolve them without a checkout. | Compiled init/up/send/status/stop/start smoke, first model request, voice asset resolution, and offline npm pack/install smoke. The compiled distribution also runs both evaluation homes. |

## Completed verification

- `bun test test/supervisor-*.test.ts` from `packages/cli`: **169 passed, 0 failed**, 1,177 assertions across 27 files.
- Core Session execution/recovery suite: **37 passed**, 180 assertions, including cancelled roots and nested background children remaining stopped while unrelated claimed Sessions recover.
- Existing server process and CLI web UI tests: **4 passed**, 212 assertions.
- Canonical `bun run check`: **41/41 tasks passed**. Scoped formatting and `git diff --check` passed.
- The installed Linux CLI completed the first-request knowledge and managed restart smoke, with **17 assertions**, and the packed offline npm install smoke. No package was published.
- The actual ship task's five Bun tests passed independently on the VM after local landing.

The active-work crash test exposed that private stdio servers had never enabled native restart recovery. Managed startup now waits for the supervisor tool bridge before releasing native recovery. A second crash-window test exposed that a durable cancellation could resume one extra model request; recovery now terminalizes cancelled roots and their background descendants before resuming other claims. The public idle interruption behavior and ordinary managed-service recovery path remain unchanged.

## Evaluation box

The default VM entry is `ssh -t shuvcode-test.exe.xyz shuvcode supervisor`, using `/home/exedev/fleet-home` and optional background Herdr views. `ssh -t shuvcode-test.exe.xyz herdr` shows the fleet in Herdr. The October 6, 2026 PDT reset removed the earlier trial homes, Sessions, worktrees, wrappers, watcher, and test SSH route. It retained the operator's `2password` project and left a fresh firstmate with no messages or tasks, with all runtimes stopped. The [three-repository evaluation](native-supervisor-herdr.md#exedev-testing-october-6-2026-pdt) retains the historical observations and the separate ShuvBro publication status.

The retired parity evaluation used `/home/exedev/eval/parity`, with its delegate at `/home/exedev/eval/secondmate` and the original pilot at `/home/exedev/eval/pilot`.

The real model was `eval/gpt-6-sol` through exe.dev's configured ChatGPT integration. The two-project evaluation produced:

- `normalize-label`: ship receipt verified, five tests passed, and local fast-forward landing at `1830d6e2f8cbc5ada784190845c1c6f699e91c65`.
- `inspect-input`: waited for landing, received a keyed user-only answer through the board, and completed with its report preserved.
- `remote-readme` and `remote-recovery`: delegated through SSH, completed in the second home, and returned accepted report bytes. Stopping the test SSH route produced an unknown result without local fallback; retry after restoration reused the same destination work.

The first live workflow needed operator intervention while the native question tool and lead continuation were repaired. It is evidence of the repaired workflow, not a first-attempt autonomy score. Later fixture coverage protects the repaired question and continuation behavior.

Both homes were restarted onto the compiled distribution with the same leads, six total work items, decisions, and receipts preserved. Real model replies then confirmed each home could read its private marker and the primary's shared marker, without sharing the primary's private marker. Shared update history survived another delegate restart. The temporary active markers were retired through stow afterward.

The reset removed the runtime homes and their session data. Local test/build evidence is under `/home/shuv/.cache/agent-ws/native-supervisor-parity/` and the historical compiled artifact under `/home/shuv/.cache/agent-ws/native-supervisor-final-package/`.

## Remaining production qualification

Run an explicitly authorized live forge/no-mistakes workflow, configured Relay delivery, and actual speech/device checks before relying on those integrations. Qualify remote placement across two separate hosts. Measure idle CPU/RSS, multi-worker latency, task quality, intervention rate, and token cost against the same ShuvBro tasks and model before claiming better speed, cost, or production readiness.

Evidence freshness is enforced as a changed supplied evidence value; the supervisor does not independently verify whether the fact is true. Unreachable or over-budget delegates remain unresolved until curation, an operator budget change, or an explicit retry. Legacy shared caches without UUID ownership require reconciliation.

Managed mode fences the native process it owns before clearing uncertain admission. External-server mode cannot supply that ownership guarantee and retains uncertainty as a blocker. Production ShuvBro services were not changed.
Loading
Loading