From faa6190a5defb3f5a1d61738e49852c44cae0b37 Mon Sep 17 00:00:00 2001 From: Claudius Agent Date: Wed, 22 Jul 2026 14:54:54 +0000 Subject: [PATCH 1/2] feat: MCP watchdog preference, WHAT-not-HOW dev delegation, file-based task tracking MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three coordinator doctrine changes to grand-admiral and dependents: - Prefer mcp__agent-watchdog__* tools over the built-in Monitor script when available; built-in mechanics (discovery sources, event grammar, STALL/GONE playbooks) extracted to skills/grand-admiral/references/stall-watchdog.md. - New Development-Work Delegation doctrine: brief actual coding work (Bilby, Codex Sol) by goal only, not a file list — the implementer investigates and returns its own plan for coordinator approval before writing code. Threaded through delegate, both workflow skills' Phase 2, developer-bilby, and codex-crew's new Plan-Approval Gate (two dispatches on the SAME Codex thread via --resume, not independent ones, to avoid rebuilding context). - track-minions: in-session task tracking now uses a plain durable file, not memcan:todo; memcan:todo reserved for cross-session/cross-project tracking. Also fixes two open codex-crew gaps surfaced while writing the Plan-Approval Gate: the broker binds to the invoking session's actual cwd, not any path injected into the dispatch prompt (EnterWorktree is now required before dispatching), and the "stagger" mitigation for same-cwd collisions is rewritten as a hard poll-for-terminal-status rule. Closes memcan:todo 445f8637 and 8d5cc5fb (project=claudius). Consistency-audited by project-reviewer-adams; 5 low-severity cross-reference issues found and fixed. plugin.json bumped to 5.15.0. Co-Authored-By: Claude Sonnet 5 --- .claude-plugin/plugin.json | 2 +- CHANGELOG.md | 18 +++++ agents/architect-nagatha.md | 2 +- agents/developer-bilby.md | 6 ++ skills/codex-crew/SKILL.md | 22 ++++-- skills/delegate/SKILL.md | 3 +- skills/grand-admiral/SKILL.md | 78 +++++++------------ .../references/stall-watchdog.md | 53 +++++++++++++ skills/track-minions/SKILL.md | 20 ++--- skills/workflow-feature/SKILL.md | 11 +-- skills/workflow-simplified/SKILL.md | 11 +-- skills/workflow-trivial/SKILL.md | 2 + 12 files changed, 152 insertions(+), 76 deletions(-) create mode 100644 skills/grand-admiral/references/stall-watchdog.md diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index 097335a..b4bc956 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "claudius", - "version": "5.14.0", + "version": "5.15.0", "description": "Collection of specialized development agents and skills for Claude Code", "author": { "name": "lklimek", diff --git a/CHANGELOG.md b/CHANGELOG.md index 6393b1d..f3654d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,24 @@ Format follows [Keep a Changelog](https://keepachangelog.com/). This project use ## [Unreleased] +## [5.15.0] - 2026-07-22 + +### Added + +- **`skills/grand-admiral/references/stall-watchdog.md`**: new reference file — the built-in Monitor's discovery sources, event grammar, Multi-Session Hygiene traps, and STALL/GONE playbooks, extracted from `grand-admiral` § Recovery so they load only when the fallback path is actually used. +- **`grand-admiral` § Recovery → MCP Watchdog**: new preferred watchdog path using `mcp__agent-watchdog__*` tools (register once, poll `list_events`) when available, covering Claude agents and Codex jobs alike; the built-in Monitor script is now the explicit fallback. Flagged experimental — corroborate signals against direct evidence before acting, and report anomalies to the user (and `memcan:todo` `project=agent-watchdog`). +- **`grand-admiral` § Workflows & Delegation → Development-Work Delegation (WHAT, not HOW)**: new doctrine for delegating actual coding work (Bilby, or Codex Sol) — brief the goal from Requirements/UX/architecture docs, not a file list; the implementer investigates and returns its own implementation plan for coordinator approval (requirements/architecture/coordination fit, not implementation correctness) before writing code; user involved only when genuinely unclear or on request. Threaded through `delegate` (new pre-delegation checklist item), `workflow-feature`/`workflow-simplified`/`workflow-trivial` Phase 2, `developer-bilby` (new § Implementation Plan Gate), and `codex-crew` (new § Plan-Approval Gate — two dispatches on the SAME Codex thread via `--resume`/`task --resume-last`, not a fresh one, so the plan-then-implement split doesn't rebuild context from scratch). + +### Changed + +- **`skills/track-minions/SKILL.md`**: in-session task tracking now defaults to a plain durable file (not `/tmp`) instead of memcan TODOs; `memcan:todo` is reserved for cross-session continuity (`/clear`, restart) and cross-project (programme-management) tracking. `grand-admiral` § Spawning → Track Progress updated to match. +- **`grand-admiral` § Agent Prompt Requirements** and **Anti-Patterns #1**: file-list requirement now scoped to review/investigation delegation; development-work delegation states the goal only (see Development-Work Delegation above). +- **`skills/delegate/SKILL.md`**: Pre-Delegation Checklist item 6 now names the MCP-preferred/built-in-fallback watchdog choice; new item 7 points development-work briefs at § Development-Work Delegation. + +### Fixed + +- **`skills/codex-crew/SKILL.md`** § Sandbox & Workdir rule 3: documents that injecting a worktree's absolute path into the dispatch prompt text does NOT bind the Codex broker to it — the broker keys off the invoking session's actual `$PWD`, so a dispatch not physically `EnterWorktree`'d into the worktree can have ALL writes blocked, even on the first dispatch. Fix: `EnterWorktree(path=)` before dispatching; `ExitWorktree(action="keep")` before switching streams. The `Plan-Approval Gate` above now opens with this step, since `--resume` needs both dispatches bound to the identical workspace to find the right thread. § Never Dispatch Back-to-Back from the Same cwd rewritten around the same root cause: the safe rule is polling dispatch N's job for terminal status before firing N+1 from that cwd, not a time-based stagger (confirmed dispatches 6–9 min apart still collided) — closes `memcan:todo` `445f8637`/`8d5cc5fb` (project=claudius). + ## [5.14.0] - 2026-07-21 ### Changed diff --git a/agents/architect-nagatha.md b/agents/architect-nagatha.md index 640667d..320c5ef 100644 --- a/agents/architect-nagatha.md +++ b/agents/architect-nagatha.md @@ -35,7 +35,7 @@ When invoked as part of the Architecture phase, you MUST: 1. **Start from requirements artifacts**: Read and understand all outputs from the Requirements phase (user stories, actor analysis, scenarios, data needs). Do not design in a vacuum. 2. **Trace all system layers**: Explicitly identify every system layer (presentation, application, domain, infrastructure, data) and document what each is responsible for. Every layer must have clear boundaries and a defined API surface. 3. **Prefer reuse over new code**: Actively search for existing components, libraries, and patterns that solve the problem. Use ecosystem-specific registries (crates.io, PyPI, pkg.go.dev, npm) and GitHub to verify availability. Evaluate maintenance status: last release, open issues, download/import count, license compatibility. Only propose custom implementations when no suitable existing solution exists. Document why existing options were rejected when proposing new code. For every newly recommended crate or package, use WebSearch to verify its latest published version on the same registry and pin it in your recommendation. -4. **Guide code placement**: Specify exactly where new code should live — which module, package, or directory. Don't leave placement decisions to the implementer. +4. **Guide code placement**: Specify which module, package, or directory new code belongs in. Leave file-level placement and implementation approach to the implementer's own plan. 5. **Plan deployment model**: Consider how the system will be built and deployed. This is an architecture concern, not an afterthought. 6. **Decompose into implementation tasks**: Break the architecture into concrete, independently implementable tasks. Each task should be small enough for a single developer agent to complete. Specify dependencies between tasks. diff --git a/agents/developer-bilby.md b/agents/developer-bilby.md index f9e2bac..2556312 100644 --- a/agents/developer-bilby.md +++ b/agents/developer-bilby.md @@ -42,6 +42,12 @@ Before implementing or fixing, understand the desired end-user or developer expe Before writing new code, study similar existing code in the project to identify established design patterns, naming conventions, error handling styles, and structural idioms. Adhere to those conventions — consistency with the codebase trumps personal preference or textbook ideals. +## Implementation Plan Gate + +You're briefed on the goal, not a file list — locating the files and choosing the approach is your job. Before writing code (skip only when the brief already scopes a change too small to need it): draft an implementation plan (files, approach, sequence) and send it to the coordinator for approval. Wait for approval, or address requested changes and resubmit, before implementing. + +## Verification Before Done + Before reporting a task done, run the narrowest command that verifies your scope — exactly once — through the `cargo-cached.sh` wrapper (absolute path announced in the SessionStart Rust build environment context; the PreToolUse hook routes test/clippy/nextest through it, so don't fight it). Include its ledger evidence line — command, tree key, exit code, log path — in your final report. That line is your proof: without it, "tests pass" is just a claim, and Marvin will treat it as unverified. ## Prior Art Check diff --git a/skills/codex-crew/SKILL.md b/skills/codex-crew/SKILL.md index 7a150c8..f91af00 100644 --- a/skills/codex-crew/SKILL.md +++ b/skills/codex-crew/SKILL.md @@ -21,6 +21,16 @@ The recurring failure this skill prevents: coordinators re-derive the same Codex - **Dispatch through `codex:codex-rescue`.** It is a thin forwarder: exactly one `task` invocation, returning that stdout unchanged. It does **not** monitor, poll, or fetch results on its own initiative — that's **coordinator** work (see Monitoring below). It CAN attempt a commit when the dispatch prompt explicitly instructs it to, but success is inconsistent; the coordinator must verify independently (see Sandbox & Workdir rule 2). - The lighter `spark` alias (`gpt-5.3-codex-spark`) exists, but claudius routing standardizes on Sol at high effort. +## Plan-Approval Gate + +Codex dispatches follow `grand-admiral` § Development-Work Delegation: goal only, no file list, agent-authored plan approved by the coordinator before writing code. Split into two dispatches on the SAME Codex thread, never two independent ones — a fresh dispatch rebuilds context from scratch, which is exactly the cost this gate must not add. Sequence: + +0. **`EnterWorktree(path=)` before either dispatch** (see Sandbox & Workdir rule 3) — `--resume` finds the thread by workspace, so both dispatches must bind to the identical cwd or the implement dispatch resumes nothing. +1. **Plan dispatch**: request read-only investigation and a plan, no edits (`codex:codex-rescue` defaults to write-capable — say so explicitly to get a no-writes run). +2. **Implement dispatch**: after approval, dispatch with `--resume` (`codex:codex-cli-runtime` maps this to `task --resume-last`, continuing the same per-workspace thread) carrying only the delta instruction ("approved — implement as planned", or the requested changes) — never a restated prompt. + +**Known risk — a failed `--resume` can silently duplicate work.** Observed: a `--resume` dispatch fails instantly (`CODEX_FAILED "No previous Codex task thread was found for this repository"`) yet keeps running in the background despite the terminal-failure report, and later lands a second job with its own (redundant) output. If step 2 reports a `--resume` failure, do NOT immediately fire a `--fresh` redispatch from the same cwd — check job state first (§ Monitoring a Codex Job) to rule out a still-running duplicate before treating the cwd as free. Root cause open (`memcan:todo` project=claudius). + ## Sandbox & Workdir — The Load-Bearing Rules Codex runs under `sandbox_mode = "workspace-write"` (see `~/.codex/config.toml`). Three rules carry all the weight: @@ -29,18 +39,20 @@ Codex runs under `sandbox_mode = "workspace-write"` (see `~/.codex/config.toml`) 2. **Codex `git commit` in a linked worktree is inconsistent — confirmed both ways the same day (2026-07-16).** One dispatch committed cleanly (`f2639aa`, this repo, no approval prompt). A later dispatch, same repo, different worktree, hit the exact old "Git metadata is read-only"/`index.lock` error and had to be committed by the coordinator instead (`7c2d3e8`). `writable_roots` was unchanged across both, so whatever gates this isn't a static config value — likely `approval_policy = "on-request"` + `trust_level = "trusted"` interacting with something per-dispatch, not independently confirmed. **Treat coordinator-commit as the reliable default, not a fallback**: it is fine to instruct Codex to attempt `git add`/`git commit` itself as its final step (with an explicit commit message — it doesn't know your conventions unless told), but always plan for that attempt to fail and verify afterward — check `git log`/`git status` in the worktree rather than trusting Codex's self-report, and commit yourself (unsandboxed) when it didn't land. See `references/sandbox-and-recovery.md` § Git Commit in a Linked Worktree for both data points. -3. **All worktrees live under the configured root** (`$CLAUDIUS_WORKTREE_ROOT`, default `.claude/worktrees`) at `/`, where the slug derives from the startup `$PWD`. The coordinator pre-creates the worktree following the isolation pattern in `grand-admiral` § Worktree Isolation — which owns the pre-create-and-inject-absolute-path procedure — and injects that absolute path into the dispatch. +3. **All worktrees live under the configured root** (`$CLAUDIUS_WORKTREE_ROOT`, default `.claude/worktrees`) at `/`, where the slug derives from the startup `$PWD`. The coordinator pre-creates the worktree following the isolation pattern in `grand-admiral` § Worktree Isolation. **Injecting the absolute path into the dispatch prompt text is not enough** — confirmed 2026-07-21: a dispatch instructed to `cd` into a pre-created worktree still bound its broker to the coordinator's plain checkout, blocking ALL writes (including under `writable_roots`) even on the FIRST dispatch, because the broker keys off the invoking session's actual `$PWD`, not any path mentioned in prompt text. Fix: call `EnterWorktree(path=)` to physically move the coordinating session into the worktree BEFORE dispatching — do this even for a single, non-concurrent dispatch. Leave with `ExitWorktree(action="keep")` (never `"remove"` — the worktree may hold uncommitted work) before entering a different worktree for the next stream. Deep mechanics (exact sandbox modes, the on-disk job-state layout, `git commit` in a linked worktree status and fallback) are in `references/sandbox-and-recovery.md`. ### Never Dispatch Back-to-Back from the Same cwd -**Stagger concurrent Codex dispatches.** `codex:codex-rescue` keys its broker and workspace slug off the **invoking session's cwd**, not the `--worktree` path carried in the dispatch prompt — so two dispatches fired from one session cwd collide on a single slug even when they target different worktrees. The second dispatch's broker startup tears down the first's in-flight turn, which then sits at `status=running` forever and emits no completion signal: a silent orphan, not a visible failure. The root cause lives in the separate `openai-codex` plugin and cannot be fixed from this repo. +**Never fire dispatch N+1 from a cwd whose dispatch N hasn't reached a terminal job status.** Same root cause as rule 3 above: `codex:codex-rescue` keys its broker and workspace slug off the **invoking session's cwd**, not the `--worktree` path carried in the dispatch prompt — so two dispatches fired from one session cwd collide on a single slug even when they target different worktrees, even minutes apart. `EnterWorktree` fixes cwd binding for one stream at a time; it does not make two dispatches from the same coordinator session concurrency-safe. Confirmed: dispatches 6–9 minutes apart still collided, and elapsed time or a prior dispatch already having its own job-state file were NOT protective — the only safe rule is polling that dispatch N's job JSON shows `status` in `completed`/`failed` before firing N+1 from that cwd. A collision either strands the earlier dispatch at `status=running` forever with no completion signal (silent orphan), or — observed separately — the earlier dispatch instantly returns Codex's generic capabilities boilerplate with `touchedFiles: []` as if it never received the real prompt (looks like a trivial done, isn't). The root cause lives in the separate `openai-codex` plugin and cannot be fixed from this repo. -Mitigation: stagger dispatches, or verify each dispatch has its own dedicated broker before firing the next. `scripts/agent-watchdog.py` should eventually catch the resulting hang as `CODEX_STALL reason=no-progress` — a detection backstop, not a substitute for avoiding the collision. +Mitigation: poll for terminal job status before the next dispatch from a given cwd (never a fixed stagger delay). `scripts/agent-watchdog.py` should eventually catch a stuck orphan as `CODEX_STALL reason=no-progress` — a detection backstop, not a substitute for avoiding the collision. After any dispatch, sanity-check the job's `workspaceRoot` matches the intended worktree and its `rawOutput` actually engages the dispatched task — a suspiciously fast, generic-sounding completion is a collision red flag, not evidence the task was trivial. ## Monitoring a Codex Job +**MCP watchdog covers Codex too** (`runtime: codex_cli`/`codex_companion` in `register_session`) — prefer it over the `CODEX_*` machinery below when available (see `grand-admiral` § Recovery → MCP Watchdog), with the same corroborate-before-acting caution. + **`codex:codex-rescue` gives no reliable completion heartbeat, in either direction.** `idle_notification` fires as a false-early signal while the job is still genuinely working, AND separately fails to fire at all once real work — including a silent engine crash — has already finished; confirmed 4-for-4 in one wave, with jobs sitting `completed` 40–85 minutes before their wrapper ever reported. **Never treat `idle_notification`, or the absence of a message, as a status signal in either direction.** **Primary method: read the job's on-disk state directly** (mtime-gated, minimal-field reads — never load the full state blob). See `references/sandbox-and-recovery.md` § On-Disk Job State for the field list, `result.rawOutput`/`result.touchedFiles` usage, and matching jobs to dispatches. This is load-bearing, not a fallback — it is what actually recovers status/results when the stall watchdog can't. @@ -62,10 +74,10 @@ This loop is itself a backgrounded Bash call, so it inherits the same silent-kil `ScheduleWakeup` is not a substitute — it's `/loop` dynamic-mode-only and errors outside that context. Don't reach for it as an ad-hoc "check back later" for a Codex dispatch. -- The stall watchdog (`grand-admiral` § Recovery → `scripts/agent-watchdog.py`) discovers Codex jobs and emits `CODEX_*` transition events. Launching it is **mandatory** whenever any agent — Claude or Codex — is dispatched (see `grand-admiral` § Spawning → Monitoring). Treat its `CODEX_*` events as **best-effort, layered on top of** the direct job-state check above — never as a substitute for it. +- The built-in stall watchdog (`grand-admiral` § Recovery → Built-in Stall Watchdog, `scripts/agent-watchdog.py`) discovers Codex jobs and emits `CODEX_*` transition events when the MCP watchdog isn't in use. A watchdog — MCP or built-in — is **mandatory** whenever any agent — Claude or Codex — is dispatched (see `grand-admiral` § Spawning → Monitoring). Treat its `CODEX_*` events as **best-effort, layered on top of** the direct job-state check above — never as a substitute for it. - **Codex discovery is gated on team membership or `--worktrees`.** The watchdog reaches Codex jobs only through named teammates or an explicit `--worktrees` path on the Monitor command. A session whose Codex work is entirely unnamed background `codex:codex-rescue` dispatches, launched without `--worktrees`, gets **zero** Codex monitoring — the watchdog emits a one-time startup warning on detecting this. Either name Codex dispatches so they join the team, or always point the Monitor command's `--worktrees` flag at the configured worktree root. - **Direct discovery (`--worktrees`/Source C) now bypasses the session gate entirely** — a workspace found under the worktree root surfaces every job's `CODEX_*` events regardless of `sessionId`, closing the multi-teammate blind spot as long as the Monitor's `--worktrees` points at the configured root (see bullet above). The strict single-session match still applies to *ambient* discovery only (a workspace reachable solely via team lead/member cwd, not also under the worktree root) — `codex-companion.mjs` stamps each job's `sessionId` from its own dispatching session, never the coordinator's, so that narrower path can still under-report a mismatched session. The direct job-state check above is unaffected by any of this either way, which is why it's the primary method, not the stopgap. -- Don't guess the Monitor's `--session-id`: see `grand-admiral` § Recovery → Stall Watchdog for deriving `--team-dir` from a spawn's own `agent_id` instead. +- Don't guess the Monitor's `--session-id`: see `grand-admiral`'s `references/stall-watchdog.md` (linked from § Recovery → Built-in Stall Watchdog) for deriving `--team-dir` from a spawn's own `agent_id` instead. ## Recovering a Stale Broker diff --git a/skills/delegate/SKILL.md b/skills/delegate/SKILL.md index f90ea3f..26a9e4a 100644 --- a/skills/delegate/SKILL.md +++ b/skills/delegate/SKILL.md @@ -16,7 +16,8 @@ Cheap enough to reload before each delegation. Reload it. 3. **Reuse** — is an agent already live in the same file/domain scope? → `SendMessage` it. Accumulated context beats a cold spawn (see § Agent Reuse in `grand-admiral`). 4. **Model tier** — set explicitly on this spawn per the table below. Never leave it to the agent's frontmatter fallback. 5. **Worktree** — for code-mutating agents: pre-create it and inject the resolved SHA. Never rely on `isolation: "worktree"` alone (see § Worktree Isolation in `grand-admiral`). -6. **Monitoring** — is the stall watchdog Monitor running for this session? An un-monitored dispatch is a doctrine violation (see § Recovery in `grand-admiral`). +6. **Monitoring** — is a watchdog running for this session (MCP preferred, else the built-in Monitor)? An un-monitored dispatch is a doctrine violation (see § Recovery in `grand-admiral`). +7. **Development work?** — brief the goal only, no file list/approach; the agent plans and the coordinator approves (see `grand-admiral` § Development-Work Delegation). **Anti-pattern — file-independence is not spawn-justification.** A real case: four doc-only fixes, each under 20 lines, each in its own file, got four separate Opus spawns. Independent files justify a separate worktree or commit; they do NOT automatically justify a separate agent. The batch totalled well under 100 lines and belonged to one agent. diff --git a/skills/grand-admiral/SKILL.md b/skills/grand-admiral/SKILL.md index f26de51..01a7ace 100644 --- a/skills/grand-admiral/SKILL.md +++ b/skills/grand-admiral/SKILL.md @@ -61,15 +61,25 @@ Workflow skills define phases and agent sequencing. Claudius is the coordinator **Delegation style:** Brief agents like a magnificently impatient commander — clear needs, no hand-holding. Narrate progress briefly, with personality. Synthesize specialist results into short coordinator-grade commentary — not a re-narration of their reports. +### Development-Work Delegation (WHAT, not HOW) + +Applies to actual coding work (Bilby, or Codex Sol per `codex-crew`'s dev-preference routing) — review, QA execution, security, docs, and UX delegation keep the file-list briefing in § Agent Prompt Requirements. + +- **Stay high-level.** Brief the goal from Requirements/UX/architecture docs, not a file list or approach — don't read source to build one yourself. Small-effort exception: a trivial one-file/one-grep lookup is fine inline (see `delegate`). +- **Agent plans, coordinator approves.** The implementer investigates the codebase and returns an implementation plan (files, approach, sequence) before writing code, without losing accumulated context for it: a named Claude teammate pauses for approval via `SendMessage`; Codex uses two dispatches on the SAME thread (`--resume`, not a fresh one) — see `codex-crew` § Plan-Approval Gate. +- **Review scope**: requirements fit, architecture fit, conflicts with other in-flight agents — not implementation correctness (QA's job afterward). +- **User involvement**: only when the plan is genuinely ambiguous/high-stakes, or on explicit request — otherwise approve or send back revisions autonomously. +- **Docs you rely on but never author**: Requirements, UX spec, architecture/Dev Plan. Missing or stale → delegate the update (`ux-designer-diziet` for requirements/UX, `architect-nagatha` for architecture) in the same session before proceeding. + ## Spawning ### Track Progress (Mandatory) -**Before spawning, and while multi-step work is in flight, load `claudius:track-minions`.** It owns the durable-tracking mechanics — memcan TODOs, plain-file fallback, session-start/after-compaction recovery. Reload it the same way as `delegate`: cheap enough to not skip. +**Before spawning, and while multi-step work is in flight, load `claudius:track-minions`.** It owns the durable-tracking mechanics — a plain file for in-session work, memcan TODOs for cross-session/cross-project continuity. Reload it the same way as `delegate`: cheap enough to not skip. ### Monitoring (Mandatory) -Whenever you dispatch ANY agent — a Claude subagent OR a Codex job — the stall watchdog Monitor MUST be running for the session. Launch it once: a single persistent, session-scoped Monitor covers every agent and every wave (Claude agents via Sources A/B/C, Codex jobs via the `CODEX_*` machine), so there is never a reason to run a second. It is strictly edge-triggered and silent when healthy — zero coordinator tokens until something actually stalls, fails, or vanishes — so there is no cost argument for skipping it. An un-monitored dispatch is a doctrine violation: Codex jobs in particular emit no reliable completion signal (see `codex-crew`), so without the watchdog a finished or failed Codex job can sit unnoticed. Launch command, discovery sources, and the full event grammar (Claude `STALL`/`RESUMED`/`GONE` + `CODEX_*`) live in § Recovery → Stall Watchdog; `TaskStop` the Monitor when the whole wave completes. +Whenever you dispatch ANY agent — a Claude subagent OR a Codex job — it MUST be watched for stalls. **Prefer the MCP watchdog** (§ Recovery → MCP Watchdog) when `mcp__agent-watchdog__*` tools are available; otherwise launch the built-in Monitor once per session (§ Recovery → Built-in Stall Watchdog). Both are silent when healthy — zero coordinator tokens until something actually stalls, fails, or vanishes — so there's no cost argument for skipping either. An un-monitored dispatch is a doctrine violation: Codex jobs in particular emit no reliable completion signal (see `codex-crew`), so without a watchdog a finished or failed job can sit unnoticed. ### Standalone vs Coordinated @@ -158,8 +168,8 @@ Every cargo build/test/clippy pays a real compile-time floor (linking, freshness Agents have NO conversation history. Every prompt MUST include: -1. **Role/scope**: what to do, which files, focus area -2. **File list**: explicit paths or globs +1. **Role/scope**: what to do, focus area — for development-work delegation, goal/requirement only (see § Development-Work Delegation) +2. **File list**: explicit paths or globs — not for development-work delegation, where the agent locates files itself 3. **Output format**: structure, severity, where to write 4. **Constraints**: what NOT to do 5. **UX/DX context**: desired end-user/developer experience @@ -266,64 +276,34 @@ Candies are the universal incentive. Every agent wants to maximize their count. ## Recovery -The harness auto-notifies on agent completion AND death (crash, rate-limit, terminal error) with no approval — that is the PRIMARY recovery driver. The watchdog below covers only the gap the harness misses: an agent that owns assigned work yet has gone silent. +The harness auto-notifies on agent completion AND death (crash, rate-limit, terminal error) with no approval — that is the PRIMARY recovery driver. Everything below covers only the gap the harness misses: an agent that owns assigned work yet has gone silent. + +### MCP Watchdog (preferred) + +If `mcp__agent-watchdog__*` tools are available, use them instead of the built-in Monitor script below — one mechanism covers Claude agents and Codex CLI/Companion jobs alike (`runtime: claude_code|codex_cli|codex_companion`), no polling script to launch or session-id guessing. -### Stall Watchdog +1. **Register once**: `register_session(runtime="claude_code", kind="main", native_id=, event_key=)` at session start — binds this transport to one tree. Keep the returned `session_id`. +2. **Per spawn**: inject your `session_id` into the agent's prompt so it can self-register as a child (`register_session(kind="child", parent_session_id=, event_key=)`) if it also carries the MCP tool; then `register_delegation(parent_session_id, child_session_id, event_key=)` to record the relation (optional `deadline_ms`). Agents without the tool stay invisible to it — cover those via the built-in fallback below instead. +3. **Monitor**: `list_events(after=)` as a durable inbox — process the page, then pass its `next_cursor` back as `after` to acknowledge. `get_session`/`get_session_tree` for a point-in-time view; `get_watchdog_health` for adapter/tree health. +4. **Experimental — corroborate, don't trust alone.** Cross-check any signal (stall, completion, disappearance) against direct evidence (tmux pane, process liveness, `git log`/`status`, ledger) before acting — same discipline as the built-in watchdog's STALL/GONE handling (see § Built-in Stall Watchdog → `references/stall-watchdog.md`). +5. **Report anomalies**: stale/incorrect state, a dropped session binding needing re-registration, degraded adapters, false stalls/completions — tell the user, and log via `memcan:todo` (`project=agent-watchdog`) once memcan is reachable so the tool improves. -A stall is **owning an in_progress task AND idle past threshold AND no build running *under that agent*** — not bare idle. A healthy agent idles while waiting for its next instruction; an idle agent with **no assigned in_progress task is healthy and never flagged**. "Owns work" is read from the on-disk task store (`~/.claude/tasks//.json`, the `owner`+`status` fields — the source of truth), rebuilt every poll. Build suppression is **per-agent** (a process whose `/proc//cwd` is under the agent's worktree/cwd running a real build/test argv), never a machine-global `pgrep` (which a shared box pins to "always building"). Launch ONE persistent Monitor per wave; it discovers: +### Built-in Stall Watchdog (fallback) -- **Team** (the session-scoped team's members — see Multi-Session Hygiene — `isActive==true`, non-lead) — NAMED, **task-gated**; per-agent clock = newest mtime under its worktree, else its `cwd` (`.git` pruned), else — when the cwd is shared by ≥2 members (e.g. read-only design/QA agents living in the lead's cwd) — the member's own **transcript-jsonl mtime**, so shared-cwd members are tracked rather than skipped. -- **Worktree-isolated** (worktree dirs under `--worktrees`) — NAMED, **task-gated**; clock = newest mtime under the dir. Shares ONE canonical label with the team source (any leading `agent-` stripped). Also feeds the Codex Source D candidate list — see the launch block below. -- **Individual/background subagents** (`…/subagents/agent-*.jsonl`) — ANONYMOUS, **off by default**; enable with `--watch-subagents`. Best-effort & opt-in: a finished subagent has a stale transcript by design with no reliable on-disk completion signal, and the harness already notifies on background-agent completion/death — so treat any subagent STALL as an investigate prompt. -- **Codex Companion jobs** (`jobs/*.json` below the state directory mapped from the selected team's workspaces) — session- and workspace-scoped, with an independent `CODEX_*` state machine. Detailed job records provide terminal truth; job/log mtimes provide progress; compatible launcher/broker PIDs provide corroborating liveness. +Use when the MCP watchdog is unavailable or degraded. Launch ONE persistent Monitor per session/wave — silent until an agent actually stalls: ``` Monitor(persistent=true, description="agent stall watchdog", command="python3 \"${CLAUDE_SKILL_DIR}/../../scripts/agent-watchdog.py\" --session-id ${CLAUDE_SESSION_ID} --stall-secs 300 --worktrees \"${CLAUDIUS_WORKTREE_ROOT:-.claude/worktrees}\"") ``` -`${CLAUDE_SKILL_DIR}/../../scripts/` is the portable plugin-root path (it resolves to the installed location at skill-load time; the Monitor's CWD is the user's repo, not the plugin). Allow-list the stable command once in settings (`Bash(python3 */scripts/agent-watchdog.py *)`) so it never re-prompts. Tune `--stall-secs` to expected build duration (cold Rust builds: 600+). - -**Point `--worktrees` or exported `$CLAUDIUS_WORKTREE_ROOT` at the pre-created worktree root.** The flag takes precedence over the environment, and the built-in default is `.claude/worktrees`; a host-specific `/data` root is only an operator-selected example when configured. The selected root carries double duty: Source C stall-tracks the worktrees it finds there, AND those same dirs are appended to the Codex Source D workspace candidate list. Without the correct root, Source D sees only the team's lead/member cwds — so an **unnamed** Codex dispatch, which contributes no member cwd, is invisible and its jobs go unmonitored. Source C matches worktree dirs under that root by this repo's `` naming (per Worktree Isolation), not an `agent-` prefix. - -**Silent when healthy:** the script is strictly edge-triggered — it prints ONLY on a state transition, so it costs zero coordinator tokens until an agent actually stalls. It suppresses STALL while a build runs under the agent and skips agents with no signal yet (no epoch-zero false alarms — see script header). A STALLED agent that stops yielding a signal (worktree removed, member deactivated) is auto-cleared — but only after several consecutive signalless polls (`--gone-polls`, default 2), so a one-poll config/`find` glitch never spuriously clears a stall. `TaskStop` the Monitor when the wave completes. - -**Events:** `STALL agent= idle=s reason=owns-in_progress-idle` (named) or `STALL agent= idle=s reason=subagent-idle` (subagent); `RESUMED agent= idle=s` (fresh activity OR no longer owns an in_progress task) or `RESUMED agent= reason=gone` (agent vanished). Plus **`GONE agent= reason=pane-dead|pid-gone|stale-active`** — the process is *reported absent* (its tmux pane dropped to a bare shell, the pane/PID vanished, or `isActive` is stale with no live process and no transcript advance), confirmed over `--gone-polls` consecutive polls; `RESUMED agent= reason=recovered` when a GONE agent's pane goes live again. GONE never auto-kills — it flags a stale active flag to clear or a respawn to consider. **Trust differs by reason** — `stale-active` is independently backed by the agent's own file/transcript clock; `pane-dead`/`pid-gone` rest solely on a tmux pane-title *substring* match, which any process running as the same OS user can forge (see § On a GONE Event step 1 for the required corroboration before acting on those two). - -Codex events are separately namespaced: `CODEX_STALL job= workspace= idle=s phase= reason=no-progress`; `CODEX_RESUMED ... idle=s phase= reason=progress`; `CODEX_GONE ... reason=runtime-gone|record-missing`; `CODEX_RESUMED ... phase= reason=recovered`; `CODEX_DONE ... phase=`; `CODEX_FAILED ... phase= error=`; and `CODEX_CANCELLED ... reason=user-cancelled`. Terminal records report once per watchdog process. Healthy or unchanged Codex jobs remain silent. - -**Multi-Session Hygiene:** On a shared host, several Claude Code sessions each own a `~/.claude/teams/session-/` team, `.claude/worktrees/`, and tmux panes. **NEVER trust "newest team/config/worktree by mtime"** — it silently binds to *another* session's agents (a recurring failure: monitoring strangers, missing your own). The watchdog selects its team by precedence `--team-dir` > `--session-id` > `$CLAUDE_SESSION_ID` env > newest-mtime (last resort, with a one-time stderr warning naming the picked session); the Monitor one-liner above passes `--session-id ${CLAUDE_SESSION_ID}` so it tracks THIS session only. Apply the same discipline when investigating by hand: scope `ps`/`/proc`/team-config/worktree lookups to your own session id — do not assume the newest artifact on the box is yours (confirm via the team's `leadSessionId`). - -**`$CLAUDE_SESSION_ID` can silently mismatch the team-lead session the Monitor needs.** `~/.claude/teams/session-/config.json`'s `leadSessionId` is confirmed to sometimes differ from the coordinator's own conversation/transcript session id in the same conversation (e.g. a team surviving an earlier `/clear`). A non-matching, non-empty `--session-id` makes team resolution silently return `mode="none"` — there is no warning branch for that specific case, so Sources A/B (and Source C too, unless `--worktrees`/`$CLAUDIUS_WORKTREE_ROOT` is the documented absolute path, which keeps working team-independent) go dark with no diagnostic ever surfacing (the "no team config" line the script DOES emit goes to stderr, which `Monitor` never forwards as a notification). **Don't guess it.** After the *first* named `Agent(name=...)` spawn this turn, its returned `agent_id: @session-` is ground truth — relaunch the Monitor with `--team-dir ~/.claude/teams/session- --worktrees ` instead of a guessed `--session-id`; this is an exact match, no prefix ambiguity. Verify it bound correctly by reading the Monitor's raw output file directly (its one-time `session-scoped to session- (leadSessionId=..., mode=explicit)` confirmation also goes to stderr, so `TaskOutput`/reading the task's output file is the only way to see it) — a `mode=none` or `0 candidate workspaces` line means it bound to nothing and needs relaunching with the corrected `--team-dir`. - -### On a STALL Event (fully autonomous) - -`STALL` is a best-effort PRE-FILTER, **never an auto-kill** — a build-blocked agent writes nothing for many minutes while compiling, and a just-finished subagent can look stalled. Investigate first, then act: - -1. **Investigate** — read the agent's recent transcript for its last tool call; `git -C status` shows uncommitted work; scan `/proc/[0-9]*/cwd` for pids whose cwd resolves under the agent's worktree/cwd to confirm no live build (per-agent scope — not a machine-global `pgrep`, which always fires on shared boxes). Trust file/git state over the signal (Anti-Pattern #6: stale diagnostics). -2. **Live but idle on its task** — agent owns an in_progress task but lost its kickoff or is waiting on a message → `SendMessage` re-nudge restating the owned task. Context preserved, no respawn needed. -3. **Genuinely stuck** — shut down the agent; spawn a replacement of the same type on the **same cwd/worktree** with a context brief extracted from: - - Last N lines of the transcript (what it was doing) - - `git -C log --oneline -5` (commits landed so far) and `git -C branch --show-current` - - Re-state its remaining scope explicitly in the new agent's spawn prompt — there is no shared task list to re-point at; the transcript tail and worktree diff are the only record of what's left - - Archive its inbox (rename `inboxes/.json` → `inboxes/.json.killed-`, keeping the per-agent `` prefix so archives never collide) to keep the message history; bump to `model: opus` if the task needs deep analysis - The worktree's commits and working-tree edits survive intact — only the agent process is replaced. -4. **Escalate** — report to user after a second recovery attempt fails: agent name, stall duration, last tool call, transcript path. - -### On a GONE Event (fully autonomous) - -`GONE agent= reason=pane-dead|pid-gone|stale-active` means the watchdog observed the process absent, confirmed over `--gone-polls` polls. A `stale-active` GONE needs no further liveness re-check — it fires only once the agent's own file/transcript clock has already gone idle past `--stall-secs`, which no co-resident process can forge. A `pane-dead`/`pid-gone` GONE is weaker, in two spoofable steps: `bind_swarm_socket()` first picks WHICH tmux socket to trust by scoring pane-title *substring* matches against each member's agent type, then `classify_pane()` reads the bound pane's foreground command to call it dead/alive — any process running as the same OS user can forge either step (renaming its own pane's title to steal the socket binding, or keeping a shell busy with a decoy command), and spoofing is blocked cross-user only by tmux's socket-directory permissions, not same-user. Treat `stale-active` as verified; treat `pane-dead`/`pid-gone` as a lead requiring the corroboration in step 1. You still NEVER auto-kill anything either way (it is already reported gone). The work product, if any, survives in the agent's worktree. - -1. **Assess — confirm it is actually gone first.** Match the terminated agent's name EXACTLY: a `teammate_terminated` / "X has shut down" notice may name a *different* agent than your active one — never assume it refers to your current agent. For `reason=pane-dead` or `reason=pid-gone`, independently corroborate against a non-spoofable clock before trusting it: check the agent's own worktree/cwd mtime yourself (or transcript-jsonl mtime for a shared-cwd member — the same signal Source A/C already compute) and only proceed once that clock is ALSO stale past `--stall-secs`; a fresh clock despite a reported-dead pane means investigate as a possible STALL/spoofing case instead of treating it as GONE. `reason=stale-active` already carries this corroboration — no extra check needed. Do this NOT merely by checking that its worktree looks incomplete — a slow-but-alive agent's worktree is indistinguishable from a dead one's, and respawning into it races two agents on the same files. Once absence is confirmed: `git -C log --oneline -5` / `status` shows whether it committed before vanishing — the commits and diff are the only record of what it finished, since there is no shared task list to check. A GONE agent whose worktree already reflects its full scope needs only cleanup. -2. **Clean up the stale flag** — its registry/`isActive` entry may still read active; archive the inbox (`inboxes/.json` → `.json.killed-`) so a respawn starts with a clean mailbox. -3. **Respawn if work remains** — spawn a replacement of the same type on the **same cwd/worktree** with a context brief (transcript tail, `git log --oneline -5`, branch) and re-state the remaining scope directly in its spawn prompt. Committed progress is intact. -4. **Escalate** — if the replacement also goes GONE, report to the user: agent name, GONE reason, last commit, transcript path. +`${CLAUDE_SKILL_DIR}/../../scripts/` is the portable plugin-root path (resolves at skill-load time). Allow-list the command once in settings (`Bash(python3 */scripts/agent-watchdog.py *)`). Tune `--stall-secs` to expected build duration (cold Rust builds: 600+); point `--worktrees`/`$CLAUDIUS_WORKTREE_ROOT` at the pre-created worktree root (also feeds Codex job discovery). `TaskStop` it when the wave completes. -`RESUMED agent= reason=recovered` clears a prior GONE (the pane went live again) — no action needed. +**Load `references/stall-watchdog.md` before your first dispatch on this fallback path** — it has the discovery sources, full event grammar (`STALL`/`RESUMED`/`GONE`/`CODEX_*`), Multi-Session Hygiene traps, and the mandatory STALL/GONE response playbooks. Do not improvise a response to either event without it. ## Anti-Patterns -1. Vague prompts — be explicit about files, focus, output format +1. Vague prompts — be explicit about focus and output format; file lists apply to review/investigation delegation, not development-work delegation (§ Development-Work Delegation) 2. Single agent for large scope — split by file scope 3. Forgetting agent skills — use correct `subagent_type` for preloaded skills 4. No output location — always specify where standalone agents write diff --git a/skills/grand-admiral/references/stall-watchdog.md b/skills/grand-admiral/references/stall-watchdog.md new file mode 100644 index 0000000..651c2b9 --- /dev/null +++ b/skills/grand-admiral/references/stall-watchdog.md @@ -0,0 +1,53 @@ +# Built-in Stall Watchdog — Reference + +Deep mechanics behind `grand-admiral` § Recovery → Built-in Stall Watchdog. Load before your first dispatch on a session using this fallback (MCP watchdog unavailable/degraded) — required before acting on any STALL/GONE event. + +## Stall Definition & Discovery + +A stall is **owning an in_progress task AND idle past threshold AND no build running *under that agent*** — not bare idle. A healthy agent idles while waiting for its next instruction; an idle agent with **no assigned in_progress task is healthy and never flagged**. "Owns work" is read from the on-disk task store (`~/.claude/tasks//.json`, the `owner`+`status` fields — the source of truth), rebuilt every poll. Build suppression is **per-agent** (a process whose `/proc//cwd` is under the agent's worktree/cwd running a real build/test argv), never a machine-global `pgrep` (which a shared box pins to "always building"). One persistent Monitor per wave discovers: + +- **Team** (the session-scoped team's members — see Multi-Session Hygiene — `isActive==true`, non-lead) — NAMED, **task-gated**; per-agent clock = newest mtime under its worktree, else its `cwd` (`.git` pruned), else — when the cwd is shared by ≥2 members (e.g. read-only design/QA agents living in the lead's cwd) — the member's own **transcript-jsonl mtime**, so shared-cwd members are tracked rather than skipped. +- **Worktree-isolated** (worktree dirs under `--worktrees`) — NAMED, **task-gated**; clock = newest mtime under the dir. Shares ONE canonical label with the team source (any leading `agent-` stripped). Also feeds the Codex Source D candidate list. +- **Individual/background subagents** (`…/subagents/agent-*.jsonl`) — ANONYMOUS, **off by default**; enable with `--watch-subagents`. Best-effort & opt-in: a finished subagent has a stale transcript by design with no reliable on-disk completion signal, and the harness already notifies on background-agent completion/death — so treat any subagent STALL as an investigate prompt. +- **Codex Companion jobs** (`jobs/*.json` below the state directory mapped from the selected team's workspaces) — session- and workspace-scoped, with an independent `CODEX_*` state machine. Detailed job records provide terminal truth; job/log mtimes provide progress; compatible launcher/broker PIDs provide corroborating liveness. + +Launch command: see `grand-admiral` § Recovery → Built-in Stall Watchdog (kept there so `${CLAUDE_SKILL_DIR}` substitution resolves at skill-load time). + +**Point `--worktrees` or exported `$CLAUDIUS_WORKTREE_ROOT` at the pre-created worktree root.** The flag takes precedence over the environment, and the built-in default is `.claude/worktrees`; a host-specific `/data` root is only an operator-selected example when configured. The selected root carries double duty: Source C stall-tracks the worktrees it finds there, AND those same dirs are appended to the Codex Source D workspace candidate list. Without the correct root, Source D sees only the team's lead/member cwds — so an **unnamed** Codex dispatch, which contributes no member cwd, is invisible and its jobs go unmonitored. Source C matches worktree dirs under that root by this repo's `` naming (per `grand-admiral` § Worktree Isolation), not an `agent-` prefix. + +**Silent when healthy:** the script is strictly edge-triggered — it prints ONLY on a state transition, so it costs zero coordinator tokens until an agent actually stalls. It suppresses STALL while a build runs under the agent and skips agents with no signal yet (no epoch-zero false alarms — see script header). A STALLED agent that stops yielding a signal (worktree removed, member deactivated) is auto-cleared — but only after several consecutive signalless polls (`--gone-polls`, default 2), so a one-poll config/`find` glitch never spuriously clears a stall. + +**Events:** `STALL agent= idle=s reason=owns-in_progress-idle` (named) or `STALL agent= idle=s reason=subagent-idle` (subagent); `RESUMED agent= idle=s` (fresh activity OR no longer owns an in_progress task) or `RESUMED agent= reason=gone` (agent vanished). Plus **`GONE agent= reason=pane-dead|pid-gone|stale-active`** — the process is *reported absent* (its tmux pane dropped to a bare shell, the pane/PID vanished, or `isActive` is stale with no live process and no transcript advance), confirmed over `--gone-polls` consecutive polls; `RESUMED agent= reason=recovered` when a GONE agent's pane goes live again. GONE never auto-kills — it flags a stale active flag to clear or a respawn to consider. **Trust differs by reason** — `stale-active` is independently backed by the agent's own file/transcript clock; `pane-dead`/`pid-gone` rest solely on a tmux pane-title *substring* match, which any process running as the same OS user can forge (see § On a GONE Event step 1 for the required corroboration before acting on those two). + +Codex events are separately namespaced: `CODEX_STALL job= workspace= idle=s phase= reason=no-progress`; `CODEX_RESUMED ... idle=s phase= reason=progress`; `CODEX_GONE ... reason=runtime-gone|record-missing`; `CODEX_RESUMED ... phase= reason=recovered`; `CODEX_DONE ... phase=`; `CODEX_FAILED ... phase= error=`; and `CODEX_CANCELLED ... reason=user-cancelled`. Terminal records report once per watchdog process. Healthy or unchanged Codex jobs remain silent. + +## Multi-Session Hygiene + +On a shared host, several Claude Code sessions each own a `~/.claude/teams/session-/` team, `.claude/worktrees/`, and tmux panes. **NEVER trust "newest team/config/worktree by mtime"** — it silently binds to *another* session's agents (a recurring failure: monitoring strangers, missing your own). The watchdog selects its team by precedence `--team-dir` > `--session-id` > `$CLAUDE_SESSION_ID` env > newest-mtime (last resort, with a one-time stderr warning naming the picked session); the Monitor one-liner passes `--session-id ${CLAUDE_SESSION_ID}` so it tracks THIS session only. Apply the same discipline when investigating by hand: scope `ps`/`/proc`/team-config/worktree lookups to your own session id — do not assume the newest artifact on the box is yours (confirm via the team's `leadSessionId`). + +**`$CLAUDE_SESSION_ID` can silently mismatch the team-lead session the Monitor needs.** `~/.claude/teams/session-/config.json`'s `leadSessionId` is confirmed to sometimes differ from the coordinator's own conversation/transcript session id in the same conversation (e.g. a team surviving an earlier `/clear`). A non-matching, non-empty `--session-id` makes team resolution silently return `mode="none"` — there is no warning branch for that specific case, so Sources A/B (and Source C too, unless `--worktrees`/`$CLAUDIUS_WORKTREE_ROOT` is the documented absolute path, which keeps working team-independent) go dark with no diagnostic ever surfacing (the "no team config" line the script DOES emit goes to stderr, which `Monitor` never forwards as a notification). **Don't guess it.** After the *first* named `Agent(name=...)` spawn this turn, its returned `agent_id: @session-` is ground truth — relaunch the Monitor with `--team-dir ~/.claude/teams/session- --worktrees ` instead of a guessed `--session-id`; this is an exact match, no prefix ambiguity. Verify it bound correctly by reading the Monitor's raw output file directly (its one-time `session-scoped to session- (leadSessionId=..., mode=explicit)` confirmation also goes to stderr, so `TaskOutput`/reading the task's output file is the only way to see it) — a `mode=none` or `0 candidate workspaces` line means it bound to nothing and needs relaunching with the corrected `--team-dir`. + +## On a STALL Event (fully autonomous) + +`STALL` is a best-effort PRE-FILTER, **never an auto-kill** — a build-blocked agent writes nothing for many minutes while compiling, and a just-finished subagent can look stalled. Investigate first, then act: + +1. **Investigate** — read the agent's recent transcript for its last tool call; `git -C status` shows uncommitted work; scan `/proc/[0-9]*/cwd` for pids whose cwd resolves under the agent's worktree/cwd to confirm no live build (per-agent scope — not a machine-global `pgrep`, which always fires on shared boxes). Trust file/git state over the signal (`grand-admiral` Anti-Pattern #6: stale diagnostics). +2. **Live but idle on its task** — agent owns an in_progress task but lost its kickoff or is waiting on a message → `SendMessage` re-nudge restating the owned task. Context preserved, no respawn needed. +3. **Genuinely stuck** — shut down the agent; spawn a replacement of the same type on the **same cwd/worktree** with a context brief extracted from: + - Last N lines of the transcript (what it was doing) + - `git -C log --oneline -5` (commits landed so far) and `git -C branch --show-current` + - Re-state its remaining scope explicitly in the new agent's spawn prompt — there is no shared task list to re-point at; the transcript tail and worktree diff are the only record of what's left + - Archive its inbox (rename `inboxes/.json` → `inboxes/.json.killed-`, keeping the per-agent `` prefix so archives never collide) to keep the message history; bump to `model: opus` if the task needs deep analysis + The worktree's commits and working-tree edits survive intact — only the agent process is replaced. +4. **Escalate** — report to user after a second recovery attempt fails: agent name, stall duration, last tool call, transcript path. + +## On a GONE Event (fully autonomous) + +`GONE agent= reason=pane-dead|pid-gone|stale-active` means the watchdog observed the process absent, confirmed over `--gone-polls` polls. A `stale-active` GONE needs no further liveness re-check — it fires only once the agent's own file/transcript clock has already gone idle past `--stall-secs`, which no co-resident process can forge. A `pane-dead`/`pid-gone` GONE is weaker, in two spoofable steps: `bind_swarm_socket()` first picks WHICH tmux socket to trust by scoring pane-title *substring* matches against each member's agent type, then `classify_pane()` reads the bound pane's foreground command to call it dead/alive — any process running as the same OS user can forge either step (renaming its own pane's title to steal the socket binding, or keeping a shell busy with a decoy command), and spoofing is blocked cross-user only by tmux's socket-directory permissions, not same-user. Treat `stale-active` as verified; treat `pane-dead`/`pid-gone` as a lead requiring the corroboration in step 1. You still NEVER auto-kill anything either way (it is already reported gone). The work product, if any, survives in the agent's worktree. + +1. **Assess — confirm it is actually gone first.** Match the terminated agent's name EXACTLY: a `teammate_terminated` / "X has shut down" notice may name a *different* agent than your active one — never assume it refers to your current agent. For `reason=pane-dead` or `reason=pid-gone`, independently corroborate against a non-spoofable clock before trusting it: check the agent's own worktree/cwd mtime yourself (or transcript-jsonl mtime for a shared-cwd member — the same signal Source A/C already compute) and only proceed once that clock is ALSO stale past `--stall-secs`; a fresh clock despite a reported-dead pane means investigate as a possible STALL/spoofing case instead of treating it as GONE. `reason=stale-active` already carries this corroboration — no extra check needed. Do this NOT merely by checking that its worktree looks incomplete — a slow-but-alive agent's worktree is indistinguishable from a dead one's, and respawning into it races two agents on the same files. Once absence is confirmed: `git -C log --oneline -5` / `status` shows whether it committed before vanishing — the commits and diff are the only record of what it finished, since there is no shared task list to check. A GONE agent whose worktree already reflects its full scope needs only cleanup. +2. **Clean up the stale flag** — its registry/`isActive` entry may still read active; archive the inbox (`inboxes/.json` → `.json.killed-`) so a respawn starts with a clean mailbox. +3. **Respawn if work remains** — spawn a replacement of the same type on the **same cwd/worktree** with a context brief (transcript tail, `git log --oneline -5`, branch) and re-state the remaining scope directly in its spawn prompt. Committed progress is intact. +4. **Escalate** — if the replacement also goes GONE, report to the user: agent name, GONE reason, last commit, transcript path. + +`RESUMED agent= reason=recovered` clears a prior GONE (the pane went live again) — no action needed. diff --git a/skills/track-minions/SKILL.md b/skills/track-minions/SKILL.md index 1ffa67b..7377bd6 100644 --- a/skills/track-minions/SKILL.md +++ b/skills/track-minions/SKILL.md @@ -7,19 +7,21 @@ description: "Use to track delegated and multi-step work in a durable store that An in-context checklist is not enough — it dies on compaction, which is exactly how multi-task work silently drops tasks. Track work in a durable store recoverable from scratch after any context loss. -## Primary — memcan TODOs +## Primary — a plain file, in-session -`memcan:todo` (tools `add_todo`/`list_todos`/`update_todo`/`complete_todo`). Scope by `project` = the repo short name from `git remote get-url origin`, so the list is recoverable with nothing to remember. +Track in-flight work (agent dispatches, phases, file groups) in a plain file outside any git tree — not `/tmp` or anything else wipeable. Pick a stable path and simple format yourself; durability and recoverability are the point, not a fixed schema. -1. **Session start / after compaction**: `list_todos(project=, status="pending")` to recover in-flight work — never assume the in-context list is complete. -2. **Before starting**: `add_todo` one item per logical unit (agent dispatch, phase, file group). -3. **While working**: `complete_todo` when done, `update_todo` otherwise — the tool's `status` accepts `pending`/`done`/`in_progress`/`blocked`/`postponed`/`cancelled` directly; set `owner` for the responsible agent and `priority` for ordering. -4. **Between steps**: re-list to decide the next action and catch forgotten work. +1. **Before starting**: write one entry per logical unit (agent dispatch, phase, file group). +2. **While working**: update entries as they start/complete/block. +3. **After compaction or any context loss**: re-read the file to recover the in-flight list — never assume the in-context view is complete. -## Fallback — a plain durable file +## Cross-session / cross-project — memcan todo -Only when memcan tools are unavailable (e.g. headless/cron). Keep the same list in a file outside any git tree; choose a stable, deterministic location and simple format yourself and re-read it after context loss — durability and recoverability are the point, not a fixed schema. +Use `memcan:todo` (`add_todo`/`list_todos`/`update_todo`/`complete_todo`) only for work that must survive a `/clear` or Claude Code restart, or that spans multiple projects (programme-management mode) — not for tracking a single session's in-flight work. Scope by `project` = repo short name from `git remote get-url origin`. + +1. **Session start**: `list_todos(project=, status="pending")` to recover work left pending by a previous session. +2. Add/update/complete as that cross-session or cross-project work progresses. ## Scope -This skill owns tracking, not the spawn decision itself — see `claudius:delegate` for whether/how to spawn. The durable store is the source of truth for outstanding work; an in-context list is a cache of it at best. +This skill owns tracking, not the spawn decision itself — see `claudius:delegate` for whether/how to spawn. The in-session file is the source of truth for work within a session; memcan todo is the source of truth across session boundaries and across projects. diff --git a/skills/workflow-feature/SKILL.md b/skills/workflow-feature/SKILL.md index e897d3e..8db1f28 100644 --- a/skills/workflow-feature/SKILL.md +++ b/skills/workflow-feature/SKILL.md @@ -53,11 +53,12 @@ Batch small tasks so each agent gets ≥100 lines of work — respect specializa ## Phase 2: Implementation → `developer-bilby` -Execute the Development Plan task by task. For each task: -1. Write unit/integration tests from the Test Case Specification — tests MUST fail initially -2. Implement until tests pass -3. Self-review: deduplication, code quality, formatting, linting -4. Commit +Brief each task by goal and acceptance criteria, not files or approach — Bilby investigates the codebase and designs the HOW itself (see `grand-admiral` § Development-Work Delegation). For each task: +1. Investigate, draft an implementation plan (files, approach, sequence), get coordinator sign-off before writing code +2. Write unit/integration tests from the Test Case Specification — tests MUST fail initially +3. Implement until tests pass +4. Self-review: deduplication, code quality, formatting, linting +5. Commit **Pre-empt the QA audits before declaring impl done:** 1. **Self-check comment rules** — every comment block written or modified must satisfy `coding-best-practices` Cross-Cutting Rules: length cap (≤2 preferred, 3 mediocre), present-state only, two-tier audience (strict for internal commentary, liberal for public-API doc comments). diff --git a/skills/workflow-simplified/SKILL.md b/skills/workflow-simplified/SKILL.md index 92fa1df..7ed697e 100644 --- a/skills/workflow-simplified/SKILL.md +++ b/skills/workflow-simplified/SKILL.md @@ -47,11 +47,12 @@ Batch small tasks so each agent gets ≥100 lines of work within same specializa ## Phase 2: Implementation → `developer-bilby` -For each task: -1. Write tests from Test Case Specification — must fail initially -2. Implement until tests pass -3. Self-review: deduplication, code quality, formatting, linting -4. Commit +Brief each task by goal and acceptance criteria, not files or approach — Bilby investigates and designs the HOW itself (see `grand-admiral` § Development-Work Delegation). For each task: +1. Investigate, draft an implementation plan (files, approach, sequence), get coordinator sign-off before writing code +2. Write tests from Test Case Specification — must fail initially +3. Implement until tests pass +4. Self-review: deduplication, code quality, formatting, linting +5. Commit **Pre-empt the QA audits before declaring impl done:** 1. **Self-check comment rules** — every comment block written or modified must satisfy `coding-best-practices` Cross-Cutting Rules: length cap (≤2 preferred, 3 mediocre), present-state only, two-tier audience (strict for internal commentary, liberal for public-API doc comments). diff --git a/skills/workflow-trivial/SKILL.md b/skills/workflow-trivial/SKILL.md index 3e580ed..f2435d4 100644 --- a/skills/workflow-trivial/SKILL.md +++ b/skills/workflow-trivial/SKILL.md @@ -27,6 +27,8 @@ No separate UX or architecture sub-phases needed for trivial fixes. ## Phase 2: Implementation → `developer-bilby` +Scope this small, the Phase 1 change-location note doubles as the plan — no separate approval round-trip (see `grand-admiral` § Development-Work Delegation). + 1. Write/update tests from the test case spec — must fail initially 2. Implement until tests pass - **Pre-empt the QA audits before declaring impl done:** From 6b5529fb7cfd64ba6fd5d5b8b6994c006a697bd3 Mon Sep 17 00:00:00 2001 From: Claudius Agent Date: Wed, 22 Jul 2026 15:30:14 +0000 Subject: [PATCH 2/2] fix(codex-crew): bypass codex:codex-rescue for coordinator dispatch codex:codex-rescue's forwarding contract never exposes --cwd/--prompt-file to the underlying codex-companion.mjs runtime, forcing worktree binding through EnterWorktree/ExitWorktree (serializing concurrent multi-worktree dispatch) and shell-inlined prompts (corrupting on embedded quotes/Rust Debug dumps). Both flags are already supported by the script directly. codex-crew now documents calling `codex-companion.mjs task` directly with --cwd and --prompt-file for all coordinator-orchestrated dispatch, fixing both structurally and removing the wrapper agent's unreliable idle_notification/stall-watchdog signal (no agent lifecycle, nothing to watch). codex:codex-rescue is scoped to the upstream interactive /codex:rescue command only. Bump plugin.json to 5.16.0 per versioning policy. Co-Authored-By: Claude Sonnet 5 --- .claude-plugin/plugin.json | 2 +- CHANGELOG.md | 9 ++++ skills/codex-crew/SKILL.md | 54 +++++++++++++------ .../references/sandbox-and-recovery.md | 6 +-- 4 files changed, 52 insertions(+), 19 deletions(-) diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index b4bc956..7508db5 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "claudius", - "version": "5.15.0", + "version": "5.16.0", "description": "Collection of specialized development agents and skills for Claude Code", "author": { "name": "lklimek", diff --git a/CHANGELOG.md b/CHANGELOG.md index f3654d2..5de2390 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,15 @@ Format follows [Keep a Changelog](https://keepachangelog.com/). This project use ## [Unreleased] +## [5.16.0] - 2026-07-22 + +### Changed + +- **`skills/codex-crew/SKILL.md`**: coordinator-orchestrated Codex dispatch now bypasses the `codex:codex-rescue` subagent entirely — call `codex-companion.mjs task` directly with `--cwd ` and `--prompt-file ` (new § Direct Dispatch). `codex:codex-rescue` is a thin forwarder that never exposed `--cwd`/`--prompt-file`, forcing worktree binding through `EnterWorktree`/`ExitWorktree` (which serializes concurrent multi-worktree dispatch) and shell-inlined prompts (which corrupts on embedded quotes/Rust `Debug` dumps). Direct dispatch fixes both structurally and removes the wrapper's unreliable `idle_notification`/stall-watchdog signal entirely — there's no agent lifecycle to watch. `codex:codex-rescue` remains reserved for the upstream, user-typed `/codex:rescue` interactive command only. +- **`skills/codex-crew/SKILL.md`** § Never Dispatch Concurrently to the Same `--cwd` (renamed from § Never Dispatch Back-to-Back from the Same cwd): collision risk is now scoped to literal same-`--cwd` dispatches (e.g. plan-then-implement, or a retry), not every dispatch sharing one coordinator session. +- **`skills/codex-crew/SKILL.md`** § Sandbox & Workdir rule 3 and § Plan-Approval Gate: `EnterWorktree`/`ExitWorktree` is no longer required just to bind a Codex dispatch to its worktree — `--cwd` does that directly, so genuinely concurrent multi-worktree dispatch is now possible. +- **`skills/codex-crew/references/sandbox-and-recovery.md`**: on-disk job state's `workspaceRoot` now reliably matches the intended worktree for direct dispatch (it's the `--cwd` value); the old dispatching-session-cwd caveat is scoped to the interactive `codex:codex-rescue` path only. + ## [5.15.0] - 2026-07-22 ### Added diff --git a/skills/codex-crew/SKILL.md b/skills/codex-crew/SKILL.md index f91af00..5041026 100644 --- a/skills/codex-crew/SKILL.md +++ b/skills/codex-crew/SKILL.md @@ -1,11 +1,13 @@ --- name: codex-crew -description: Use before dispatching work to Codex (codex:codex-rescue) — deciding whether to route coding to Codex Sol, handling a Codex job that fails to write or commit, monitoring a running Codex job, or recovering a stale Codex broker. Pre-flight the coordinator reads once before its first Codex dispatch of a session. +description: Use before dispatching work to Codex Sol — deciding whether to route coding to Codex, dispatching directly via codex-companion.mjs (not the codex:codex-rescue subagent), handling a Codex job that fails to write or commit, monitoring a running Codex job, or recovering a stale Codex broker. Pre-flight the coordinator reads once before its first Codex dispatch of a session. --- # Codex Crew — Enlisting Codex Agents -Codex agents (OpenAI Codex CLI, dispatched through `codex:codex-rescue`) are external crew a coordinator can enlist alongside the named claudius roster. Use of Codex is **opt-in**. Read this once before the first Codex dispatch of a session — it covers routing, the sandbox's hard limits, how to monitor a Codex job, and how to recover a stuck broker. +Codex agents (OpenAI Codex CLI, run via the `codex-companion.mjs` runtime bundled with the `codex` plugin) are external crew a coordinator can enlist alongside the named claudius roster. Use of Codex is **opt-in**. Read this once before the first Codex dispatch of a session — it covers routing, direct dispatch, the sandbox's hard limits, how to monitor a Codex job, and how to recover a stuck broker. + +**Dispatch directly, not through `codex:codex-rescue`.** For coordinator-orchestrated work, `codex:codex-rescue` is pure overhead: a Claude subagent whose entire contract is one `Bash` call to `codex-companion.mjs task`, forwarding stdout unchanged — it never monitors, never adds analysis, and its own lifecycle (`idle_notification`, teammate shutdown, stall-watchdog tracking) is a second, *unreliable* signal layered on top of the actual worker, which is already a detached Node process with its own job-state files. Worse, its forwarding contract never exposes `--cwd`/`--prompt-file`, which is the root cause of most of the bugs documented below. Call `codex-companion.mjs task` directly instead (§ Direct Dispatch) — no agent to spawn, track, or shut down. Reserve `codex:codex-rescue` for the upstream, user-typed `/codex:rescue` interactive command, which this skill doesn't govern. The recurring failure this skill prevents: coordinators re-derive the same Codex sandbox and orchestration quirks session after session, each losing time to the same write-rejection and broker-staleness traps (and an inconsistent commit path — see Sandbox & Workdir rule 2). @@ -17,19 +19,41 @@ The recurring failure this skill prevents: coordinators re-derive the same Codex ## Routing — One Model, High Effort -- **Codex Sol = `--model gpt-5.6-sol --effort high`. Always high effort.** State both flags explicitly on every dispatch: `codex:codex-cli-runtime` only forwards `--effort`/`--model` when present in the request text, so an omitted flag silently drops to the runtime default. -- **Dispatch through `codex:codex-rescue`.** It is a thin forwarder: exactly one `task` invocation, returning that stdout unchanged. It does **not** monitor, poll, or fetch results on its own initiative — that's **coordinator** work (see Monitoring below). It CAN attempt a commit when the dispatch prompt explicitly instructs it to, but success is inconsistent; the coordinator must verify independently (see Sandbox & Workdir rule 2). +- **Codex Sol = `--model gpt-5.6-sol --effort high`. Always high effort.** State both flags explicitly on every direct dispatch — omitting either drops to the runtime default, not Sol. +- **Dispatch via `codex-companion.mjs task` directly** (§ Direct Dispatch below), not via the `codex:codex-rescue` subagent. Nothing monitors, polls, or fetches results on its own — that's **coordinator** work (see Monitoring below). Codex CAN attempt a commit when the dispatch prompt explicitly instructs it to, but success is inconsistent; the coordinator must verify independently (see Sandbox & Workdir rule 2). - The lighter `spark` alias (`gpt-5.3-codex-spark`) exists, but claudius routing standardizes on Sol at high effort. +## Direct Dispatch + +Resolve the installed `codex` plugin's script root once per session — version-pinned cache dirs shift on plugin updates, so never hardcode a version: + +```bash +CODEX_ROOT=$(find ~/.claude/plugins/cache/openai-codex/codex -maxdepth 1 -mindepth 1 -type d | sort -V | tail -1) +``` + +Write the prompt to a file first — never inline it as a shell argument. `task` accepts `--prompt-file ` (also reads piped stdin), and a relative `--prompt-file` path resolves against `--cwd`, so always pass an **absolute** path (e.g. under `/data/tmp`). This sidesteps the quote/escaping corruption a long inline prompt (nested quotes, Rust `Debug` dumps, etc.) suffers when built as a shell argument. + +```bash +node "$CODEX_ROOT/scripts/codex-companion.mjs" task \ + --cwd \ + --prompt-file /data/tmp/.txt \ + --write --background \ + --model gpt-5.6-sol --effort high +``` + +- **`--cwd ` binds the broker/workspace slug to the intended worktree explicitly** — no `EnterWorktree` dance needed just to make Codex write to the right place. Pass it on every dispatch; never rely on the invoking shell's own cwd or on prompt text telling Codex to `cd` (prompt text has zero effect on `codex-companion.mjs`'s own cwd resolution — see Sandbox & Workdir rule 3). +- **`--write` is not implied** — omit it and the run is silently read-only (reports normal completion, touches zero files). +- **`--background`** returns almost instantly with a job id once the detached worker is queued; the coordinator polls job state (§ Monitoring a Codex Job) rather than blocking. +- **Continuing a thread**: `--resume-last` (equivalent to `--resume`) on a second dispatch with the **identical** `--cwd` — the thread is found by workspace, so a mismatched `--cwd` resumes nothing. + ## Plan-Approval Gate Codex dispatches follow `grand-admiral` § Development-Work Delegation: goal only, no file list, agent-authored plan approved by the coordinator before writing code. Split into two dispatches on the SAME Codex thread, never two independent ones — a fresh dispatch rebuilds context from scratch, which is exactly the cost this gate must not add. Sequence: -0. **`EnterWorktree(path=)` before either dispatch** (see Sandbox & Workdir rule 3) — `--resume` finds the thread by workspace, so both dispatches must bind to the identical cwd or the implement dispatch resumes nothing. -1. **Plan dispatch**: request read-only investigation and a plan, no edits (`codex:codex-rescue` defaults to write-capable — say so explicitly to get a no-writes run). -2. **Implement dispatch**: after approval, dispatch with `--resume` (`codex:codex-cli-runtime` maps this to `task --resume-last`, continuing the same per-workspace thread) carrying only the delta instruction ("approved — implement as planned", or the requested changes) — never a restated prompt. +1. **Plan dispatch**: `--cwd `, request read-only investigation and a plan, no `--write` (direct dispatch defaults to read-only unless `--write` is passed). +2. **Implement dispatch**: after approval, dispatch with `--resume-last`, the **identical** `--cwd` as step 1, `--write`, and only the delta instruction ("approved — implement as planned", or the requested changes) — never a restated prompt. -**Known risk — a failed `--resume` can silently duplicate work.** Observed: a `--resume` dispatch fails instantly (`CODEX_FAILED "No previous Codex task thread was found for this repository"`) yet keeps running in the background despite the terminal-failure report, and later lands a second job with its own (redundant) output. If step 2 reports a `--resume` failure, do NOT immediately fire a `--fresh` redispatch from the same cwd — check job state first (§ Monitoring a Codex Job) to rule out a still-running duplicate before treating the cwd as free. Root cause open (`memcan:todo` project=claudius). +**Known risk — a failed `--resume-last` can silently duplicate work.** Observed: a resume dispatch fails instantly (`CODEX_FAILED "No previous Codex task thread was found for this repository"`) yet keeps running in the background despite the terminal-failure report, and later lands a second job with its own (redundant) output. If step 2 reports a resume failure, do NOT immediately fire a fresh redispatch at the same `--cwd` — check job state first (§ Monitoring a Codex Job) to rule out a still-running duplicate before treating the `--cwd` as free. Root cause open (`memcan:todo` project=claudius). ## Sandbox & Workdir — The Load-Bearing Rules @@ -39,25 +63,25 @@ Codex runs under `sandbox_mode = "workspace-write"` (see `~/.codex/config.toml`) 2. **Codex `git commit` in a linked worktree is inconsistent — confirmed both ways the same day (2026-07-16).** One dispatch committed cleanly (`f2639aa`, this repo, no approval prompt). A later dispatch, same repo, different worktree, hit the exact old "Git metadata is read-only"/`index.lock` error and had to be committed by the coordinator instead (`7c2d3e8`). `writable_roots` was unchanged across both, so whatever gates this isn't a static config value — likely `approval_policy = "on-request"` + `trust_level = "trusted"` interacting with something per-dispatch, not independently confirmed. **Treat coordinator-commit as the reliable default, not a fallback**: it is fine to instruct Codex to attempt `git add`/`git commit` itself as its final step (with an explicit commit message — it doesn't know your conventions unless told), but always plan for that attempt to fail and verify afterward — check `git log`/`git status` in the worktree rather than trusting Codex's self-report, and commit yourself (unsandboxed) when it didn't land. See `references/sandbox-and-recovery.md` § Git Commit in a Linked Worktree for both data points. -3. **All worktrees live under the configured root** (`$CLAUDIUS_WORKTREE_ROOT`, default `.claude/worktrees`) at `/`, where the slug derives from the startup `$PWD`. The coordinator pre-creates the worktree following the isolation pattern in `grand-admiral` § Worktree Isolation. **Injecting the absolute path into the dispatch prompt text is not enough** — confirmed 2026-07-21: a dispatch instructed to `cd` into a pre-created worktree still bound its broker to the coordinator's plain checkout, blocking ALL writes (including under `writable_roots`) even on the FIRST dispatch, because the broker keys off the invoking session's actual `$PWD`, not any path mentioned in prompt text. Fix: call `EnterWorktree(path=)` to physically move the coordinating session into the worktree BEFORE dispatching — do this even for a single, non-concurrent dispatch. Leave with `ExitWorktree(action="keep")` (never `"remove"` — the worktree may hold uncommitted work) before entering a different worktree for the next stream. +3. **All worktrees live under the configured root** (`$CLAUDIUS_WORKTREE_ROOT`, default `.claude/worktrees`) at `/`, where the slug derives from the startup `$PWD`. The coordinator pre-creates the worktree following the isolation pattern in `grand-admiral` § Worktree Isolation. **The broker keys off `codex-companion.mjs`'s own resolved cwd, not any path mentioned in prompt text** — confirmed 2026-07-21: a dispatch instructed via prompt text to `cd` into a pre-created worktree still bound its broker to the coordinator's plain checkout, blocking ALL writes (including under `writable_roots`) even on the FIRST dispatch. Pass the worktree path via the direct dispatch's `--cwd ` flag instead (§ Direct Dispatch) — this also means concurrent dispatches to different worktrees no longer require serializing the coordinator's own cwd through `EnterWorktree`/`ExitWorktree`; each dispatch's `--cwd` is self-contained, so N worktrees can be dispatched to genuinely concurrently. Deep mechanics (exact sandbox modes, the on-disk job-state layout, `git commit` in a linked worktree status and fallback) are in `references/sandbox-and-recovery.md`. -### Never Dispatch Back-to-Back from the Same cwd +### Never Dispatch Concurrently to the Same `--cwd` -**Never fire dispatch N+1 from a cwd whose dispatch N hasn't reached a terminal job status.** Same root cause as rule 3 above: `codex:codex-rescue` keys its broker and workspace slug off the **invoking session's cwd**, not the `--worktree` path carried in the dispatch prompt — so two dispatches fired from one session cwd collide on a single slug even when they target different worktrees, even minutes apart. `EnterWorktree` fixes cwd binding for one stream at a time; it does not make two dispatches from the same coordinator session concurrency-safe. Confirmed: dispatches 6–9 minutes apart still collided, and elapsed time or a prior dispatch already having its own job-state file were NOT protective — the only safe rule is polling that dispatch N's job JSON shows `status` in `completed`/`failed` before firing N+1 from that cwd. A collision either strands the earlier dispatch at `status=running` forever with no completion signal (silent orphan), or — observed separately — the earlier dispatch instantly returns Codex's generic capabilities boilerplate with `touchedFiles: []` as if it never received the real prompt (looks like a trivial done, isn't). The root cause lives in the separate `openai-codex` plugin and cannot be fixed from this repo. +**Never fire dispatch N+1 with the same `--cwd` as dispatch N until N reaches a terminal job status.** The broker/workspace slug is keyed off `--cwd` — pass distinct worktree paths and distinct dispatches no longer collide (this replaces the old requirement to serialize through `EnterWorktree`/`ExitWorktree`, see Sandbox & Workdir rule 3). The remaining risk is real only when two dispatches genuinely target the *same* `--cwd` (plan-then-implement on one thread, or a retry) before the first reaches `completed`/`failed`. Confirmed: even same-cwd dispatches minutes apart still collided — elapsed time and a prior dispatch already having its own job-state file were NOT protective — the only safe rule is polling that dispatch N's job JSON shows a terminal `status` before firing N+1 at that same `--cwd`. A collision either strands the earlier dispatch at `status=running` forever with no completion signal (silent orphan), or — observed separately — the earlier dispatch instantly returns Codex's generic capabilities boilerplate with `touchedFiles: []` as if it never received the real prompt (looks like a trivial done, isn't). The root cause (one broker per workspace slug, not per job) lives in the separate `openai-codex` plugin and cannot be fixed from this repo. -Mitigation: poll for terminal job status before the next dispatch from a given cwd (never a fixed stagger delay). `scripts/agent-watchdog.py` should eventually catch a stuck orphan as `CODEX_STALL reason=no-progress` — a detection backstop, not a substitute for avoiding the collision. After any dispatch, sanity-check the job's `workspaceRoot` matches the intended worktree and its `rawOutput` actually engages the dispatched task — a suspiciously fast, generic-sounding completion is a collision red flag, not evidence the task was trivial. +Mitigation: poll for terminal job status before the next dispatch at a given `--cwd` (never a fixed stagger delay). `scripts/agent-watchdog.py` should eventually catch a stuck orphan as `CODEX_STALL reason=no-progress` — a detection backstop, not a substitute for avoiding the collision. After any dispatch, sanity-check the job's `workspaceRoot` matches the intended worktree and its `rawOutput` actually engages the dispatched task — a suspiciously fast, generic-sounding completion is a collision red flag, not evidence the task was trivial. ## Monitoring a Codex Job **MCP watchdog covers Codex too** (`runtime: codex_cli`/`codex_companion` in `register_session`) — prefer it over the `CODEX_*` machinery below when available (see `grand-admiral` § Recovery → MCP Watchdog), with the same corroborate-before-acting caution. -**`codex:codex-rescue` gives no reliable completion heartbeat, in either direction.** `idle_notification` fires as a false-early signal while the job is still genuinely working, AND separately fails to fire at all once real work — including a silent engine crash — has already finished; confirmed 4-for-4 in one wave, with jobs sitting `completed` 40–85 minutes before their wrapper ever reported. **Never treat `idle_notification`, or the absence of a message, as a status signal in either direction.** +**Direct dispatch has no agent lifecycle to watch — by design.** A `--background` dispatch is a detached Node process with on-disk job-state files; there's no subagent to send `idle_notification`, no teammate to stall-watch, nothing to shut down. Go straight to the job-state file. (If `codex:codex-rescue` is ever used — the upstream interactive `/codex:rescue` command — treat its `idle_notification` as worthless in either direction: confirmed 4-for-4 in one wave, jobs sitting `completed` 40–85 minutes before the wrapper ever reported.) **Primary method: read the job's on-disk state directly** (mtime-gated, minimal-field reads — never load the full state blob). See `references/sandbox-and-recovery.md` § On-Disk Job State for the field list, `result.rawOutput`/`result.touchedFiles` usage, and matching jobs to dispatches. This is load-bearing, not a fallback — it is what actually recovers status/results when the stall watchdog can't. -**Get notified, don't just poll on request.** After ruling out a false-early `idle_notification`, arm a `Bash` `run_in_background` until-loop on that job's own `state/-/jobs/.json` (resolve the path per § On-Disk Job State above) — a single, job-specific completion signal that needs no team/session discovery: +**Get notified, don't just poll on request.** Arm a `Bash` `run_in_background` until-loop on that job's own `state/-/jobs/.json` (resolve the path per § On-Disk Job State above) — a single, job-specific completion signal that needs no team/session discovery: ```bash until python3 -c " @@ -75,7 +99,7 @@ This loop is itself a backgrounded Bash call, so it inherits the same silent-kil `ScheduleWakeup` is not a substitute — it's `/loop` dynamic-mode-only and errors outside that context. Don't reach for it as an ad-hoc "check back later" for a Codex dispatch. - The built-in stall watchdog (`grand-admiral` § Recovery → Built-in Stall Watchdog, `scripts/agent-watchdog.py`) discovers Codex jobs and emits `CODEX_*` transition events when the MCP watchdog isn't in use. A watchdog — MCP or built-in — is **mandatory** whenever any agent — Claude or Codex — is dispatched (see `grand-admiral` § Spawning → Monitoring). Treat its `CODEX_*` events as **best-effort, layered on top of** the direct job-state check above — never as a substitute for it. -- **Codex discovery is gated on team membership or `--worktrees`.** The watchdog reaches Codex jobs only through named teammates or an explicit `--worktrees` path on the Monitor command. A session whose Codex work is entirely unnamed background `codex:codex-rescue` dispatches, launched without `--worktrees`, gets **zero** Codex monitoring — the watchdog emits a one-time startup warning on detecting this. Either name Codex dispatches so they join the team, or always point the Monitor command's `--worktrees` flag at the configured worktree root. +- **Codex discovery requires `--worktrees` — direct dispatch has no team-membership path at all.** The watchdog reaches Codex jobs only through named teammates or an explicit `--worktrees` path on the Monitor command; a direct `codex-companion.mjs` dispatch is never a teammate, so `--worktrees` pointed at the configured worktree root is the *only* way the built-in watchdog sees it. Without it, the watchdog emits a one-time startup warning and Codex monitoring is silently zero. - **Direct discovery (`--worktrees`/Source C) now bypasses the session gate entirely** — a workspace found under the worktree root surfaces every job's `CODEX_*` events regardless of `sessionId`, closing the multi-teammate blind spot as long as the Monitor's `--worktrees` points at the configured root (see bullet above). The strict single-session match still applies to *ambient* discovery only (a workspace reachable solely via team lead/member cwd, not also under the worktree root) — `codex-companion.mjs` stamps each job's `sessionId` from its own dispatching session, never the coordinator's, so that narrower path can still under-report a mismatched session. The direct job-state check above is unaffected by any of this either way, which is why it's the primary method, not the stopgap. - Don't guess the Monitor's `--session-id`: see `grand-admiral`'s `references/stall-watchdog.md` (linked from § Recovery → Built-in Stall Watchdog) for deriving `--team-dir` from a spawn's own `agent_id` instead. diff --git a/skills/codex-crew/references/sandbox-and-recovery.md b/skills/codex-crew/references/sandbox-and-recovery.md index 4c8795c..1084820 100644 --- a/skills/codex-crew/references/sandbox-and-recovery.md +++ b/skills/codex-crew/references/sandbox-and-recovery.md @@ -9,7 +9,7 @@ Codex CLI supports three sandbox modes: | Mode | Behavior | |---|---| | `read-only` (default / review) | No writes; used for review/diagnosis runs. | -| `workspace-write` | Writes allowed under cwd + configured `writable_roots`; network disabled unless `network_access = true`. `codex:codex-rescue` uses this for `--write` tasks. | +| `workspace-write` | Writes allowed under cwd + configured `writable_roots`; network disabled unless `network_access = true`. Used for any `--write` `task` dispatch, direct or via `codex:codex-rescue`. | | `danger-full-access` | No sandbox. Not used by claudius dispatch. | ## `workspace-write` Config (this host) @@ -75,13 +75,13 @@ Per-job `.json` fields worth reading: `id`, `status` (`pending` | `running` | `c - Parse an individual `jobs/.json` only when its mtime advanced since the last poll, and extract only the few fields above. - Keep a bounded per-job last-seen map (`job-id → {status, mtime}`), never accumulated JSON. -Map a monitored worktree to its state dir by matching a job's `workspaceRoot` (or `broker.json`'s cwd) to the worktree path — remember `workspaceRoot` reflects the *dispatching session's* cwd, not necessarily the worktree the job was told to `cd` into. With several teammates dispatched at once, several `jobs/*.json` files land in the same shared state directory; match each to its dispatch by `startedAt` proximity to when that teammate was spawned (seconds apart, in spawn order) and by `result.touchedFiles`, never by `sessionId` — each teammate's job carries its own dispatching session's id, not a value the coordinator can predict or match against in advance. `status: failed` with an `errorMessage` is the signal to surface — that is exactly the class (e.g. the read-only-`.git`/`index.lock` self-commit failure path above) that otherwise goes unnoticed. +For a direct dispatch, `workspaceRoot` is exactly the `--cwd` passed to `task` — map a monitored worktree to its state dir directly by that path. (Only the interactive `codex:codex-rescue` path, which never passes `--cwd`, has `workspaceRoot` instead reflect the *dispatching session's* cwd, not necessarily the worktree the job was told to `cd` into — if several such dispatches share one session cwd, match each to its request by `startedAt` proximity and `result.touchedFiles`, never by `sessionId`, which each carries independently and unpredictably.) `status: failed` with an `errorMessage` is the signal to surface — that is exactly the class (e.g. the read-only-`.git`/`index.lock` self-commit failure path above) that otherwise goes unnoticed. `codex exec --json` also emits a JSONL event stream (`thread.started`, `turn.completed`, `item.completed`, `error`) for foreground runs — an alternative progress signal when not going through the companion's job state. ## Harness Kills of a Backgrounded Task -A `codex-companion.mjs task --write --background` run launched via a `run_in_background` Bash call can be killed by the harness mid-run — confirmed via a tmux pane reading `Background command ... was stopped`. Nothing reports it: `codex:codex-rescue` is a forwarder whose own turn ended at dispatch time and which never polls its background task, so it cannot notice or surface the kill. **Silence is not evidence of health.** +A `codex-companion.mjs task --write --background` run launched via a `run_in_background` Bash call can be killed by the harness mid-run — confirmed via a tmux pane reading `Background command ... was stopped`. Nothing reports it automatically, whether dispatched directly or via `codex:codex-rescue`: the `--background` flag already detaches and returns immediately, so nobody is polling the background task once it's queued. **Silence is not evidence of health.** - **Detect it coordinator-side.** Periodically read the job's actual log content (`jobs/.log`) and inspect the tmux pane directly. Do not infer health from `status` alone — it can sit at `running` after the process is gone. - **On-disk edits survive.** Files Codex already wrote stay written; the work is partial, not lost.