fix(agent-watchdog): bound Codex job scan, cover untested branches; codex-crew commit-doctrine update

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "claudius",
-  "version": "5.9.0",
+  "version": "5.10.0",
   "description": "Collection of specialized development agents and skills for Claude Code",
   "author": {
     "name": "lklimek",

CHANGELOG.md

@@ -6,6 +6,25 @@ Format follows [Keep a Changelog](https://keepachangelog.com/). This project use
 
 ## [Unreleased]
 
+## [5.10.0] - 2026-07-16
+
+### Fixed
+
+- **`scripts/agent-watchdog.py`** `CodexScanner.scan()`: add a 6-hour terminal-job retention window so the tracked/returned Codex record set and downstream state tracking stay bounded over long sessions; active jobs are retained regardless of age. Per-poll `jobs/*.json` **enumeration** itself is unchanged and still scales with total accumulated job files on disk (nothing prunes them) — the existing mtime cache and one-time slow-glob warning are preserved for that reason.
+- **`scripts/agent-watchdog.py`** `CodexScanner.scan()`: apply the terminal-job retention cutoff before a job's `sessionId` is folded into the session-disambiguation set, not just before it lands in `records` — an aged-out terminal job's session no longer counts as an ambiguity candidate for `_session()`, closing a gap where a long-dead session could still spuriously disable Source D (or steer selection) months after retention was meant to age it out.
+- **`scripts/agent-watchdog.py`** `CodexScanner.scan()`: warn once when a discovered job's self-reported `workspaceRoot` doesn't canonical-match the candidate being scanned, closing the silent-skip gap implicated in a real incident. Matching behavior itself is unchanged.
+- **`scripts/agent-watchdog.py`** `Watchdog.poll_once()`: warn once when there's a session to monitor but zero discoverable Codex candidate workspaces — either no team config, or a team whose lead and active members report no cwd, with no discovered agent worktrees either. Both shapes previously discovered nothing and warned nothing, indistinguishable from a healthy fleet with no Codex activity; the warning names which shape it hit so its suggested remedy fits. Discovery behavior itself is unchanged.
+
+### Added
+
+- **`tests/test_agent_watchdog.py`**: cover previously-untested default-path branches — a real `git init` Source-D fixture (`git_toplevel()`/`resolve_workspace()`), `Watchdog._task_dir`, the default relative `_worktrees` resolution path, `_subagent_dirs` autodetect, and `member_transcripts()` fallback/ambiguity branches.
+
+### Changed
+
+- **`skills/grand-admiral/SKILL.md`** § Terminating Teammates: note that a `TaskStop` success response for a Monitor-wrapped background process doesn't prove the underlying OS process actually died.
+- **`skills/codex-crew/SKILL.md`** + `references/sandbox-and-recovery.md`: Codex `git commit` inside a linked worktree is confirmed inconsistent (observed both ways 2026-07-16). One dispatch committed as `f2639aa`; a later dispatch in a different worktree hit the exact old `index.lock`/read-only error and was committed by the coordinator as `7c2d3e8`. `writable_roots` was unchanged; the likely enabling lever is `approval_policy = "on-request"` + the project's `trust_level = "trusted"`, not a sandbox-path change. Coordinator-commit is the reliable default, not just a fallback for a regression.
+- **`skills/ci-dance/SKILL.md`** § Step 2: document the fallback when `/ci-dance` itself runs as a delegated (non-lead) teammate — named stream spawns fail outright on a flat team roster ("teammates cannot spawn other teammates"); use unnamed background subagents and rely on Step 3's merge-time reconciliation instead of the claim/completion protocol.
+
 ## [5.9.0] - 2026-07-15
 
 ### Changed

scripts/agent-watchdog.py

@@ -52,6 +52,8 @@
 # per workspace when enumerating them exceeds this, signalling the directory has
 # grown enough to justify pruning old records or a bounded-scan optimization.
 JOBS_GLOB_WARN_SECS = 0.5
+# Keep active jobs indefinitely, but age terminal records out after six hours.
+TERMINAL_JOB_RETENTION_SECS = 6 * 60 * 60
 STATE_VERSION_PREFIX = re.compile(r'^\s*\{\s*"version"\s*:\s*')
 # Sentinel: the state file is a JSON object but its version could not be read
 # from the bounded header (file too large and "version" is not near the start).
@@ -880,11 +882,19 @@ def _minimal_broker(value: Any) -> Any:
             return value
         return {"pid": value.get("pid"), "endpoint": value.get("endpoint")}
 
-    def scan(self, candidates: Iterable[Path], effective_session: str) -> ScanResult:
-        """Scan every retained matching job without enumerating global state."""
+    def scan(
+        self,
+        candidates: Iterable[Path],
+        effective_session: str,
+        now: float | None = None,
+    ) -> ScanResult:
+        """Scan active and recent matching jobs without enumerating global state."""
         warnings: list[tuple[str, str]] = []
         if not effective_session:
             return ScanResult([], warnings)
+        terminal_cutoff = (
+            time.time() if now is None else now
+        ) - TERMINAL_JOB_RETENTION_SECS
         workspaces: dict[Path, WorkspaceInfo] = {}
         for candidate in candidates:
             try:
@@ -949,7 +959,7 @@ def scan(self, candidates: Iterable[Path], effective_session: str) -> ScanResult
                     f"codex-jobs-glob-slow:{info.key}",
                     f"Codex job enumeration for {info.key} took {glob_elapsed:.2f}s "
                     f"({len(job_paths)} files); per-poll cost scales with retained "
-                    "jobs/*.json — prune old job records or add a bounded scan",
+                    "jobs/*.json — prune old job records",
                     warnings,
                 )
             for job_path in job_paths:
@@ -964,7 +974,33 @@ def scan(self, candidates: Iterable[Path], effective_session: str) -> ScanResult
                 except OSError:
                     raw_canonical = Path(os.path.realpath(workspace_root))
                 if raw_canonical != info.canonical:
+                    job_session = raw.get("sessionId")
+                    plausibly_tracked = not (
+                        isinstance(job_session, str)
+                        and job_session
+                        and not _prefix_matches(job_session, effective_session)
+                    )
+                    if plausibly_tracked:
+                        self._warning(
+                            f"codex-job-workspace-mismatch:{info.key}:{job_path.name}",
+                            f"Codex job {job_path.name!r} in {info.key} reports "
+                            f"workspaceRoot={workspace_root!r} which doesn't match this "
+                            f"candidate's resolved path {info.canonical} — job skipped, "
+                            "may be silently invisible to CODEX_* events",
+                            warnings,
+                        )
                     continue
+                persisted_status = raw.get("status")
+                if persisted_status not in ACTIVE_STATUSES:
+                    terminal_status = TERMINAL_STATUSES.get(str(persisted_status), "")
+                    if terminal_status:
+                        job_mtime = safe_mtime(job_path)
+                        if job_mtime is not None and job_mtime < terminal_cutoff:
+                            # Aged-out terminal job: exclude from session
+                            # disambiguation too, not just from records —
+                            # otherwise a long-dead session can still make
+                            # _session() see it as a live ambiguity candidate.
+                            continue
                 session = raw.get("sessionId")
                 if isinstance(session, str) and session:
                     sessions.add(session)
@@ -1000,6 +1036,13 @@ def scan(self, candidates: Iterable[Path], effective_session: str) -> ScanResult
                         warnings,
                     )
                     continue
+                job_mtime = safe_mtime(job_path)
+                if (
+                    status != "active"
+                    and job_mtime is not None
+                    and job_mtime < terminal_cutoff
+                ):
+                    continue
                 matching.append((job_path, raw, status))
             active_count = sum(status == "active" for _, _, status in matching)
             broker = self.cache.read(
@@ -1793,7 +1836,24 @@ def poll_once(self, now: int | None = None) -> list[str]:
         codex_candidates.extend(source_c)
         codex_records: Iterable[CodexRecord] = ()
         if effective_session:
-            scan = self.scanner.scan(codex_candidates, effective_session)
+            if not codex_candidates:
+                if team is None:
+                    cause = "no team config and no discovered agent worktrees"
+                    remedy = "dispatch at least one NAMED teammate to create a team"
+                else:
+                    cause = (
+                        "a team config with no active member or lead cwd, and no "
+                        "discovered agent worktrees"
+                    )
+                    remedy = "dispatch a NAMED teammate that reports a cwd"
+                self.warn_once(
+                    "codex-zero-candidates",
+                    f"Codex Source D: {cause} — 0 candidate workspaces to scan "
+                    "this poll; Codex job monitoring is effectively disabled "
+                    f"({remedy}, or verify --worktrees points at where Codex "
+                    "worktrees actually live, if Codex jobs are expected).",
+                )
+            scan = self.scanner.scan(codex_candidates, effective_session, epoch)
             for warning_key, message in scan.warnings:
                 self.warn_once(warning_key, message)
             codex_records = scan.records

skills/ci-dance/SKILL.md

@@ -77,6 +77,8 @@ Spawn each stream as a named `Agent()` — every session has one implicit team,
 - `grumpy-stream`
 - `review-stream`
 
+**Named spawning requires this skill to be running in the session lead.** If a lead delegates the whole `/ci-dance` invocation to a teammate rather than running it itself, every named spawn above fails outright — "Teammates cannot spawn other teammates" (flat team roster). If you find yourself running this skill as a non-lead teammate: spawn the three streams as **unnamed** background subagents instead (omit `name`), skip the entire Inter-Stream Communication claim/completion protocol below (unnamed agents can't be addressed by `SendMessage`), and rely solely on Step 3's merge-time cherry-pick/conflict resolution as the overlap trust boundary — it already degrades gracefully to this. Step 3/6's `shutdown_request` likewise doesn't apply to unnamed subagents; they simply run to completion.
+
 ### Team-spawn worktree quirk
 
 See `grand-admiral` § Worktree Isolation for the canonical write-up. Summary for ci-dance:

skills/codex-crew/SKILL.md

@@ -1,13 +1,13 @@
 ---
 name: codex-crew
-description: Use before dispatching work to Codex (codex:codex-rescue) — deciding whether to route coding to Codex Sol, handling a Codex job that cannot commit or write, monitoring a running Codex job, or recovering a stale Codex broker. Pre-flight the coordinator reads once before its first Codex dispatch of a session.
+description: Use before dispatching work to Codex (codex:codex-rescue) — deciding whether to route coding to Codex Sol, handling a Codex job that fails to write or commit, monitoring a running Codex job, or recovering a stale Codex broker. Pre-flight the coordinator reads once before its first Codex dispatch of a session.
 ---
 
 # Codex Crew — Enlisting Codex Agents
 
 Codex agents (OpenAI Codex CLI, dispatched through `codex:codex-rescue`) are external crew a coordinator can enlist alongside the named claudius roster. Use of Codex is **opt-in**. Read this once before the first Codex dispatch of a session — it covers routing, the sandbox's hard limits, how to monitor a Codex job, and how to recover a stuck broker.
 
-The recurring failure this skill prevents: coordinators re-derive the same Codex sandbox and orchestration quirks session after session, each losing time to the same commit-block, write-rejection, and broker-staleness traps.
+The recurring failure this skill prevents: coordinators re-derive the same Codex sandbox and orchestration quirks session after session, each losing time to the same write-rejection and broker-staleness traps (and an inconsistent commit path — see Sandbox & Workdir rule 2).
 
 ## When to Enlist Codex
 
@@ -18,7 +18,7 @@ The recurring failure this skill prevents: coordinators re-derive the same Codex
 ## Routing — One Model, High Effort
 
 - **Codex Sol = `--model gpt-5.6-sol --effort high`. Always high effort.** State both flags explicitly on every dispatch: `codex:codex-cli-runtime` only forwards `--effort`/`--model` when present in the request text, so an omitted flag silently drops to the runtime default.
-- **Dispatch through `codex:codex-rescue`.** It is a thin forwarder: exactly one `task` invocation, returning that stdout unchanged. It does **not** monitor, poll, fetch results, commit, or inspect the repo — every one of those is **coordinator** work (see Monitoring, and Coordinator Commits below).
+- **Dispatch through `codex:codex-rescue`.** It is a thin forwarder: exactly one `task` invocation, returning that stdout unchanged. It does **not** monitor, poll, or fetch results on its own initiative — that's **coordinator** work (see Monitoring below). It CAN attempt a commit when the dispatch prompt explicitly instructs it to, but success is inconsistent; the coordinator must verify independently (see Sandbox & Workdir rule 2).
 - The lighter `spark` alias (`gpt-5.3-codex-spark`) exists, but claudius routing standardizes on Sol at high effort.
 
 ## Sandbox & Workdir — The Load-Bearing Rules
@@ -27,11 +27,11 @@ Codex runs under `sandbox_mode = "workspace-write"` (see `~/.codex/config.toml`)
 
 1. **Write scope = cwd + configured `writable_roots`.** On this host `writable_roots` includes `/data/git-worktrees`, `/data/tmp`, `/data/artifacts`, `/data/target` (the shared cargo target dir), plus `network_access = true`. So worktrees under `/data/git-worktrees/<slug>` (the mandatory global worktree location) **are** writable by Codex, scratch under `/data/tmp` and `/data/artifacts` is writable, cargo build output under `/data/target` is writable, and sandboxed tests **can** bind localhost sockets. Paths outside cwd and `writable_roots` are read-only.
 
-2. **Codex CANNOT `git commit` in a linked worktree — the coordinator commits on its behalf.** A linked worktree's git metadata lives outside the sandbox's writable set, so the commit is rejected (mechanics: `references/sandbox-and-recovery.md` § Why Codex Cannot `git commit`). This is normal, not a failure to troubleshoot. **Pattern: Codex writes the files; the coordinator (unsandboxed) runs `git add`/`git commit`.** Plan every Codex dispatch with a coordinator commit step — never wait on Codex to commit.
+2. **Codex `git commit` in a linked worktree is inconsistent — confirmed both ways the same day (2026-07-16).** One dispatch committed cleanly (`f2639aa`, this repo, no approval prompt). A later dispatch, same repo, different worktree, hit the exact old "Git metadata is read-only"/`index.lock` error and had to be committed by the coordinator instead (`7c2d3e8`). `writable_roots` was unchanged across both, so whatever gates this isn't a static config value — likely `approval_policy = "on-request"` + `trust_level = "trusted"` interacting with something per-dispatch, not independently confirmed. **Treat coordinator-commit as the reliable default, not a fallback**: it is fine to instruct Codex to attempt `git add`/`git commit` itself as its final step (with an explicit commit message — it doesn't know your conventions unless told), but always plan for that attempt to fail and verify afterward — check `git log`/`git status` in the worktree rather than trusting Codex's self-report, and commit yourself (unsandboxed) when it didn't land. See `references/sandbox-and-recovery.md` § Git Commit in a Linked Worktree for both data points.
 
 3. **All worktrees live under `/data/git-worktrees/<slug>`** (global environment rule; slug = the startup `$PWD` path). The coordinator pre-creates the worktree following the isolation pattern in `grand-admiral` § Worktree Isolation — which owns the pre-create-and-inject-absolute-path procedure, not this concrete path — and injects that absolute path into the dispatch.
 
-Deep mechanics (exact sandbox modes, the on-disk job-state layout, why `git commit` is blocked) are in `references/sandbox-and-recovery.md`.
+Deep mechanics (exact sandbox modes, the on-disk job-state layout, `git commit` in a linked worktree status and fallback) are in `references/sandbox-and-recovery.md`.
 
 ## Monitoring a Codex Job
 
@@ -50,4 +50,4 @@ Recovery: find the orphaned broker PID (its `--cwd` points at the old worktree p
 
 ## Additional Resources
 
-- **`references/sandbox-and-recovery.md`** — sandbox modes, the `workspace-write` config, on-disk job-state layout for monitoring, the git-commit block explained, and copy-paste broker-recovery commands.
+- **`references/sandbox-and-recovery.md`** — sandbox modes, the `workspace-write` config, on-disk job-state layout for monitoring, git-commit-in-a-worktree status (inconsistent) and its fallback, and copy-paste broker-recovery commands.