From 99b0f1ea6ac6c07fa635d9cd89138ea1ed9f486b Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Fri, 15 May 2026 04:40:21 -0400 Subject: [PATCH 1/2] feat(autonomous-loop): wire cron-sentinel-mutex into Step 1 refresh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the PR #3375 "Next step (not in this PR): wire this into the autonomous-loop substrate so the <> tick body invokes the mutex at the top and defers when peers are detected." Added to docs/AUTONOMOUS-LOOP-PER-TICK.md Step 1 (Refresh): - New `cron-sentinel-mutex.ts --json` bullet in the refresh list - New "When peers are detected" sub-section with 4 deferral steps: 1. Avoid `git worktree add` (worktree-prune-race rationale) 2. Continue with non-git-mutating work (bus, audits, planning) 3. Bus-publish a deferral envelope if substrate matters past tick 4. Re-check next tick (contention windows resolve in 1-3 min) - Special case: exit code 251 (PGREP_ERROR_EXIT) — proceed but log Per the 3-surface canonical convergence, this update propagates to Otto-CLI (auto-loaded next cold-boot), Otto-Desktop routine (cites this file), and B-0448 cloud routine (when shipped — will cite this file). The discipline is ADVISORY, not a hard gate: the mutex reports state, the tick body decides. Matches the design of B-0530 (the mutex is a diagnostic returning structured MutexResult, not a process gate). Composes with: - PR #3370 (worktree-prune-race root cause + B-0519 Pattern 8) - PR #3375 (mutex implementation) - PR #3377 (borrow-on-existing pattern — alternative when peer contention is encountered) - PR #3386 (bulk rule-link depth fix across affected shards) Co-Authored-By: Claude --- docs/AUTONOMOUS-LOOP-PER-TICK.md | 35 ++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/AUTONOMOUS-LOOP-PER-TICK.md b/docs/AUTONOMOUS-LOOP-PER-TICK.md index 7cd7019c87..f6aca99f61 100644 --- a/docs/AUTONOMOUS-LOOP-PER-TICK.md +++ b/docs/AUTONOMOUS-LOOP-PER-TICK.md @@ -38,6 +38,41 @@ Never act on stale state. Minimum refresh: - `bun tools/github/poll-pr-gate-batch.ts --all-open` — current state of all my open PRs - `git fetch origin main && git status` — main HEAD + local state - `CronList` — verify the autonomous-loop sentinel is still armed +- `bun tools/orchestrator-checks/cron-sentinel-mutex.ts --json` — detect concurrent Otto-CLI peer sessions + ([B-0530](backlog/P3/B-0530-cron-sentinel-mutex-prevent-otto-cli-self-contention-2026-05-15.md); + Pattern 8 of [B-0519](backlog/P3/B-0519-multi-otto-branch-state-contamination-rca-2026-05-14.md)) + +#### When peers are detected + +If the mutex check reports `peerDetected: true` (or exits with code +1..250), the tick body should: + +1. **Avoid `git worktree add`** — the worktree-prune-race RCA in PR + #3370 documents that concurrent `git worktree add` from a peer + Otto-CLI on the same `.git/` directory causes + `Interrupted system call` failures on `.git/objects/pack` followed + by git's automatic rollback of the partially-populated worktree. + If a worktree is genuinely needed, prefer the + ["borrow-on-existing pattern"](../.claude/rules/claim-acquire-before-worktree-work.md) + landed in PR #3377. +2. **Continue with non-git-mutating work** — bus envelope publishing, + read-only audits, planning, etc. are safe and don't contend. +3. **Bus-publish a deferral envelope** if substrate observation + matters past this tick: + + ```bash + bun tools/bus/bus.ts publish --from otto-cli --to '*' \ + --topic shadow-catch \ + --payload '{"finding":"tick deferred — peer Otto-CLI detected", ...}' + ``` + +4. **Re-check next tick** — peer-Otto-CLI sessions typically wrap + their cron-tick work within 1-3 minutes; the contention window + resolves naturally. + +If the mutex exits with code 251 (`PGREP_ERROR_EXIT`), the mutex +itself is unreliable — proceed with normal work but log the failure +in the tick shard for future-Otto context. ### 2. Apply Holding-without-named-dependency discipline From 5ed1431719a561bfb8474ef196b2327d9bfde99a Mon Sep 17 00:00:00 2001 From: Aaron Stainback Date: Fri, 15 May 2026 05:15:12 -0400 Subject: [PATCH 2/2] fix(autonomous-loop): correct cron-sentinel-mutex exit-code range and exit-251 guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Exit code range for peerDetected=true is 2..250 (Math.min(1+peerCount,250)), not 1..250; exit 1 is unreachable when peers are detected - Replace `{..., ...}` JSON placeholder in bus.ts publish example with valid JSON so the command doesn't hard-fail when copy-pasted - Exit 251 (PGREP_ERROR_EXIT) means pgrep failed and state is unknown; treat as peer-detected for git-mutating ops (defer worktree add), matching the 'caller should defer' comment in the implementation Addresses Codex P1 and 3× Copilot P1 threads on PR #3390. Co-Authored-By: Claude Sonnet 4.6 --- docs/AUTONOMOUS-LOOP-PER-TICK.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/AUTONOMOUS-LOOP-PER-TICK.md b/docs/AUTONOMOUS-LOOP-PER-TICK.md index f6aca99f61..2bfaaeffc4 100644 --- a/docs/AUTONOMOUS-LOOP-PER-TICK.md +++ b/docs/AUTONOMOUS-LOOP-PER-TICK.md @@ -45,7 +45,8 @@ Never act on stale state. Minimum refresh: #### When peers are detected If the mutex check reports `peerDetected: true` (or exits with code -1..250), the tick body should: +2..250 — `Math.min(1 + peerCount, 250)`; exit 1 is not reachable when +peers are detected), the tick body should: 1. **Avoid `git worktree add`** — the worktree-prune-race RCA in PR #3370 documents that concurrent `git worktree add` from a peer @@ -63,16 +64,19 @@ If the mutex check reports `peerDetected: true` (or exits with code ```bash bun tools/bus/bus.ts publish --from otto-cli --to '*' \ --topic shadow-catch \ - --payload '{"finding":"tick deferred — peer Otto-CLI detected", ...}' + --payload '{"finding":"tick deferred — peer Otto-CLI detected"}' ``` 4. **Re-check next tick** — peer-Otto-CLI sessions typically wrap their cron-tick work within 1-3 minutes; the contention window resolves naturally. -If the mutex exits with code 251 (`PGREP_ERROR_EXIT`), the mutex -itself is unreliable — proceed with normal work but log the failure -in the tick shard for future-Otto context. +If the mutex exits with code 251 (`PGREP_ERROR_EXIT`), pgrep itself +failed — mutex state is unknown. Treat the same as peer-detected for +git-mutating operations: **defer `git worktree add`**, continue with +non-git-mutating work, and log the failure in the tick shard for +future-Otto context. The safe assumption under unknown state is to +avoid operations that contend on `.git/objects/pack`. ### 2. Apply Holding-without-named-dependency discipline