Skip to content

fix(supervision): make Grok primary turn-end supervision reliable without headless resume - #461

Closed
korallis wants to merge 11 commits into
kunchenguid:mainfrom
korallis:fm/fm-supervision-fix
Closed

korallis wants to merge 11 commits into
kunchenguid:mainfrom
korallis:fm/fm-supervision-fix

Conversation

@korallis

Copy link
Copy Markdown
Contributor

Intent

Production readiness for Grok primary supervision reliability after repeated silent fleet gaps (2026-07-11).

Goals:

  1. Stop headless grok --resume; mechanical detached watcher ensure + state/.supervision-gap; FM_HOME preferred; legacy loop-guard only.
  2. Quote-aware seatbelt fail-closed with proper quote scanner (no apostrophe cross-quote false-allow; quoted-exec bypass fixed).
  3. Document arm-only re-arm; AGENTS.md inventory; fm-guard surfaces and retires gap marker when healthy.
  4. Ensure-lock only after mkdir + atomic stale age recovery; log cap; unit tests green.

Push target is the korallis/firstmate fork (write access); open PR to kunchenguid/firstmate. Do not merge.

What Changed

  • Replaced the Grok primary turn-end path's headless grok --resume with a mechanical guard (bin/fm-turnend-guard-grok.sh): on a blind turn it writes a durable state/.supervision-gap marker and directly ensures a detached bin/fm-watch.sh, using an ensure lock taken only after mkdir with atomic stale-age recovery and a size-capped ensure log; the legacy GROK_TURNEND_GUARD_ACTIVE env is kept as a loop-guard no-op.
  • Hardened the watcher-arm seatbelt (bin/fm-arm-command-policy.mjs) to fail closed on quoting: a proper quote scanner eliminates the apostrophe cross-quote false-allow, quoted commands in execution position (including interpreter -c quoted exec) are denied, and --command argument stripping is scoped to seatbelt tools only.
  • bin/fm-guard.sh now surfaces state/.supervision-gap in the watcher-down banner and retires the marker once supervision is healthy, with docs (docs/turnend-guard.md, supervision protocol and architecture docs) synced to the new lock, log, and re-arm behavior; note the branch base also carries prior local-main commits not yet pushed to origin, so the raw diff is wider than the supervision work itself.

Risk Assessment

✅ Low: The delta since the last review round is a 14-line, test-backed fix that I verified empirically closes both reported quoted-exec bypass shapes without regressing legitimate allow cases; only contrived defense-in-depth edges remain.

Testing

Baseline full suite was already green; re-ran the three branch-focused suites (turn-end guard, arm pretool seatbelt, watcher lock) green, then demonstrated the change end-to-end with the real hooks: the Grok Stop hook wrote the durable supervision-gap marker and mechanically ensured a real detached watcher without ever spawning headless grok --resume, fm-guard surfaced the gap in its watcher-down banner and retired it once healthy, the ensure lock/log-cap and legacy loop-guard behaved as documented, and the quote-aware seatbelt allowed previously false-denied diagnostic commands while denying quoted-exec, apostrophe-mispair, interpreter -c, and foreign --command executions. No screenshot artifacts because the change has no rendered UI surface — the terminal hook transcripts are the end-user surface and are captured as evidence.

Evidence: Grok turn-end guard E2E transcript (gap marker, watcher ensure, banner, retirement, loop-guard, log cap)

===== SETUP: sandbox primary home built from the branch's real scripts =====
sandbox home: /tmp/fm-grok-e2e.eMAJqT  (task demo-1 in flight, live tmux window fm-demo-1)

$ ls -A /tmp/fm-grok-e2e.eMAJqT/state
demo-1.meta
-> no watcher lock, no beacon: the fleet is unsupervised. Old behavior: a
   grok turn end here spawned headless 'grok --resume' zombies and did nothing.

===== 1. Grok Stop hook fires on turn end (payload piped exactly as grok does) =====
hook exit code: 0 (passive contract: always 0, no output)

===== 2. Evidence: durable gap marker written =====

$ cat /tmp/fm-grok-e2e.eMAJqT/state/.supervision-gap
2026-07-11T08:19:49Z
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  TURN WOULD END BLIND - SUPERVISION IS OFF
●  1 task(s) in flight, but no live watcher holds this home lock (last beat: never).
●  resume supervision with bin/fm-watch-arm.sh as its own Claude Code background task, never shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

===== 3. Evidence: real watcher mechanically ensured (detached, singleton) =====
watch.lock pid: 87128

$ ps -o pid,command -p 87128
  PID COMMAND
87128 bash /tmp/fm-grok-e2e.eMAJqT/bin/fm-watch.sh

$ ls -l /tmp/fm-grok-e2e.eMAJqT/state/.last-watcher-beat
-rw-r--r--@ 1 leebarry  wheel  0 Jul 11 09:19 /tmp/fm-grok-e2e.eMAJqT/state/.last-watcher-beat
ensure log:

$ cat /tmp/fm-grok-e2e.eMAJqT/state/.turnend-watch-ensure.log

===== 4. Evidence: NO headless grok --resume was spawned =====
OK: decoy grok binary was never invoked

===== 5. Later failure: watcher dies, beacon goes stale -> next fleet action shows the alarm =====
$ bin/fm-guard.sh   (as any supervision script would run it)
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no watcher has a fresh beacon (last beat: 205921191s ago, grace 300s).
●  state/.supervision-gap present (since 2026-07-11T08:19:49Z) - a prior primary turn ended blind; see docs/turnend-guard.md.
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  resume supervision with bin/fm-watch-arm.sh as its own Claude Code background task, never shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
guard exit: 0 ; gap marker kept while down: yes

===== 6. Recovery: next grok turn end re-ensures the watcher =====
hook exit code: 0
fresh watcher pid: 87953 (old was 87128)

$ ps -o pid,command -p 87953
  PID COMMAND
87953 bash /tmp/fm-grok-e2e.eMAJqT/bin/fm-watch.sh

===== 7. Healthy again: fm-guard is silent and retires the gap marker =====
$ bin/fm-guard.sh
guard exit: 0 (no banner)
gap marker retired: yes

===== 8. Legacy loop-guard env: nested resume dies immediately, writes nothing =====
hook exit code: 0
gap marker written: no
watcher relaunched: no

===== 9. Ensure-log size cap: oversized log is trimmed to last 500 lines =====
log bytes before: 288894  after: 7500  (old lines kept: 500)

===== DONE =====
Evidence: Quote-aware seatbelt E2E transcript (10 allow/deny cases through the real PreToolUse transport)
== Diagnostic mentions that used to false-deny and block the Grok primary ==

--- quoted mention is data, not execution
    command: for x in 1; do echo "don't run bin/fm-watch-arm.sh"; done
    result: ALLOW (expected ALLOW)  [OK]

--- rg over docs mentioning the watcher
    command: rg -n 'bin/fm-watch-arm.sh' docs
    result: ALLOW (expected ALLOW)  [OK]

--- grep -c is a count flag, not an interpreter
    command: if true; then grep -c "bin/fm-watch-arm.sh" docs; fi
    result: ALLOW (expected ALLOW)  [OK]

--- seatbelt's own --command fixture
    command: if true; then bin/fm-arm-pretool-check.sh --command bin/fm-watch-arm.sh; fi
    result: ALLOW (expected ALLOW)  [OK]

== Real protected executions that must stay (or newly become) denied ==

--- apostrophes must not cross-pair and hide a real background arm
    command: for x in 1; do echo "don't"; done && for y in 1; do bin/fm-watch-arm.sh & done && echo "won't"
    result: DENY (expected DENY)  [OK]
    decision: {"decision":"deny","reason":"[unclassifiable-protected-command] unsupported or malformed shell syntax contains a protected watcher command"}

--- quoted path in command position still executes (quoted-exec bypass)
    command: for i in 1; do "bin/fm-watch-arm.sh" & done
    result: DENY (expected DENY)  [OK]
    decision: {"decision":"deny","reason":"[unclassifiable-protected-command] unsupported or malformed shell syntax contains a protected watcher command"}

--- single-quoted direct watcher execution
    command: while true; do 'bin/fm-watch.sh'; done
    result: DENY (expected DENY)  [OK]
    decision: {"decision":"deny","reason":"[unclassifiable-protected-command] unsupported or malformed shell syntax contains a protected watcher command"}

--- interpreter -c executes its quoted payload
    command: if true; then bash -c "bin/fm-watch-arm.sh"; fi
    result: DENY (expected DENY)  [OK]
    decision: {"decision":"deny","reason":"[unclassifiable-protected-command] unsupported or malformed shell syntax contains a protected watcher command"}

--- foreign tool's --command is execution, not a fixture
    command: if true; then su root --command bin/fm-watch-arm.sh; fi
    result: DENY (expected DENY)  [OK]
    decision: {"decision":"deny","reason":"[unclassifiable-protected-command] unsupported or malformed shell syntax contains a protected watcher command"}

== The blessed arm shape stays allowed ==

--- standalone watcher arm
    command: bin/fm-watch-arm.sh
    result: ALLOW (expected ALLOW)  [OK]

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⚠️ **Rebase** - 1 warning

Push main to origin, or rebase your branch onto origin/main, before gating.

🔧 Fix applied.
1 warning still open:

Push main to origin, or rebase your branch onto origin/main, before gating.

⚠️ **Review** - 1 info
  • ⚠️ bin/fm-arm-command-policy.mjs:70 - The new unsupported-grammar fallback strips quoted regions as data unless they open command position, so a quoted protected path executed through an interpreter is false-allowed: if true; then bash -c "bin/fm-watch-arm.sh"; fi returns allow (verified with node), while the identical bash -c "bin/fm-watch-arm.sh" outside the if denies as watcher-nested, and the pre-change fallback (rawMentionsProtected) denied any mention. Same gap via stripCommandFlagArgs for tools whose --command flag executes its argument (e.g. if true; then su root --command bin/fm-watch-arm.sh; fi allows). Broad-kill shapes remain denied, so the destructive cases are still covered, but this is a real execution shape the seatbelt used to catch. Consider keeping quoted content visible when the preceding word is an interpreter -c/-lc style flag, or restricting --command stripping to fm-arm-pretool-check.sh invocations.
  • ℹ️ bin/fm-turnend-guard-grok.sh:86 - The ensure-lock steal is not fully atomic despite the comment: two racers that both pass the age>=60 check can interleave so racer B's mv steals racer A's freshly recreated lock (age-check TOCTOU), and both proceed; A's EXIT cleanup then rmdir's B's lock. Consequence is bounded - at worst two detached fm-watch.sh spawns, and the watcher's own singleton lock self-evicts the duplicate - so this is acceptable, but the "two racers can never both claim it" comment overclaims.
  • ℹ️ bin/fm-turnend-guard-grok.sh:101 - Log rotation replaces the inode (tail > tmp && mv), but a previously detached watcher still holds an open append fd on the old unlinked inode; its subsequent output bypasses the size cap invisibly until that watcher exits, and the visible log misses it. Debug-only log with bounded practical growth, so acceptable as-is; appending rotation (truncate-in-place) would avoid it if it ever matters.
  • ℹ️ bin/fm-guard.sh:75 - The gap-marker retire keys on the loose beacon-freshness predicate (fm_supervision_status FM_SUP_WATCHER_FRESH), while the writer (fm-turnend-guard-grok.sh via fm_watcher_healthy) requires an identity-matched lock. A fresh beacon with a mismatched lock identity would retire a gap the turn-end guard still considers blind; self-correcting because the marker is rewritten on the next blind turn, so worst case is a briefly missing banner line.

🔧 Fix: deny interpreter -c quoted exec; scope --command stripping to seatbelt tools
1 info still open:

  • ℹ️ bin/fm-arm-command-policy.mjs:68 - The interpreter -c detection requires the shell name immediately before the -c flag, so if true; then bash -o errexit -c "bin/fm-watch-arm.sh"; fi still false-allows in the unsupported-grammar fallback, and non-shell interpreters (python3 -c 'os.system("bin/fm-watch-arm.sh")') allow on both the main and fallback paths (verified with node). Contrived shapes for a seatbelt guarding against accidental unsafe commands; the realistic sh/bash/zsh -c shapes from the round-1 finding are now covered with tests. Acceptable as-is.
✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
  • Baseline configured command (all tests/*.test.sh under tmux) ran green before this session
  • bash tests/fm-turnend-guard.test.sh (38 checks: predicate, shared hook, grok adapter gap/ensure/lock/loop-guard, per-harness hook wiring)
  • bash tests/fm-arm-pretool-check.test.sh (matrix + direct policy contract incl. new quoted-data allow, quote-mispair deny, quoted-exec deny, interpreter -c deny, --command scoping)
  • bash tests/fm-watcher-lock.test.sh (incl. new gap-marker surfaced-in-banner and retired-only-when-healthy assertions)
  • Manual E2E: piped Grok Stop payload through the real bin/fm-turnend-guard-grok.sh in a sandbox primary home with a live tmux task window and a decoy grok on PATH; verified gap marker content, real detached fm-watch.sh with singleton lock + fresh beacon, zero grok invocations, banner surfacing + healthy-path retirement via bin/fm-guard.sh, legacy GROK_TURNEND_GUARD_ACTIVE no-op, and ensure-log 500-line trim of a 288KB log
  • Manual E2E: submitted 10 representative commands through the real bin/fm-arm-pretool-check.sh stdin transport (Grok toolInput schema); all ALLOW/DENY outcomes and Grok-shaped decision objects matched expectations
  • Confirmed no stray demo processes/tmux sessions and a clean worktree afterward
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

korallis added 11 commits July 11, 2026 05:57
Stop headless grok --resume from the turn-end adapter (zombie hang path).
Ensure a detached watcher and write state/.supervision-gap instead.
Quote-aware seatbelt fail-closed so diagnostic shells mentioning arm paths
as data are allowed; real unquoted if/for arm bodies still deny.
Document the standalone arm-only re-arm shape and update unit tests.
Light breadcrumb when a prior Grok blind turn left a gap marker, so the
pull-based guard points operators at the durable record.
@kunchenguid

kunchenguid commented Jul 13, 2026 •

Copy link
Copy Markdown
Owner

Thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#461 at 4ddb46ba.

@kunchenguid

Copy link
Copy Markdown
Owner

Automated reminder: this PR still looks blocked on a rebase or merge conflict fix.

If you are still interested, please rebase onto the current base branch, resolve the conflict, and push.

If I do not hear back, I may close this as inactive.

@kunchenguid

Copy link
Copy Markdown
Owner

I am closing this because it has been waiting on a rebase or merge-conflict fix since 2026-07-13, and I have not seen a comment or push since then.

If you still want to keep working on this, please reopen it or open a new PR and mention this one.

Happy to take another look when there is an update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants