Skip to content

fix(bin): stop the wake drain forking a process per status line, stat, and cursor row - #79

Merged
MrGTV-love merged 22 commits into
mainfrom
fm/fm-wake-drain-fast
Oct 10, 2026
Merged

MrGTV-love merged 22 commits into
mainfrom
fm/fm-wake-drain-fast

Conversation

@MrGTV-love

@MrGTV-love MrGTV-love commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Intent

I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.

A recommended architecture should come with proof.

Context (the backlog record for this bug, as filed):

Wake drain spawns about 4,500 processes per run; cut it so a locked session start meets the under-20-second target (repo: firstmate) (kind: ship) (priority: 1) (since 2026-10-07)
Found by fm-session-start-fast (2026-10-08): after that fix, a full locked session start fell from 107s to 51s on a scratch copy of the main home, but the wake drain alone stays about 22-29s because it creates about 4,500 processes per run. The under-20-second target is not met until the drain is fixed. Measure the drain's process count and time first, then remove the per-line subprocess fan-out. Mode no-mistakes, yolo per firstmate registry.

What Changed

  • The wake drain's status helpers now return values through out-variables instead of command substitutions. This covers status_presentation_cursor_offset, status_outcome_backstop_cursor_offset, status_new_lines_since_cursor, _fm_decision_key_into, _fm_decision_fold_line_into, status_line_note, and _fm_decision_drop. The stdout forms still exist and keep their old bytes. The incremental decision fold folds each line in-process. Routine closes against an empty open set skip the key and note parse. basename and $(printf '\t') are replaced with parameter expansion and $'\t'.
  • bin/fm-status-io-lib.sh adds one batched stat call per scan (_fm_status_stat_batch_into). It also adds a combined identity/size/mtime reader (_fm_status_stat_raw / _fm_status_stat_into) that falls back to one per-file stat. The identity and size reader seams still bypass the batch, and post-read validation in status_snapshot_latest_event still uses a fresh stat. Presentation-manifest reads use _fm_read_file_into, which checks the read length against the file size. A short or failed read is an error, not empty data.
  • status_acknowledge_presented_snapshot now reads and classifies each task's unread span one time. It returns the acknowledgement rows and the UNREAD STATUS rows together, so print_unread_status_section prints the same bytes the receipt acknowledges and does not re-scan the logs. The changes are documented in docs/architecture.md. New tests cover subshell growth as history and fleet size grow, exact fold/checkpoint bytes, manifest read failures, and an append during the latest-event read.

Risk Assessment

✅ Low: The fork-free fold, the batched stat facts and the checked manifest read keep the old byte and failure contracts on every production path I traced: fresh validation before commit, drain work under the presentation lock, and exact fold semantics. Behavioral drain tests cover them, and the one open latency concern was declined by the user.

Testing

I ran the real drain from base and target code roots against the same disposable lab homes. Each home had 25 cold status logs with 1,500 keyed resolutions. Target drain time fell from about 36 s to about 4 s. Subshell entries fell from 13,571 to 202. Stdout and cursor bytes did not change. A real Claude primary on a private fm-lab tmux socket did a locked session start three times, in 7.4 s, 9.7 s and 11.3 s. Each run had a matching lock and completion PID and a complete digest. Load was 32–50 during these runs. The HEAD change's snapshot-race guarantee and the retained helpers passed live. The two focused test suites passed. The manifest-read-failure and flat-fork-count scenario was not driven live in a lab; the unread-status suite (19/19 ok) is its only cover, so it is marked untested. I removed all labs, private tmux servers and the extracted base tree. There is no UI surface, so the evidence is CLI transcripts and JSON measurements.

  • Live validation: ✅ go - 6 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Cold drain over 25 status logs (1,500 keyed resolutions) finishes much faster and with far fewer processes than base ✅ pass live drain-compare-transcript.txt: base 36.7/38.5/34.1 s, 13,571 subshell entries, 488 execs; target 3.9/4.0/3.8 s, 202 subshell entries, 414 execs
Drain output and open-decision cursor bytes are unchanged versus base ✅ pass live drain-compare-transcript.txt: 'IDENTICAL stdout' and 'IDENTICAL cursor bytes (25 files)'; drain-*-task0-open-decisions-cursor.bytes
Warm second drain on the same home stays fast ✅ pass live drain-compare-transcript.txt: warm base 20.3 s vs target 3.6 s
Real Claude locked session start over 25 cold logs completes in under 20 seconds ✅ pass live locked-start-run{1,2,3}-measurement.json: 7.393 s, 9.684 s, 11.300 s; lock and completion PID match the live claude process; digest complete, not truncated
Note appended after the presentation snapshot is not acked; the next drain shows it once and nothing replays (HEAD review fix) ✅ pass live snapshot-race-transcript.txt: ALL CHECKS PASSED; tests/fm-wake-drain-unread-status.test.sh 'presentation cursor advances only through its captured endpoint'
Retained helpers scan_unread_surface_lines and scan_unread_surface_snapshot still return unread rows after batching removal ✅ pass live snapshot-race-transcript.txt 'retained helpers still work'; fm-parent-channel-scan-exclusion.test.log
Manifest read failure aborts without replay or replacement receipts; fork counts stay flat as history and fleet grow ⏸️ untested no The prior payload did not establish a live result. This scenario was not driven separately in a disposable lab home in that run. Its only cover was the existing suite tests/fm-wake-drain-unread-status…
Evidence: Validation summary (all numbers)

Source: Validation summary (all numbers)

Target e98f8c95 vs base 924f479c, real bin/fm-wake-drain.sh, /bin/bash 3.2, disposable marked lab homes.
Fleet: 25 cold status logs x 60 keyed resolved/working pairs (1500 keyed resolutions) + needs-decision/blocked/note on every 5th task.

Cold drain wall time:   base 36.7 / 38.5 / 34.1 s   target 3.9 / 4.0 / 3.8 s   (load ~50-70)
Warm 2nd drain:         base 20.3 s                 target 3.6 s
Subshell entries:       base 13,571                 target 202
External execs:         base 488                    target 414
Stdout base vs target:  IDENTICAL
Open-decision cursors:  IDENTICAL bytes in all 25 files (ident= device:inode:mtime line excluded)

Real Claude locked session start (fm-session-start.sh --source startup via SessionStart hook,
private fm-lab tmux socket, 25 cold logs, no primed cursors):
  run1  7.393 s  load 33.1 -> 32.1  lock=completion=claude pid 78562  digest complete, not truncated
  run2  9.684 s  load 42.8 -> 45.4  lock=completion=claude pid 85082  digest complete, not truncated
  run3 11.300 s  load 47.8 -> 49.5  lock=completion=claude pid 51341  digest complete, not truncated
  limit 20 s: all three under.

HEAD review fix: snapshot race driven through the real drain (note appended after the presentation
snapshot is not acked; next drain shows it once; no replay). Retained helpers scan_unread_surface_lines
and scan_unread_surface_snapshot still return unread rows.
Targeted suites: tests/fm-wake-drain-unread-status.test.sh 19/19 ok; tests/fm-parent-channel-scan-exclusion.test.sh ok.
All lab homes, private tmux servers, and the extracted base tree removed.
Evidence: Base vs target drain comparison transcript

Source: Base vs target drain comparison transcript

host: Darwin 27.0.0 arm64; bash under test: GNU bash, version 3.2.57(1)-release (arm64-apple-darwin26)
fleet: 25 cold status logs x 60 keyed resolved/working pairs (+needs-decision/blocked/note on every 5th task)
load at start: { 87.38 97.15 73.19 }

=== base (~/tmp/fm-base.xG6Z8U) ===
cold drain rep 1: exit=0 wall=36.729s load={ 69.88 91.45 72.21 }
cold drain rep 2: exit=0 wall=38.501s load={ 60.15 86.64 71.23 }
cold drain rep 3: exit=0 wall=34.103s load={ 58.57 83.32 70.64 }
subshell entries (cold drain, DEBUG trap): 13571 exit=0
external execs (cold drain, PATH shims): 488 exit=0
  by command: perl=125 rm=109 basename=71 ps=29 mv=26 wc=25 dirname=25 tr=14 readlink=13 cat=12 sh=10 uname=5 mktemp=5 bash=5 rmdir=3 mkdir=3 ln=3 timeout=1 tasks-axi=1 node=1 git=1 awk=1 
warm second drain: exit=0 wall=20.329s

=== target (~/.no-mistakes/worktrees/32d18ed9638d/01M4H90MTFF5AXBWANAFDMBF3V) ===
cold drain rep 1: exit=0 wall=3.923s load={ 51.11 73.70 68.30 }
cold drain rep 2: exit=0 wall=3.982s load={ 50.94 73.29 68.18 }
cold drain rep 3: exit=0 wall=3.759s load={ 49.26 72.57 67.96 }
subshell entries (cold drain, DEBUG trap): 202 exit=0
external execs (cold drain, PATH shims): 414 exit=0
  by command: perl=100 rm=84 basename=46 ps=29 mv=26 wc=25 dirname=25 tr=14 readlink=13 cat=12 sh=10 uname=5 mktemp=5 bash=5 rmdir=3 mkdir=3 ln=3 git=2 timeout=1 tasks-axi=1 node=1 awk=1 
warm second drain: exit=0 wall=3.604s

load at end: { 51.29 71.83 67.77 }

=== base vs target cold-drain stdout diff (rep1) ===
IDENTICAL stdout
=== base vs target open-decision cursor digests (ident= device:inode:mtime line excluded) ===
IDENTICAL cursor bytes (25 files)
=== target cold drain stdout (rep1) ===
UNREAD STATUS (new since last drain, not re-printed after this presentation):
task0 note: informational note for task 0
task10 note: informational note for task 10
task15 note: informational note for task 15
task20 note: informational note for task 20
task5 note: informational note for task 5
OPEN DECISIONS (still open, folded from the durable status logs - not just the latest line):
task0 [key=pick-0] needs-decision: choose option for task 0
task0 [key=wall-0] blocked: blocked on input 0
task10 [key=pick-10] needs-decision: choose option for task 10
task10 [key=wall-10] blocked: blocked on input 10
task15 [key=pick-15] needs-decision: choose option for task 15
task15 [key=wall-15] blocked: blocked on input 15
task20 [key=pick-20] needs-decision: choose option for task 20
task20 [key=wall-20] blocked: blocked on input 20
task5 [key=pick-5] needs-decision: choose option for task 5
task5 [key=wall-5] blocked: blocked on input 5
OPEN DECISIONS: close one by answering it: bin/fm-send.sh <task> --resolve-key <key> '<answer>'
=== target cold drain stderr (rep1) ===
Evidence: Drain comparison driver

Source: Drain comparison driver

#!/bin/bash
# drain-compare-driver.sh <base-root> <target-root> <evidence-dir>
# Runs the REAL bin/fm-wake-drain.sh from each code root against identical,
# disposable, marked lab homes holding 25 cold status logs (60 keyed resolved/
# working pairs each = 1500 keyed resolutions, plus a few open decisions).
# Measures: wall time (3 cold reps), subshell entries (DEBUG-trap, same method
# as tests/fm-wake-drain-unread-status.test.sh), and external execs (PATH shims).
# Compares stdout and open-decision cursor bytes between base and target.
set -u
BASE=$1 TARGET=$2 EV=$3
WORK=$(mktemp -d "${TMPDIR:-/tmp}/fm-drain-cmp.XXXXXX")
trap 'rm -rf "$WORK"' EXIT
TASKS=${TASKS:-25} PAIRS=${PAIRS:-60} REPS=${REPS:-3}

# Shim dir: one shim per executable name on PATH; counts every exec by name.
SHIM="$WORK/shim"; mkdir -p "$SHIM"
IFS=: read -r -a dirs <<< "$PATH"
for d in "${dirs[@]}"; do
  [ -d "$d" ] || continue
  for f in "$d"/*; do
    n=${f##*/}
    [ -x "$f" ] && [ ! -d "$f" ] || continue
    [ -e "$SHIM/$n" ] && continue
    printf '#!/bin/sh\nprintf "%%s\\n" %q >> "$FM_EXEC_LOG"\nexec %q "$@"\n' "$n" "$f" > "$SHIM/$n"
    chmod +x "$SHIM/$n"
  done
done

build_fleet() {  # <state>
  local state=$1 t i
  for ((t = 0; t < TASKS; t++)); do
    {
      printf 'working: starting task %s\n' "$t"
      for ((i = 0; i < PAIRS; i++)); do
        printf 'resolved [key=side-%s]: routine close %s\n' "$i" "$i"
        printf 'working: step %s\n' "$i"
      done
      if [ $((t % 5)) -eq 0 ]; then
        printf 'needs-decision [key=pick-%s]: choose option for task %s\n' "$t" "$t"
        printf 'blocked [key=wall-%s]: blocked on input %s\n' "$t" "$t"
        printf 'note: informational note for task %s\n' "$t"
      fi
    } > "$state/task$t.status"
  done
}

new_lab() {  # prints a fresh marked lab home with the fleet
  local lab
  lab=$(mktemp -d "$WORK/fm-lab.XXXXXX")
  rmdir "$lab"
  "$TARGET/bin/fm-lab-home.sh" create "$lab" >/dev/null || { echo "lab create failed" >&2; exit 1; }
  mkdir -p "$lab/tmux"
  build_fleet "$lab/state"
  printf '%s\n' "$lab"
}

run_drain() {  # <root> <lab> <mode> <out-prefix>
  local root=$1 lab=$2 mode=$3 out=$4
  local cleanenv=(env -u NO_MISTAKES_GATE -u FM_GATE_REFUSE_BYPASS -u FM_ROOT_OVERRIDE -u FM_STATE_OVERRIDE
    -u FM_DATA_OVERRIDE -u FM_CONFIG_OVERRIDE -u FM_PROJECTS_OVERRIDE -u TMUX -u TMUX_PANE
    FM_HOME="$lab" TMUX_TMPDIR="$lab/tmux")
  case $mode in
    plain)
      "${cleanenv[@]}" /bin/bash "$root/bin/fm-wake-drain.sh" > "$out.stdout" 2> "$out.stderr" ;;
    subshell)
      : > "$out.subshells"
      "${cleanenv[@]}" SUBSHELL_ENTRY_COUNT="$out.subshells" SUBSHELL_ENTRY_DRAIN="$root/bin/fm-wake-drain.sh" /bin/bash -c '
        set -T
        seen_depth=0
        trap '\''if [ "$BASH_SUBSHELL" != "$seen_depth" ]; then seen_depth=$BASH_SUBSHELL; printf x >> "$SUBSHELL_ENTRY_COUNT"; fi'\'' DEBUG
        . "$SUBSHELL_ENTRY_DRAIN"
      ' > "$out.stdout" 2> "$out.stderr" ;;
    exec)
      : > "$out.execs"
      "${cleanenv[@]}" FM_EXEC_LOG="$out.execs" PATH="$SHIM:$PATH" /bin/bash "$root/bin/fm-wake-drain.sh" > "$out.stdout" 2> "$out.stderr" ;;
  esac
}

now() { /usr/bin/perl -MTime::HiRes=time -e 'printf "%.3f", time'; }

{
echo "host: $(uname -srm); bash under test: $(/bin/bash --version | head -1)"
echo "fleet: $TASKS cold status logs x $PAIRS keyed resolved/working pairs (+needs-decision/blocked/note on every 5th task)"
echo "load at start: $(sysctl -n vm.loadavg)"
for side in base target; do
  root=$BASE; [ "$side" = target ] && root=$TARGET
  echo
  echo "=== $side ($root) ==="
  for ((r = 1; r <= REPS; r++)); do
    lab=$(new_lab)
    s=$(now); run_drain "$root" "$lab" plain "$EV/drain-$side-rep$r"; rc=$?; e=$(now)
    echo "cold drain rep $r: exit=$rc wall=$(echo "$e - $s" | bc)s load=$(sysctl -n vm.loadavg)"
    if [ "$r" = 1 ]; then
      ( cd "$lab/state" && for f in .*open-decisions* ; do [ -e "$f" ] && { printf '%s\t' "$f"; grep -v "^ident=" "$f" | /usr/bin/shasum | cut -c1-16; }; done ) > "$EV/drain-$side-cursors.txt"
      ( cd "$lab/state" && ls -A ) > "$EV/drain-$side-state-files.txt"
      cp "$lab/state/.task0.open-decisions-cursor" "$EV/drain-$side-task0-open-decisions-cursor.bytes" 2>/dev/null || true
    fi
    rm -rf "$lab"
  done
  lab=$(new_lab); run_drain "$root" "$lab" subshell "$EV/drain-$side-instr"; rc=$?
  echo "subshell entries (cold drain, DEBUG trap): $(wc -c < "$EV/drain-$side-instr.subshells" | tr -d ' ') exit=$rc"
  rm -rf "$lab"
  lab=$(new_lab); run_drain "$root" "$lab" exec "$EV/drain-$side-instr"; rc=$?
  echo "external execs (cold drain, PATH shims): $(wc -l < "$EV/drain-$side-instr.execs" | tr -d ' ') exit=$rc"
  echo "  by command: $(sort "$EV/drain-$side-instr.execs" | uniq -c | sort -rn | awk '{printf "%s=%s ", $2, $1}')"
  # warm second drain on the same home: steady state after presentation
  s=$(now); run_drain "$root" "$lab" plain "$EV/drain-$side-warm"; rc=$?; e=$(now)
  echo "warm second drain: exit=$rc wall=$(echo "$e - $s" | bc)s"
  rm -rf "$lab"
done
echo
echo "load at end: $(sysctl -n vm.loadavg)"
echo
echo "=== base vs target cold-drain stdout diff (rep1) ==="
diff "$EV/drain-base-rep1.stdout" "$EV/drain-target-rep1.stdout" && echo "IDENTICAL stdout"
echo "=== base vs target open-decision cursor digests (ident= device:inode:mtime line excluded) ==="
diff "$EV/drain-base-cursors.txt" "$EV/drain-target-cursors.txt" && echo "IDENTICAL cursor bytes ($(wc -l < "$EV/drain-target-cursors.txt" | tr -d ' ') files)"
echo "=== target cold drain stdout (rep1) ==="
cat "$EV/drain-target-rep1.stdout"
echo "=== target cold drain stderr (rep1) ==="
cat "$EV/drain-target-rep1.stderr"
} 2>&1 | tee "$EV/drain-compare-transcript.txt"
Evidence: Locked start run 1 measurement

Source: Locked start run 1 measurement

{
  "elapsed_seconds": 7.393,
  "limit_seconds": 20,
  "under_limit": true,
  "load_start": [
    33.07421875,
    61.58642578125,
    64.2216796875
  ],
  "load_end": [
    32.13671875,
    60.43408203125,
    63.78076171875
  ],
  "exit_code": 0,
  "lock_pid": "78562",
  "completion_pid": "78562",
  "lock_owner_process": "78562 78538 claude",
  "initial_state": {
    "status_logs": 25,
    "decision_checkpoints": 0,
    "presentation_manifest_exists": false
  },
  "decision_checkpoints_after": 25,
  "digest_complete": true,
  "truncated": false
}
Evidence: Locked start run 2 measurement

Source: Locked start run 2 measurement

{
  "elapsed_seconds": 9.684,
  "limit_seconds": 20,
  "under_limit": true,
  "load_start": [
    42.76904296875,
    60.08203125,
    63.52294921875
  ],
  "load_end": [
    45.43359375,
    60.0927734375,
    63.48681640625
  ],
  "exit_code": 0,
  "lock_pid": "85082",
  "completion_pid": "85082",
  "lock_owner_process": "85082 85073 claude",
  "initial_state": {
    "status_logs": 25,
    "decision_checkpoints": 0,
    "presentation_manifest_exists": false
  },
  "decision_checkpoints_after": 25,
  "digest_complete": true,
  "truncated": false
}
Evidence: Locked start run 3 measurement

Source: Locked start run 3 measurement

{
  "elapsed_seconds": 11.3,
  "limit_seconds": 20,
  "under_limit": true,
  "load_start": [
    47.7763671875,
    59.62841796875,
    63.23828125
  ],
  "load_end": [
    49.50439453125,
    59.62060546875,
    63.19384765625
  ],
  "exit_code": 0,
  "lock_pid": "51341",
  "completion_pid": "51341",
  "lock_owner_process": "51341 51331 claude",
  "initial_state": {
    "status_logs": 25,
    "decision_checkpoints": 0,
    "presentation_manifest_exists": false
  },
  "decision_checkpoints_after": 25,
  "digest_complete": true,
  "truncated": false
}
Evidence: Locked start run 1 digest

Source: Locked start run 1 digest


================================================================================
SESSION START - /tmp/fm-lab.b308SV
================================================================================

LOCK
--------------------------------------------------------------------------------
lock acquired: harness pid 78562

BOOTSTRAP
--------------------------------------------------------------------------------
BOOTSTRAP_INFO: lavish-axi >=0.1.80 enables confirmed board replies; this older compatible version retains the legacy reply path, but upgrade to prevent handing back a board before its reply is accepted

WAKE QUEUE
--------------------------------------------------------------------------------
(no queued wakes)
================================================================================
SUPERVISION OPERATING INSTRUCTIONS - primary harness: claude
================================================================================
Current state:
- Lock: held by this session; this session owns normal supervision unless away mode says otherwise.
- Away/quiet mode: inactive.
- X mode: inactive; use the default watcher cadence.
- Supervision host: on; it takes away-posture wakes and, where the dialog mirror is verified, eligible attended wakes itself, and hands the rest to you (protocol at the end of this block).
- Ordinary wake: the Stop-owned auto-arm (bin/fm-claude-stop-autoarm.sh) already owns watcher continuity; drain and handle the wake, and do not arm another cycle yourself.

Mode: Claude Stop-hook-owned supervision.

When this session owns supervision and away mode is not active:
1. Drain first with `bin/fm-wake-drain.sh`.
   After handling all emitted wakes and reconciling open decisions and unread status lines, run the exact `--ack-through` command printed as `WAKE_ACK_REQUIRED`; until then the work remains durable for idempotent re-handling after interruption.
2. Routine watcher arm and re-arm are owned by the Stop `asyncRewake` hook (`bin/fm-claude-stop-autoarm.sh`), never by you.
   Every turn end while supervision is needed launches or attaches one home-scoped watcher cycle with no model command and no model tokens.
   An actionable close wakes you through the hook's exit-2 rewake, delivered as a `Stop hook feedback` message.
3. On a `Stop hook feedback` wake (`signal:`, `stale:`, `check:`, or `heartbeat`), run `bin/fm-wake-drain.sh` first and handle the wake.
   Do not run `bin/fm-watch-arm.sh` after an ordinary wake; the next turn end re-arms automatically when supervision is still needed.
   Do not invent a wake from an attach-status line alone; drain and act only on real wake records, the drain's `OPEN DECISIONS` and `UNREAD STATUS` entries, or a real watcher reason line.
4. On the one `Stop hook feedback` automatic-mechanism failure notice (`firstmate watcher auto-arm FAILED ...`), drain, inspect the automatic mechanism failure, and do not turn the notice into a repeating manual-arm loop.
5. If the Stop hook does not claim the home or reports an exhausted failure, inspect its registration and watcher startup path before ending blind.
   Keep the Stop-owned automatic mechanism as the only Claude arm owner.
6. Treat `watcher: started ...` and `watcher: attached ...` inside automatic arm output as proof that one live cycle exists.
   On attach, the arm follows verified identity-matched successors instead of exiting when the first cycle ends.
7. The durable wake queue preserves actionable events between a rewake and the next Stop-launched arm, while the bounded turn-end guard prevents a blind Stop when recovery did not start.
   No PreToolUse hook denies fleet commands based on watcher status.
   [`watcher-continuity.md`](../watcher-continuity.md) owns the exact session-lock recovery boundary.
8. The turn-end guard (`bin/fm-turnend-guard.sh --claude`) remains the final backstop.
   It requires the PID-strict live-watcher and fresh-beacon predicate at the Stop boundary, except for the Claude-specific foreign-live-owner safe exit owned by [`turnend-guard.md`](../turnend-guard.md#guard-predicates); that document also owns the distinct model-aware mid-turn pull-guard rules.
   Otherwise, it allows the stop when a watcher is healthy or an open auto-arm generation claim owns recovery, while fresh failure epochs advance the bounded one-time attended fail-open progression described there.
9. Waiting on the hook-owned cycle is silent: do not send idle progress while the watcher is parked.

The watcher itself remains `bin/fm-watch.sh`, and `bin/fm-watch-arm.sh` remains the verified arm wrapper that the Stop hook foregrounds on a home that opted out of the [supervision host](../supervision-host.md) (`config/supervision-host-off`).
Re-arm attaches to an existing healthy cycle when one is already present and follows its verified successor chain.
See [`watcher-continuity.md`](../watcher-continuity.md) for the arm-layer successor and clean-close failure contract and the Claude ownership model.

Supervision host: on for this home (`config/supervision-host-off` turns it off; [`supervision-host.md`](../supervision-host.md) owns the design).
The Stop hook runs the supervision host in the arm's place, and everything above still holds with these additions:
1. Attended (no away record, including a quiet-mode record; see [Postures](../supervision-host.md#postures)): a headless supervision session takes the wakes the supervision branch may take and never wakes you for a routine outcome, so fewer wakes reach you; check wakes, decision wakes, and whatever it cannot take still reach you exactly as above.
   `supervision-host: branch-outcome: ...` means it handled a wake and recorded captain outcomes for you: run `bin/fm-wake-drain.sh`, process each entry of its `BRANCH OUTCOMES` section as firstmate from the task's current state, because each entry says how long ago it was recorded (tell the captain, land or merge what is ready, answer or escalate a decision, or act on a blocker; your reply covers only entries still open, as if a settled one, such as a PR since merged, had never been listed), then run the `mark-processed` acknowledgement it prints; every drain presents them again until you do.
   `supervision-host: the supervision session could not take this wake ...` means the wake is yours: handle it as above.
   A failing turn may include a `supervision-host:` health note about repeated engine errors: tell the captain when it matters and handle the handed-back wake as usual; during cooldown later attended closes reach you unchanged.
   Routine outcomes never wake you; your next drain lists only visible routine outcomes under `BRANCH OUTCOMES, ROUTINE` for awareness, with nothing to acknowledge. Silent rows do not appear there, but remain available through `bin/fm-branch-outcome.sh list`.
2. Away (an away record exists and no daemon runs): the host hands each wake to a headless away session that runs the supervision branch's contract under the record, and you are parked.
   Only a wake the host hands back reaches you, as `Stop hook feedback` carrying the close plus one `supervision-host: <why>` line.
   That wake is automatic supervision, not the captain's return: drain and handle it under the away posture, and never run the return from it.
   After the return, a `supervision-host:` line naming the captain's return during a turn means that turn has visible outcomes missing from the return brief, whether the wake was handled or handed back: relay every following `supervision-host: outcome ...` line to the captain (the rows also remain in `bin/fm-branch-outcome.sh list`), then drain and handle any queued wake before acknowledging.
   Each such visible outcome is also a queued `check: supervision-host outcome <n> ... was recorded after the captain returned` wake, which the drain presents until acknowledged: relay each outcome once, whichever arrives first, and acknowledge its `BRANCH OUTCOMES` entry too when it has one. Silent outcomes remain in the store but do not generate a handoff line or check wake.
3. `supervision-host: cycle boundary ...` means the host ended its park at its bound: run `bin/fm-wake-drain.sh`, handle whatever it presents, run it

... [3833 bytes truncated] ...

 /tmp/fm-lab.b308SV/state/task11.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task12 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task12.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task13 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task13.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task14 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task14.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task15 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task15.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task16 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task16.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task17 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task17.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task18 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task18.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task19 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task19.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task2 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task2.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task20 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task20.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task21 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task21.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task22 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task22.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task23 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task23.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task24 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task24.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task3 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task3.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task4 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task4.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task5 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task5.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task6 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task6.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task7 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task7.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task8 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task8.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

--- task9 ---
status tail (last 5 line(s), each capped at 220 characters, wake-EVENT history, not current state; full log: /tmp/fm-lab.b308SV/state/task9.status):
working: step 57
resolved [key=side-58]: routine close 58
working: step 58
resolved [key=side-59]: routine close 59
working: step 59

AFK
--------------------------------------------------------------------------------
absent

================================================================================
NETWORK CHECKS
================================================================================
completed off the startup path in 3s: GitHub authentication, dead-secondmate relaunch, secondmate convergence, pending handoff delivery, project clone refresh with its drift reporting, and inactive terminal-outcome reconciliation.
(silent - no problems found)

================================================================================
CONTEXT
================================================================================

data/projects.md
--------------------------------------------------------------------------------
ABSENT

data/secondmates.md
--------------------------------------------------------------------------------
ABSENT

data/captain.md
--------------------------------------------------------------------------------
ABSENT

data/captain-shared.md (shared, main-authoritative, read-only in secondmate homes)
--------------------------------------------------------------------------------
ABSENT

data/learnings.md
--------------------------------------------------------------------------------
ABSENT

================================================================================
NEXT STEP
================================================================================
Follow the supervision operating instructions block above for harness 'claude'.
This script never starts supervision itself.

The digest above is complete for this session start. The READ-ONCE CONTRACT
section near the top of it governs what may still be read from disk.

STDERR:
Evidence: Locked start driver

Source: Locked start driver

#!/bin/bash
# locked-start-driver.sh <label>: real Claude primary on a private fm-lab tmux
# socket with a disposable marked lab home seeded with 25 cold status logs
# (60 keyed resolved/working pairs each); its SessionStart hook runs the real
# locked session start. Tears the lab down in the same turn.
set -u
label=$1
EV=~/.no-mistakes/evidence/01M4H90MTFF5AXBWANAFDMBF3V
ROOT=$PWD
LAB=$(mktemp -d /tmp/fm-lab.XXXXXX); rmdir "$LAB"
bin/fm-lab-home.sh create "$LAB" >/dev/null || exit 1
mkdir -p "$LAB/tmux"
cleanup() { TMUX_TMPDIR="$LAB/tmux" tmux -L fm-lab kill-server 2>/dev/null; rm -rf "$LAB"; echo "teardown: private fm-lab server killed; lab home removed (exists now: $([ -e "$LAB" ] && echo yes || echo no))"; }
trap cleanup EXIT
for ((t = 0; t < 25; t++)); do
  { printf 'working: starting task %s\n' "$t"
    for ((i = 0; i < 60; i++)); do printf 'resolved [key=side-%s]: routine close %s\n' "$i" "$i"; printf 'working: step %s\n' "$i"; done
  } > "$LAB/state/task$t.status"
done
settings="$EV/$label-settings.json"
printf '{"hooks":{"SessionStart":[{"matcher":"startup","hooks":[{"type":"command","command":"python3 %s/startup-hook.py","timeout":180}]}]}}\n' "$EV" > "$settings"
CMD=(claude --setting-sources "" --settings "$settings" --strict-mcp-config --tools "" --no-session-persistence --max-budget-usd 0.10 -p "Reply exactly LAB_STARTUP_OK. Do not use tools.")
echo "launch: tmux -L fm-lab (TMUX_TMPDIR=<lab>/tmux) new-session -d -s primary -x 120 -y 40 -c <run worktree> -e FM_HOME=<lab> ${CMD[*]}"
env -u NO_MISTAKES_GATE -u FM_GATE_REFUSE_BYPASS -u FM_ROOT_OVERRIDE -u FM_STATE_OVERRIDE -u FM_DATA_OVERRIDE -u FM_CONFIG_OVERRIDE -u FM_PROJECTS_OVERRIDE -u TMUX -u TMUX_PANE \
  TMUX_TMPDIR="$LAB/tmux" tmux -L fm-lab new-session -d -s primary -x 120 -y 40 -c "$ROOT" \
  -e FM_HOME="$LAB" -e FM_LIVE_EVIDENCE="$EV" -e FM_LIVE_LABEL="$label" -e FM_LIVE_ROOT="$ROOT" \
  "${CMD[@]}" || exit 1
TMUX_TMPDIR="$LAB/tmux" tmux -L fm-lab set-option -t primary remain-on-exit on >/dev/null
TMUX_TMPDIR="$LAB/tmux" /usr/bin/perl -e 'alarm 240; exec @ARGV' tmux -L fm-lab wait-for lab-startup-finished
echo "wait-for exit=$?"
for ((k = 0; k < 60; k++)); do
  TMUX_TMPDIR="$LAB/tmux" tmux -L fm-lab capture-pane -p -t primary > "$EV/$label-pane.txt" 2>/dev/null
  grep -q 'LAB_STARTUP_OK\|Pane is dead' "$EV/$label-pane.txt" && break
  /bin/sleep 2
done
echo "pane:"; grep -v '^$' "$EV/$label-pane.txt"
echo "measurement:"; cat "$EV/$label-measurement.json"
Evidence: Snapshot race live transcript

Source: Snapshot race live transcript


=== prime: present bootstrap note ===
UNREAD STATUS (new since last drain, not re-printed after this presentation):
race note: bootstrap line
CHECK ok: contains 'bootstrap line'

=== drain 1: note A present; note B appended after the presentation snapshot (inside the drain) ===
UNREAD STATUS (new since last drain, not re-printed after this presentation):
race note: A included in presentation snapshot
CHECK ok: B was appended inside the drain, after the snapshot
CHECK ok: contains 'A included in presentation snapshot'
CHECK ok: does not contain 'B appended after presentation snapshot'

=== drain 2: plain real drain ===
UNREAD STATUS (new since last drain, not re-printed after this presentation):
race note: B appended after presentation snapshot
CHECK ok: contains 'B appended after presentation snapshot'
CHECK ok: does not contain 'A included in presentation snapshot'
CHECK ok: B shown exactly once

=== drain 3: nothing replayed ===
[]
CHECK ok: does not contain 'B appended'
CHECK ok: does not contain 'A included'

=== retained helpers still work after de-batching ===
lines:
race	note: C for helper check
snapshot:
race	note: C for helper check
CHECK ok: both helpers return the unread row

=== result ===
ALL CHECKS PASSED
Evidence: Headline numbers
Cold drain 25 logs/1500 keyed: base 36.7/38.5/34.1s -> target 3.9/4.0/3.8s; subshells 13571 -> 202; execs 488 -> 414; stdout + 25 cursor files identical. Real Claude locked start: 7.39s, 9.68s, 11.30s (limit 20s).

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-classify-lib.sh:703 - Simplification: after this change no production code calls scan_unread_surface_lines (fm-classify-lib.sh:700) or scan_unread_surface_snapshot (fm-classify-lib.sh:722). print_unread_status_section (bin/fm-wake-drain.sh:434) now prints rows that status_acknowledge_presented_snapshot prepares. Only tests call the helpers: tests/fm-wake-drain-unread-status.test.sh:268/272, tests/fm-parent-channel-scan-exclusion.test.sh:153/191 and tests/fm-spawn-dispatch-profile.test.sh:1247. The change still adds _FM_STATUS_STAT_BATCH scoping and _fm_status_stat_batch_into calls to both helpers (fm-classify-lib.sh:703-704 and 724-725). The intent (cut the drain's per-line or per-row process fan-out) does not need those lines. The helpers also become a parallel copy of the unread-surface row rule that now lives inline at fm-classify-lib.sh:604-609. As a result, test_snapshot_does_not_ack_a_later_append (tests/fm-wake-drain-unread-status.test.sh:268) checks the endpoint-bound invariant through the unused helper, not the production acknowledgement path. Recommended remedy: remove the added batching from these two helpers. Then decide whether to retire scan_unread_surface_snapshot, or point that test at the drain path.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 6 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Cold drain over 25 status logs (1,500 keyed resolutions) finishes much faster and with far fewer processes than base ✅ pass live drain-compare-transcript.txt: base 36.7/38.5/34.1 s, 13,571 subshell entries, 488 execs; target 3.9/4.0/3.8 s, 202 subshell entries, 414 execs
Drain output and open-decision cursor bytes are unchanged versus base ✅ pass live drain-compare-transcript.txt: 'IDENTICAL stdout' and 'IDENTICAL cursor bytes (25 files)'; drain-*-task0-open-decisions-cursor.bytes
Warm second drain on the same home stays fast ✅ pass live drain-compare-transcript.txt: warm base 20.3 s vs target 3.6 s
Real Claude locked session start over 25 cold logs completes in under 20 seconds ✅ pass live locked-start-run{1,2,3}-measurement.json: 7.393 s, 9.684 s, 11.300 s; lock and completion PID match the live claude process; digest complete, not truncated
Note appended after the presentation snapshot is not acked; the next drain shows it once and nothing replays (HEAD review fix) ✅ pass live snapshot-race-transcript.txt: ALL CHECKS PASSED; tests/fm-wake-drain-unread-status.test.sh 'presentation cursor advances only through its captured endpoint'
Retained helpers scan_unread_surface_lines and scan_unread_surface_snapshot still return unread rows after batching removal ✅ pass live snapshot-race-transcript.txt 'retained helpers still work'; fm-parent-channel-scan-exclusion.test.log
Manifest read failure aborts without replay or replacement receipts; fork counts stay flat as history and fleet grow ⏸️ untested no The prior payload did not establish a live result. This scenario was not driven separately in a disposable lab home in that run. Its only cover was the existing suite tests/fm-wake-drain-unread-status…
  • drain-compare-driver.sh &lt;base 924f479c archive&gt; &lt;worktree&gt; &lt;evidence&gt;: real bin/fm-wake-drain.sh on base and target, 25 cold logs x 60 keyed pairs, 3 cold reps each, DEBUG-trap subshell count, PATH-shim exec count, warm second drain, stdout and cursor-byte diff
  • locked-start-driver.sh locked-start-run{1,2,3}: real claude -p primary on private tmux -L fm-lab socket with marked lab home (bin/fm-lab-home.sh create), SessionStart hook runs bin/fm-session-start.sh --source startup over 25 cold logs; lab torn down each run
  • snapshot-race-driver.sh &lt;worktree&gt;: real drain in a lab home with a note appended after the presentation snapshot; then two plain drains; then direct calls to scan_unread_surface_lines and scan_unread_surface_snapshot
  • /bin/bash tests/fm-wake-drain-unread-status.test.sh (19/19 ok)
  • /bin/bash tests/fm-parent-channel-scan-exclusion.test.sh (ok)
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
1 warning still open:

  • ⚠️ linter found issues (exit code 143)
✅ **Push** - passed

✅ No issues found.

…ne status line

A drain over a 25-task fleet entered about 6,500 subshells per run, because
every presentation-cursor row read paid IFS=$(printf '\t'), every identity,
size, and mtime read paid its own stat plus a command substitution, and every
unread resolved line paid two more substitutions for its key and note.

- Use $'\t' for the field separator in the drain-path loops.
- Give the key, note, cursor-path, identity, size, offset, and unread-line
  readers an optional out-variable form and read the manifest without cat.
- Read identity, size, and mtime from one stat per file.

On a scratch copy of the main home the drain falls from 6,534 to 466 subshell
entries with byte-identical output. A new test counts subshell entries and
fails when they grow with unread history or grow faster than linearly with the
fleet.
…ne status line

A drain over a 25-task fleet entered about 6,500 subshells per run, because
every presentation-cursor row read paid IFS=$(printf '\t'), every identity,
size, and mtime read paid its own stat plus a command substitution, and every
unread resolved line paid two more substitutions for its key and note.

- Use $'\t' for the field separator in the drain-path loops.
- Give the key, note, cursor-path, identity, size, offset, and unread-line
  readers an optional out-variable form and read the manifest without cat.
- Read identity, size, and mtime from one stat per file.

On a scratch copy of the main home the drain falls from 6,534 to 466 subshell
entries with byte-identical output. A new test counts subshell entries and
fails when they grow with unread history or grow faster than linearly with the
fleet.
…ne status line

A drain over a 25-task fleet entered about 6,500 subshells per run, because
every presentation-cursor row read paid IFS=$(printf '\t'), every identity,
size, and mtime read paid its own stat plus a command substitution, and every
unread resolved line paid two more substitutions for its key and note.

- Use $'\t' for the field separator in the drain-path loops.
- Give the key, note, cursor-path, identity, size, offset, and unread-line
  readers an optional out-variable form and read the manifest without cat.
- Read identity, size, and mtime from one stat per file.

On a scratch copy of the main home the drain falls from 6,534 to 466 subshell
entries with byte-identical output. A new test counts subshell entries and
fails when they grow with unread history or grow faster than linearly with the
fleet.
…o branch

Ordinary merge of refs/no-mistakes/recover/01M4F9B87M5A8ZB6B0YDWF5A7V.
Conflicts in six wake-drain files resolved to the pipeline side, which
carries the same fork-free drain fix rebased onto newer main plus the
review, test and document fix commits. Resulting tree equals 9bdbde2.
The five pipeline fix commits in 8f805fc are already present on this
branch as rebased equivalents with identical changed lines (dbf15ef,
fe5805b, e2af22e, a1ba6a0, 19e75d4). This ordinary merge only makes
the pushed head an ancestor so the branch is no longer diverged.
@MrGTV-love MrGTV-love changed the title fix: reduce wake drain subprocess overhead fix(bin): stop the wake drain forking a process per status line, stat, and cursor row Oct 10, 2026
@MrGTV-love
MrGTV-love merged commit 070aece into main Oct 10, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant