fix(bin): bind crew state to run heads that exist only in the gate repo

[Kallas95]

What Changed

• bin/fm-crew-state.sh: collapsed the duplicated nm_run_head_matches_worktree / nm_coarse_head_matches_worktree pair into a single nm_head_matches_worktree <run-head-rev> (the axi status variant now just reads the TOON head field and delegates). When the task worktree cannot resolve the run head, resolution falls back to the no-mistakes remote's gate repo — accepting both a bare path and the same path in file:// form — and applies the equality/ancestry test there. A live run whose head is a pipeline gate-fix commit now binds instead of falling through to a stale prior run on the same branch; diverged heads and locally-advanced worktrees still fail attribution.
• bin/fm-test-run.sh: now_ms now prefers perl -MTime::HiRes (~10ms) over a python3 that may resolve through a version-manager shim (~200ms), and each interpreter only wins if it actually prints a value — a perl without Time::HiRes or a stub python3 falls through to the next source and finally to second-precision date, rather than aborting the whole runner under set -eu.
• Tests: five new fm-crew-state cases build the real topology (throwaway worktree + real bare gate repo + gate-only commit) to cover run-step binding, the coarse runs-list fallback, a file:// remote, and the two non-attribution controls; fm-test-run adds test_timing_survives_broken_interpreters and lengthens the jobs-scheduler slow fixture from 0.5s to 2s for slot-refill headroom on loaded hosts; the fm-calm-pi missing-adapter-export subtest now probes node's native TypeScript type stripping and skips when unavailable instead of failing with ERR_UNKNOWN_FILE_EXTENSION.

Risk Assessment

✅ Low: The final round is a four-line test-diagnostic ordering fix that adopts the file's existing evidence-before-cleanup idiom verbatim, and every prior finding across the branch is now resolved with the two source changes verified behavior-correct on all paths and covered by new regression tests.

Testing

Ran the three suites touched by the range (fm-crew-state, fm-test-run, fm-calm-pi-extension) plus the project's conservative changed-file lane of 25 scripts, then produced product-level CLI evidence beyond pass/fail: an end-to-end repro that builds a real crew worktree and bare gate repo with a gate-only fix commit and runs the crew-state helper from both the base and target commits (failed -> working), including negative controls, a file:// remote variant, and the watcher absorb decision; a before/after transcript of the jobs-scheduler timing test under a slow python3 shim (deterministic pre-fix failure, 3/3 passes shipped) and of the timing test's failure diagnostics; and both directions of the calm-pi node capability gate. All targeted suites pass. The changed lane showed one unrelated red, fm-kimi-harness, caused by the host default python3 3.9.6 lacking tomllib; the Kimi files are untouched by this range and that suite passes once a tomllib-capable python3 is on PATH. No UI surface is involved, so evidence is CLI transcripts rather than screenshots. I also added a regression test for the fix's previously uncovered file:// gate-remote path; the working tree otherwise carries no transient artifacts.

Evidence: Gate-only run head: live run read as failed before, working now (full CLI transcript incl. negative controls, file:// remote, watcher decision)

=== firstmate reads the crew state === $ bin/fm-crew-state.sh incident # BASE a53ffc1 (before the fix) state: failed · source: run-step · run failed $ bin/fm-crew-state.sh incident # THIS BRANCH (gate-repo head resolution) state: working · source: run-step · validating (running) === Downstream: watcher absorb decision for the live gate-fix crew === # BASE a53ffc1 absorb class : none provably working: no -> the wake SURFACES this crew as no longer working # THIS BRANCH absorb class : working provably working: yes -> a stale/no-verb wake is ABSORBED (crew left alone)

=== Topology (real git repos) ===
crew worktree      : /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//gate-head-e2e.iKgSnO/incident/wt
  branch           : fm/feat-widget
  HEAD             : c81b85d  feat: widget
  no-mistakes remote: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//gate-head-e2e.iKgSnO/incident/gate.git
bare gate repo     : /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//gate-head-e2e.iKgSnO/incident/gate.git
  fm/feat-widget : e6e6af1  no-mistakes(review): address gate findings

Is the live run head an object in the crew worktree?
$ git -C <worktree> cat-file -e e6e6af1^{commit}
  -> NO: "fatal: Not a valid object name" - it lives only in the gate repo

------------------------------------------------------------------------
=== What the pipeline reports (fake no-mistakes CLI, real surface) ===
$ no-mistakes axi status
run:
  id: "01RUNLIVE"
  branch: fm/feat-widget
  status: running
  head: "e6e6af13337e9ffef9522120748623ff89bc6eed"
  pr: ""
  findings: none
  steps[2]{step,status,findings,duration_ms}:
    intent,completed,0,0
    review,running,0,0

$ no-mistakes runs --limit 200
  running    fm/feat-widget e6e6af1  2026-07-25 10:10
  failed     fm/feat-widget c81b85d  2026-07-25 09:00

------------------------------------------------------------------------
=== firstmate reads the crew state ===

$ bin/fm-crew-state.sh incident   # BASE a53ffc1 (before the fix)
state: failed · source: run-step · run failed

$ bin/fm-crew-state.sh incident   # THIS BRANCH (gate-repo head resolution)
state: working · source: run-step · validating (running)

verdict: REPRODUCED and FIXED - the live run read as failed before, reads working now

------------------------------------------------------------------------
=== Negative control A: diverged gate-only head must NOT be attributed ===
gate-repo run head : b6d12a9  (rewritten history, no shared ancestor)
shared ancestor with the worktree HEAD? none (diverged)

$ bin/fm-crew-state.sh diverged   # BASE a53ffc1
state: unknown · source: none · no current-state source available

$ bin/fm-crew-state.sh diverged   # THIS BRANCH
state: unknown · source: none · no current-state source available
verdict: rejected as intended (no run-step attribution; falls back to pane/status-log)

------------------------------------------------------------------------
=== Negative control B: worktree advanced past a gate-only run head ===
run head (gate-only): c16876d  (no-mistakes(review): fix from an EARLIER run)
worktree HEAD       : 8e6ca0c  (local work after that run ended)

$ bin/fm-crew-state.sh advanced   # BASE a53ffc1
state: unknown · source: none · no current-state source available

$ bin/fm-crew-state.sh advanced   # THIS BRANCH
state: unknown · source: none · no current-state source available
verdict: rejected as intended (local work invalidates attribution)

------------------------------------------------------------------------
=== file:// gate remote: the same binding must hold ===
no-mistakes remote : file:///var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//gate-head-e2e.iKgSnO/fileurl/gate.git
run head (gate-only): 7a62477

$ bin/fm-crew-state.sh fileurl   # BASE a53ffc1
state: working · source: status-log · handed to the no-mistakes gate

$ bin/fm-crew-state.sh fileurl   # THIS BRANCH
state: working · source: run-step · validating (running)
verdict: file:// remote resolves too - live run bound

------------------------------------------------------------------------
=== Downstream: watcher absorb decision for the live gate-fix crew ===

# BASE a53ffc1
crew state line : state: failed · source: run-step · run failed
absorb class    : none
provably working: no  -> the wake SURFACES this crew as no longer working

# THIS BRANCH
crew state line : state: working · source: run-step · validating (running)
absorb class    : working
provably working: yes -> a stale/no-verb wake is ABSORBED (crew left alone)

------------------------------------------------------------------------
scratch topology kept at: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//gate-head-e2e.iKgSnO

Evidence: E2E harness that produced the gate-head transcript (real worktree + bare gate repo + gate-only fix commit)

#!/usr/bin/env bash
# End-to-end demonstration of the 2026-07-25 incident and its fix:
# a LIVE no-mistakes run whose head is a pipeline gate-fix commit that exists
# ONLY in the bare gate repo (the worktree's `no-mistakes` remote) must be
# attributed to the crew (state: working), not fall through to a stale prior
# failed run on the same branch (state: failed).
#
# Builds the real topology (task worktree + real bare gate repo seeded by a
# push + a gate-only fix commit), serves the real command surface firstmate
# reads through a fake `no-mistakes` CLI, then runs the SAME crew-state helper
# twice: as shipped on this branch, and as it was at the base commit.
set -u

REPO=$1
BASE_REV=$2
WORK=$(mktemp -d "${TMPDIR:-/tmp}/gate-head-e2e.XXXXXX")

export GIT_AUTHOR_NAME=fmtest GIT_AUTHOR_EMAIL=fmtest@example.invalid
export GIT_COMMITTER_NAME=fmtest GIT_COMMITTER_EMAIL=fmtest@example.invalid

# The helper as it exists at the base commit, next to the current libs it
# sources (only fm-crew-state.sh changed in bin/ for this behavior).
BASE_BIN="$WORK/bin-base"
cp -R "$REPO/bin" "$BASE_BIN"
git -C "$REPO" show "$BASE_REV:bin/fm-crew-state.sh" >"$BASE_BIN/fm-crew-state.sh"

FAKEBIN="$WORK/fakebin"
mkdir -p "$FAKEBIN"
cat >"$WORK/no-mistakes.src" <<'SH'
#!/usr/bin/env bash
# Fake no-mistakes serving the exact surface fm-crew-state.sh reads.
set -u
case "${1:-}" in
  axi)
    shift
    [ "${1:-}" = status ] && printf '%s\n' "${FAKE_AXI_STATUS:-}" ;;
  runs)
    printf '%s\n' "${FAKE_RUNS_LIST:-}" ;;
esac
exit 0
SH
cat >"$WORK/tmux.src" <<'SH'
#!/usr/bin/env bash
set -u
case "${1:-}" in
  display-message) printf '%%1\n' ;;
  capture-pane)    printf 'all quiet\n> \n' ;;
esac
exit 0
SH
install -m 755 "$WORK/no-mistakes.src" "$FAKEBIN/no-mistakes"
install -m 755 "$WORK/tmux.src" "$FAKEBIN/tmux"

# A task worktree on <branch>, a bare gate repo wired as its `no-mistakes`
# remote and seeded by the run's initial push, plus state/<id>.meta. The
# status log is written only for the incident case; the negative controls stay
# log-free so a rejected run reads as an unmistakable "unknown / none".
build_case() {  # <case> <branch> [with-status-log]
  local case=$1 branch=$2
  mkdir -p "$WORK/$case/state"
  git init -q "$WORK/$case/wt"
  printf 'widget\n' >"$WORK/$case/wt/widget.txt"
  git -C "$WORK/$case/wt" add -A
  git -C "$WORK/$case/wt" commit -q -m 'feat: widget'
  git -C "$WORK/$case/wt" checkout -q -b "$branch"
  git init -q --bare "$WORK/$case/gate.git"
  git -C "$WORK/$case/wt" remote add no-mistakes "$WORK/$case/gate.git"
  git -C "$WORK/$case/wt" push -q no-mistakes "$branch"
  printf 'window=fm:fm-%s\nworktree=%s\nkind=ship\n' \
    "$case" "$WORK/$case/wt" >"$WORK/$case/state/$case.meta"
  [ "${3:-}" = with-status-log ] \
    && printf 'working: handed to the no-mistakes gate\n' >"$WORK/$case/state/$case.status"
  return 0
}

# A commit created directly in the bare gate repo, never fetched by the
# worktree: the shape of a pipeline gate-fix commit. Echoes its sha.
make_gate_only_commit() {  # <case> <branch> <parent> <message>
  local case=$1 branch=$2 parent=$3 msg=$4 tree sha
  tree=$(git -C "$WORK/$case/gate.git" rev-parse "${parent}^{tree}")
  sha=$(git -C "$WORK/$case/gate.git" commit-tree "$tree" -p "$parent" -m "$msg")
  git -C "$WORK/$case/gate.git" update-ref "refs/heads/$branch" "$sha"
  printf '%s\n' "$sha"
}

read_state() {  # <bin-dir> <case> <id>
  PATH="$FAKEBIN:$PATH" FM_STATE_OVERRIDE="$WORK/$2/state" \
    "$1/fm-crew-state.sh" "$3"
}

hr() { printf '\n%s\n' "------------------------------------------------------------------------"; }

# ===========================================================================
# CASE 1 - the incident: live run whose head is a gate-only fix commit, with an
#          older FAILED run on the same branch still sitting at the worktree HEAD
# ===========================================================================
BRANCH=fm/feat-widget
build_case incident "$BRANCH" with-status-log
WT="$WORK/incident/wt"
GATE="$WORK/incident/gate.git"
BASE_HEAD=$(git -C "$WT" rev-parse HEAD)
FIX_HEAD=$(make_gate_only_commit incident "$BRANCH" "$BASE_HEAD" 'no-mistakes(review): address gate findings')

printf '=== Topology (real git repos) ===\n'
printf 'crew worktree      : %s\n' "$WT"
printf '  branch           : %s\n' "$(git -C "$WT" symbolic-ref --short HEAD)"
printf '  HEAD             : %s  %s\n' "$(git -C "$WT" rev-parse --short HEAD)" \
  "$(git -C "$WT" log -1 --format=%s)"
printf '  no-mistakes remote: %s\n' "$(git -C "$WT" remote get-url no-mistakes)"
printf 'bare gate repo     : %s\n' "$GATE"
printf '  %s : %s  %s\n' "$BRANCH" "$(git -C "$GATE" rev-parse --short "$FIX_HEAD")" \
  "$(git -C "$GATE" log -1 --format=%s "$FIX_HEAD")"
printf '\nIs the live run head an object in the crew worktree?\n'
printf '$ git -C <worktree> cat-file -e %s^{commit}\n' "$(git -C "$GATE" rev-parse --short "$FIX_HEAD")"
if git -C "$WT" cat-file -e "${FIX_HEAD}^{commit}" 2>/dev/null; then
  printf '  -> yes (FIXTURE LEAK - this case would not reproduce the incident)\n'
else
  printf '  -> NO: "fatal: Not a valid object name" - it lives only in the gate repo\n'
fi

FIX_SHORT=$(git -C "$GATE" rev-parse --short=7 "$FIX_HEAD")
WT_SHORT=$(git -C "$WT" rev-parse --short=7 HEAD)
export FAKE_AXI_STATUS="run:
  id: \"01RUNLIVE\"
  branch: $BRANCH
  status: running
  head: \"$FIX_HEAD\"
  pr: \"\"
  findings: none
  steps[2]{step,status,findings,duration_ms}:
    intent,completed,0,0
    review,running,0,0"
export FAKE_RUNS_LIST="  running    $BRANCH $FIX_SHORT  2026-07-25 10:10
  failed     $BRANCH $WT_SHORT  2026-07-25 09:00"

hr
printf '=== What the pipeline reports (fake no-mistakes CLI, real surface) ===\n'
printf '$ no-mistakes axi status\n%s\n' "$FAKE_AXI_STATUS"
printf '\n$ no-mistakes runs --limit 200\n%s\n' "$FAKE_RUNS_LIST"

hr
printf '=== firstmate reads the crew state ===\n'
printf '\n$ bin/fm-crew-state.sh incident   # BASE %s (before the fix)\n' "${BASE_REV:0:7}"
BASE_OUT=$(read_state "$BASE_BIN" incident incident)
printf '%s\n' "$BASE_OUT"
printf '\n$ bin/fm-crew-state.sh incident   # THIS BRANCH (gate-repo head resolution)\n'
FIXED_OUT=$(read_state "$REPO/bin" incident incident)
printf '%s\n' "$FIXED_OUT"

printf '\nverdict: '
case "$BASE_OUT|$FIXED_OUT" in
  *"state: failed"*"|"*"state: working"*)
    printf 'REPRODUCED and FIXED - the live run read as failed before, reads working now\n' ;;
  *) printf 'UNEXPECTED - see the two lines above\n' ;;
esac

# ===========================================================================
# CASE 2 - negative control: a DIVERGED head that also exists only in the gate
#          repo must still be rejected (the widening must not weaken this)
# ===========================================================================
hr
printf '=== Negative control A: diverged gate-only head must NOT be attributed ===\n'
BRANCH2=fm/feat-rewritten
build_case diverged "$BRANCH2"
WT2="$WORK/diverged/wt"
GATE2="$WORK/diverged/gate.git"
ORPHAN_TREE=$(git -C "$GATE2" rev-parse "$(git -C "$GATE2" rev-parse "refs/heads/$BRANCH2")^{tree}")
ORPHAN=$(git -C "$GATE2" commit-tree "$ORPHAN_TREE" -m 'rewritten history, no shared ancestor')
git -C "$GATE2" update-ref "refs/heads/$BRANCH2" "$ORPHAN"
printf 'gate-repo run head : %s  (%s)\n' "$(git -C "$GATE2" rev-parse --short "$ORPHAN")" \
  "$(git -C "$GATE2" log -1 --format=%s "$ORPHAN")"
printf 'shared ancestor with the worktree HEAD? '
if git -C "$GATE2" merge-base "$(git -C "$WT2" rev-parse HEAD)" "$ORPHAN" >/dev/null 2>&1; then
  printf 'yes\n'
else
  printf 'none (diverged)\n'
fi
export FAKE_AXI_STATUS="run:
  id: \"01RUNOLD\"
  branch: $BRANCH2
  status: running
  head: \"$ORPHAN\"
  pr: \"\"
  findings: none
  steps[1]{step,status,findings,duration_ms}:
    review,running,0,0"
export FAKE_RUNS_LIST="  running    $BRANCH2 $(git -C "$GATE2" rev-parse --short=7 "$ORPHAN")  2026-07-25 10:10"
printf '\n$ bin/fm-crew-state.sh diverged   # BASE %s\n' "${BASE_REV:0:7}"
printf '%s\n' "$(read_state "$BASE_BIN" diverged diverged)"
printf '\n$ bin/fm-crew-state.sh diverged   # THIS BRANCH\n'
DIV_OUT=$(read_state "$REPO/bin" diverged diverged)
printf '%s\n' "$DIV_OUT"
printf 'verdict: '
case "$DIV_OUT" in
  *"source: run-step"*) printf 'REGRESSION - a diverged run was attributed\n' ;;
  *) printf 'rejected as intended (no run-step attribution; falls back to pane/status-log)\n' ;;
esac

# ===========================================================================
# CASE 3 - negative control: local work advanced PAST a gate-only run head
# ===========================================================================
hr
printf '=== Negative control B: worktree advanced past a gate-only run head ===\n'
BRANCH3=fm/feat-advanced
build_case advanced "$BRANCH3"
WT3="$WORK/advanced/wt"
GATE3="$WORK/advanced/gate.git"
RUN_HEAD3=$(make_gate_only_commit advanced "$BRANCH3" "$(git -C "$WT3" rev-parse HEAD)" \
  'no-mistakes(review): fix from an EARLIER run')
printf 'more\n' >>"$WT3/widget.txt"
git -C "$WT3" commit -q -am 'local work after that run ended'
printf 'run head (gate-only): %s  (%s)\n' "$(git -C "$GATE3" rev-parse --short "$RUN_HEAD3")" \
  "$(git -C "$GATE3" log -1 --format=%s "$RUN_HEAD3")"
printf 'worktree HEAD       : %s  (%s)\n' "$(git -C "$WT3" rev-parse --short HEAD)" \
  "$(git -C "$WT3" log -1 --format=%s)"
export FAKE_AXI_STATUS="run:
  id: \"01RUNSTALE\"
  branch: $BRANCH3
  status: running
  head: \"$RUN_HEAD3\"
  pr: \"\"
  findings: none
  steps[1]{step,status,findings,duration_ms}:
    review,running,0,0"
export FAKE_RUNS_LIST="  running    $BRANCH3 $(git -C "$GATE3" rev-parse --short=7 "$RUN_HEAD3")  2026-07-25 10:10"
printf '\n$ bin/fm-crew-state.sh advanced   # BASE %s\n' "${BASE_REV:0:7}"
printf '%s\n' "$(read_state "$BASE_BIN" advanced advanced)"
printf '\n$ bin/fm-crew-state.sh advanced   # THIS BRANCH\n'
ADV_OUT=$(read_state "$REPO/bin" advanced advanced)
printf '%s\n' "$ADV_OUT"
printf 'verdict: '
case "$ADV_OUT" in
  *"source: run-step"*) printf 'REGRESSION - a superseded run was attributed\n' ;;
  *) printf 'rejected as intended (local work invalidates attribution)\n' ;;
esac

# ===========================================================================
# CASE 4 - the same live run when the gate remote is wired as a file:// URL
#          (the other shape `git remote get-url` can return for a local repo)
# ===========================================================================
hr
printf '=== file:// gate remote: the same binding must hold ===\n'
BRANCH4=fm/feat-fileurl
build_case fileurl "$BRANCH4" with-status-log
WT4="$WORK/fileurl/wt"
GATE4="$WORK/fileurl/gate.git"
git -C "$WT4" remote set-url no-mistakes "file://$GATE4"
FIX4=$(make_gate_only_commit fileurl "$BRANCH4" "$(git -C "$WT4" rev-parse HEAD)" \
  'no-mistakes(review): address gate findings')
printf 'no-mistakes remote : %s\n' "$(git -C "$WT4" remote get-url no-mistakes)"
printf 'run head (gate-only): %s\n' "$(git -C "$GATE4" rev-parse --short "$FIX4")"
export FAKE_AXI_STATUS="run:
  id: \"01RUNFILEURL\"
  branch: $BRANCH4
  status: running
  head: \"$FIX4\"
  pr: \"\"
  findings: none
  steps[1]{step,status,findings,duration_ms}:
    review,running,0,0"
export FAKE_RUNS_LIST="  running    $BRANCH4 $(git -C "$GATE4" rev-parse --short=7 "$FIX4")  2026-07-25 10:10"
printf '\n$ bin/fm-crew-state.sh fileurl   # BASE %s\n' "${BASE_REV:0:7}"
printf '%s\n' "$(read_state "$BASE_BIN" fileurl fileurl)"
printf '\n$ bin/fm-crew-state.sh fileurl   # THIS BRANCH\n'
URL_OUT=$(read_state "$REPO/bin" fileurl fileurl)
printf '%s\n' "$URL_OUT"
printf 'verdict: '
case "$URL_OUT" in
  *"source: run-step"*) printf 'file:// remote resolves too - live run bound\n' ;;
  *) printf 'NOT BOUND - a file:// gate remote is not resolved\n' ;;
esac

# ===========================================================================
# The captain-visible consequence: what the always-on watcher decides for the
# incident crew (bin/fm-watch.sh -> crew_is_provably_working -> this helper).
# ===========================================================================
hr
printf '=== Downstream: watcher absorb decision for the live gate-fix crew ===\n'
export FAKE_AXI_STATUS="run:
  id: \"01RUNLIVE\"
  branch: $BRANCH
  status: running
  head: \"$FIX_HEAD\"
  pr: \"\"
  findings: none
  steps[2]{step,status,findings,duration_ms}:
    intent,completed,0,0
    review,running,0,0"
export FAKE_RUNS_LIST="  running    $BRANCH $FIX_SHORT  2026-07-25 10:10
  failed     $BRANCH $WT_SHORT  2026-07-25 09:00"

watcher_verdict() {  # <helper-bin>
  # shellcheck source=/dev/null
  ( . "$REPO/bin/fm-classify-lib.sh"
    export FM_CREW_STATE_BIN="$1/fm-crew-state.sh"
    export PATH="$FAKEBIN:$PATH"
    export FM_STATE_OVERRIDE="$WORK/incident/state"
    printf 'crew state line : %s\n' "$("$FM_CREW_STATE_BIN" incident)"
    printf 'absorb class    : %s\n' "$(crew_absorb_class incident)"
    if crew_is_provably_working incident; then
      printf 'provably working: yes -> a stale/no-verb wake is ABSORBED (crew left alone)\n'
    else
      printf 'provably working: no  -> the wake SURFACES this crew as no longer working\n'
    fi )
}
printf '\n# BASE %s\n' "${BASE_REV:0:7}"
watcher_verdict "$BASE_BIN"
printf '\n# THIS BRANCH\n'
watcher_verdict "$REPO/bin"

hr
printf 'scratch topology kept at: %s\n' "$WORK"

Evidence: Scheduler timing flake reproduced pre-fix and closed by the shipped pair, plus failure-diagnostics comparison

per-call clock cost on this host: perl -MTime::HiRes : 24ms real python3: 73ms python3 via shim: 630ms old slot-refill margin: 450ms (0.5s slow fixture) BASE test file + BASE runner -> not ok - scheduler waited for oldest worker FAIL BASE test file + THIS BRANCH runner -> PASS / FAIL / PASS (0.5s margin still marginal) THIS test file (2s) + BASE runner -> PASS SHIPPED pair, repeats 1-3/3 -> PASS, PASS, PASS BEFORE cleanup fix: not ok - a broken perl and python3 must not abort the runner: (message empty, temp already removed) SHIPPED: fm-test-run.sh: line 123: * 1000: syntax error: operand expected (runner evidence printed)

==============================================================================
PART 1 - the flake: jobs-scheduler slot refill on a host with a slow python3
==============================================================================

per-call clock cost on this host:
  perl -MTime::HiRes  : 24ms (3 calls, best of)
  real python3        : 73ms
  python3 via shim    : 630ms  <- the flaking host; now_ms ran twice per fixture
  old slot-refill margin: 450ms (0.5s slow fixture)

--- BASE test file (0.5s fixture) + BASE runner (python3 now_ms) == reported flake ---
FM_TEST_BEGIN 2026-07-30T10:53:12Z tests/fm-lint.test.sh family=pure-contract-unit expected_gate_skip=none
ok - slow fixture
FM_TEST_END 2026-07-30T10:53:12Z tests/fm-brief.test.sh exit=0 duration_ms=979 gate_skip=false
not ok - scheduler waited for oldest worker
FM_TEST_END 2026-07-30T10:53:13Z tests/fm-lint.test.sh exit=1 duration_ms=454 gate_skip=false
FM_TEST_SUMMARY total=3 failed=1 skipped_gate=0 duration_ms=3072
FM_TEST_SUMMARY_FAMILY family=pure-contract-unit count=3 duration_ms=2101 failed=1
FM_TEST_SLOWEST rank=1 script=tests/fm-brief.test.sh duration_ms=979
FM_TEST_SLOWEST rank=2 script=tests/fm-composer-lib.test.sh duration_ms=668
FM_TEST_SLOWEST rank=3 script=tests/fm-lint.test.sh duration_ms=454
fm-test-run: wrote timing artifact: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//fm-test-run-jobs-sched.SVtqWb/timing.json
not ok - jobs=2 must refill the first completed slot
subtest exit=1  wall=4414ms  verdict=FAIL

--- BASE test file (0.5s fixture) + THIS BRANCH runner (perl now_ms only), repeat 1/3 ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=3983ms  verdict=PASS

--- BASE test file (0.5s fixture) + THIS BRANCH runner (perl now_ms only), repeat 2/3 ---
FM_TEST_SLOWEST rank=2 script=tests/fm-composer-lib.test.sh duration_ms=234
FM_TEST_SLOWEST rank=3 script=tests/fm-lint.test.sh duration_ms=18
fm-test-run: wrote timing artifact: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//fm-test-run-jobs-sched.HGY83G/timing.json
not ok - jobs=2 must refill the first completed slot
subtest exit=1  wall=1816ms  verdict=FAIL

--- BASE test file (0.5s fixture) + THIS BRANCH runner (perl now_ms only), repeat 3/3 ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=4039ms  verdict=PASS

--- THIS BRANCH test file (2s fixture) + BASE runner ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=14213ms  verdict=PASS

--- SHIPPED: this branch test file + this branch runner, repeat 1/3 ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=6910ms  verdict=PASS

--- SHIPPED: this branch test file + this branch runner, repeat 2/3 ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=6960ms  verdict=PASS

--- SHIPPED: this branch test file + this branch runner, repeat 3/3 ---
ok - jobs scheduler runs proven scripts; failure propagates; non-proven refused
subtest exit=0  wall=7112ms  verdict=PASS

==============================================================================
PART 2 - failure diagnostics of the interpreter-fallthrough timing test
         (a stub `date` forces the failure path; nothing else is changed)
==============================================================================

--- BEFORE 3f1e36e+cleanup fix: temp dir removed before the message read it ---
cat: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//fm-test-run-clock.rX9fsw/err2.txt: No such file or directory
not ok - a broken perl and python3 must not abort the runner: 
subtest exit=1  wall=1337ms  verdict=FAIL

--- SHIPPED: runner stdout/stderr printed before cleanup ---
/Users/mfa/.no-mistakes/worktrees/e9bf2055a1e0/01KYS8FN38MBN0BV135A2QCGZB/bin/fm-test-run.sh: line 123: * 1000: syntax error: operand expected (error token is "* 1000")
not ok - a broken perl and python3 must not abort the runner
subtest exit=1  wall=1136ms  verdict=FAIL

scratch kept at: /var/folders/9y/4s49r60d6fz1763bfg1x5bsw0000gp/T//runner-timing.ivqfsB

Evidence: Runner-timing evidence harness and the single-subtest driver

#!/usr/bin/env bash
# Evidence for the two runner-timing commits in this range:
#   003a362 deflake the jobs scheduler timing test (perl now_ms + 2s fixture)
#   c0d39f8 print runner evidence before temp cleanup in the timing test
#
# Part 1 reproduces the reported flake on a host whose python3 resolves through
# a version-manager shim, then isolates which half of the fix closes it: the
# faster clock alone, the widened fixture alone, or the shipped pair.
# Part 2 forces the interpreter-fallthrough timing test to fail and compares the
# diagnostics it leaves behind, before and after the cleanup-ordering fix.
#
# usage: runner-timing-evidence.sh <repo-root> <base-rev> <pre-cleanup-fix-rev>
set -u

REPO=$1
BASE_REV=$2
MID_REV=$3
HERE=$(cd "$(dirname "$0")" && pwd)
W=$(mktemp -d "${TMPDIR:-/tmp}/runner-timing.XXXXXX")

# A functional python3 that costs what a pyenv shim costs (~200ms + startup).
mkdir -p "$W/slowshim" "$W/nodate"
printf '#!/usr/bin/env bash\n/bin/sleep 0.2\nexec /usr/bin/python3 "$@"\n' >"$W/python3.src"
install -m 755 "$W/python3.src" "$W/slowshim/python3"
# A `date` that answers nothing, used ONLY to force the timing test's failure
# path so the diagnostics it prints can be compared.
printf '#!/usr/bin/env bash\nexit 0\n' >"$W/date.src"
install -m 755 "$W/date.src" "$W/nodate/date"

git -C "$REPO" show "$BASE_REV:bin/fm-test-run.sh" >"$W/runner-base.src"
install -m 755 "$W/runner-base.src" "$W/runner-base.sh"
git -C "$REPO" show "$BASE_REV:tests/fm-test-run.test.sh" >"$W/testfile-base.sh"
git -C "$REPO" show "$MID_REV:tests/fm-test-run.test.sh" >"$W/testfile-precleanup.sh"

now_ms() { perl -MTime::HiRes=time -e 'printf("%d\n", time() * 1000)'; }

run_case() {  # <label> <extra-path-dir> <test-file> <runner> <fn> <tail-lines>
  local label=$1 pathdir=$2 testfile=$3 runner=$4 fn=$5 tail_n=$6 rc t0 t1
  printf '\n--- %s ---\n' "$label"
  t0=$(now_ms)
  set +e
  PATH="$pathdir:$PATH" bash "$HERE/sched-timing-driver.sh" \
    "$REPO" "$testfile" "$runner" "$fn" >"$W/out.txt" 2>&1
  rc=$?
  set -e
  t1=$(now_ms)
  tail -n "$tail_n" "$W/out.txt"
  printf 'subtest exit=%s  wall=%sms  verdict=%s\n' "$rc" "$((t1 - t0))" \
    "$([ "$rc" -eq 0 ] && echo PASS || echo FAIL)"
}

FN_SCHED=test_jobs_parallel_scheduler_and_failure_propagation
FN_CLOCK=test_timing_survives_broken_interpreters

echo "=============================================================================="
echo "PART 1 - the flake: jobs-scheduler slot refill on a host with a slow python3"
echo "=============================================================================="
printf '\nper-call clock cost on this host:\n'
printf '  perl -MTime::HiRes  : %sms (3 calls, best of)\n' \
  "$( { t0=$(now_ms); perl -MTime::HiRes=time -e 'printf("%d\n", time()*1000)' >/dev/null; t1=$(now_ms); echo $((t1-t0)); } )"
printf '  real python3        : %sms\n' \
  "$( { t0=$(now_ms); /usr/bin/python3 -c 'import time; print(int(time.time()*1000))' >/dev/null; t1=$(now_ms); echo $((t1-t0)); } )"
printf '  python3 via shim    : %sms  <- the flaking host; now_ms ran twice per fixture\n' \
  "$( { t0=$(now_ms); PATH="$W/slowshim:$PATH" python3 -c 'import time; print(int(time.time()*1000))' >/dev/null; t1=$(now_ms); echo $((t1-t0)); } )"
printf '  old slot-refill margin: 450ms (0.5s slow fixture)\n'

run_case "BASE test file (0.5s fixture) + BASE runner (python3 now_ms) == reported flake" \
  "$W/slowshim" "$W/testfile-base.sh" "$W/runner-base.sh" "$FN_SCHED" 12
for i in 1 2 3; do
  run_case "BASE test file (0.5s fixture) + THIS BRANCH runner (perl now_ms only), repeat $i/3" \
    "$W/slowshim" "$W/testfile-base.sh" "$REPO/bin/fm-test-run.sh" "$FN_SCHED" 4
done
run_case "THIS BRANCH test file (2s fixture) + BASE runner" \
  "$W/slowshim" "$REPO/tests/fm-test-run.test.sh" "$W/runner-base.sh" "$FN_SCHED" 3
for i in 1 2 3; do
  run_case "SHIPPED: this branch test file + this branch runner, repeat $i/3" \
    "$W/slowshim" "$REPO/tests/fm-test-run.test.sh" "$REPO/bin/fm-test-run.sh" "$FN_SCHED" 3
done

echo
echo "=============================================================================="
echo "PART 2 - failure diagnostics of the interpreter-fallthrough timing test"
echo "         (a stub \`date\` forces the failure path; nothing else is changed)"
echo "=============================================================================="
run_case "BEFORE ${MID_REV:0:7}+cleanup fix: temp dir removed before the message read it" \
  "$W/nodate" "$W/testfile-precleanup.sh" "$REPO/bin/fm-test-run.sh" "$FN_CLOCK" 8
run_case "SHIPPED: runner stdout/stderr printed before cleanup" \
  "$W/nodate" "$REPO/tests/fm-test-run.test.sh" "$REPO/bin/fm-test-run.sh" "$FN_CLOCK" 8

printf '\nscratch kept at: %s\n' "$W"

Evidence: calm-pi node capability gate, both directions (skips when node cannot import .ts, runs and passes when it can)

--- default node v23.3.0 --- TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" skip: node cannot import TypeScript modules for Pi calm missing-adapter-export test --- with type stripping enabled --- ok - missing Pi presentation class exports reach the independent adapter degradation path

=== calm-pi missing-adapter-export subtest: capability gate, both directions ===

host node: v23.3.0  (native .ts import needs 22.18+/24; 23.3 needs the flag)

--- 1. default node: the .ts import probe fails exactly as the subtest used to ---
$ cd <probe fixture> && node --input-type=module -e 'await import("./probe.ts")'
  throw new ERR_UNKNOWN_FILE_EXTENSION(ext, filepath);
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /private/tmp/tsprobe/probe.ts

--- 2. so the subtest skips instead of failing the suite ---
$ bin/fm-test-run.sh tests/fm-calm-pi-extension.test.sh
skip: node cannot import TypeScript modules for Pi calm missing-adapter-export test
FM_TEST_END 2026-07-30T10:41:16Z tests/fm-calm-pi-extension.test.sh exit=0 duration_ms=226 gate_skip=true
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=1 duration_ms=293

--- 3. same suite on a node that CAN import .ts (type stripping enabled) ---
$ NODE_OPTIONS="--experimental-strip-types --no-warnings" bin/fm-test-run.sh tests/fm-calm-pi-extension.test.sh
skip: installed @earendil-works/pi-coding-agent package not found
ok - Pi calm compatibility evidence never rejects a Pi version for being newer than 0.82.0, and still fails closed on a missing or malformed version
skip: installed @earendil-works/pi-coding-agent package not found
ok - missing Pi presentation class exports reach the independent adapter degradation path
skip: installed @earendil-works/pi-coding-agent package not found
skip: pi or tmux not found for Pi operational follow-up E2E
skip: pi or tmux not found for Pi Calm hidden-block geometry E2E
skip: pi or tmux not found for Pi calm interactive E2E
FM_TEST_END 2026-07-30T10:46:52Z tests/fm-calm-pi-extension.test.sh exit=0 duration_ms=380 gate_skip=true
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=1 duration_ms=450

The gate is capability-scoped, not a blanket skip: the subtest itself runs and
passes ("ok - missing Pi presentation class exports reach the independent
adapter degradation path") as soon as node can load the .ts adapters, and only
skips on a node that would have failed with ERR_UNKNOWN_FILE_EXTENSION.

Note: node 24 is not installed on this host, so the capable direction was
exercised on node 23.3.0 with type stripping switched on explicitly
(--no-warnings mirrors node 24, which strips types without an experimental warning).

- Outcome: ⚠️ 1 info across 1 run (26m19s)

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed (2) ✅

• ⚠️ bin/fm-test-run.sh:112 - The new preferred now_ms branch is guarded only by command -v perl, which does not guarantee the Time::HiRes module is installed (it ships with full perl distributions but can be absent from minimal/stripped perl packages). When the module is missing, perl -MTime::HiRes=time -e ... aborts at compile time with exit 2 and empty stdout. Because bin/fm-test-run.sh runs under set -eu (line 65), the very first caller RUN_STARTED_MS=$(now_ms) (line 1154) propagates that non-zero status and terminates the entire test runner, instead of falling through to the python3 or date branches that are already written directly below. The previous code could not fail this way: python3 -c 'import time; ...' only ran when python3 existed, and time is unconditionally available there. Fix by making the perl branch fall through on failure (e.g. capture into a variable, if ms=$(perl ... 2>/dev/null) && [ -n "$ms" ]) or by probing perl -MTime::HiRes -e1 once at script init and caching the result.
• ℹ️ bin/fm-crew-state.sh:440 - nm_head_matches_worktree repeats the same equality-then-ancestry test in both the worktree branch (lines 441-445) and the gate-repo branch (lines 450-454). The two only differ in which repo the git commands target, so the function can be flattened by resolving the repo first and running a single test: set repo=$WT, and if rev-parse --verify fails there, reassign repo to the gate path and re-resolve; then a single [ "$run_full" = "$local_full" ] && return 0 followed by git -C "$repo" merge-base --is-ancestor "$local_full" "$run_full" 2>/dev/null as the trailing command. This is behavior-identical (the equality check is unreachable in the gate branch, since a head equal to worktree HEAD always resolves locally, and --is-ancestor is reflexive anyway) and removes one of the two duplicated branches plus about eight lines.
• ℹ️ tests/fm-calm-pi-extension.test.sh:307 - The new TypeScript-import capability probe writes probe.ts into a directory with no package.json, while the fixture it gates creates {"type":"module"} at line 320 before importing its .ts adapters. The probe therefore exercises node's ambiguous-format detection for .ts rather than the explicit-ESM path the real subtest uses. If that detection does not apply (or is disabled), a node that fully supports type stripping would fail the probe with a SyntaxError and skip the missing-adapter-export regression test entirely, silently losing coverage on exactly the versions where it should run. Verified on this host (node v23.3.0) that the probe correctly fails with ERR_UNKNOWN_FILE_EXTENSION, so the gate works for the older-node case; the gap is only on capable nodes. Writing {"type":"module"} into the probe dir makes the probe match what it gates and removes the ambiguity.

🔧 Fix: harden now_ms fallbacks, flatten head match, fix ts probe
1 warning still open:

• ⚠️ tests/fm-test-run.test.sh:275 - All four failure paths of the new test_timing_survives_broken_interpreters delete the temp dir before reading from it. At lines 275-276 and 283-284 rm -rf "$tmp" runs as its own statement ahead of fail "...: $(cat "$tmp/err.txt")", and at lines 279 and 287 the group { rm -rf "$tmp"; fail "...: $(cat "$tmp/out.txt")"; } does the same. Group members execute in order and fail's arguments are expanded only when fail is invoked, i.e. after the rm, so the command substitution always reads a deleted path: the diagnostic body is empty and cat additionally writes "No such file or directory" to stderr. The embedded runner stderr/stdout - precisely the evidence needed if the now_ms fallback chain ever regresses - is therefore never shown. Fix by capturing into a local before removing (err=$(cat "$tmp/err.txt"); rm -rf "$tmp"; fail "...: $err"), or follow the idiom already used at line 494 in this same file: { cat "$tmp/out" "$tmp/err"; rm -rf "$tmp"; fail "<static message>"; }. Pass/fail detection itself is unaffected, so this is a debuggability defect rather than a wrong-result one.

🔧 Fix: print runner evidence before temp cleanup in timing test
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 info

• ℹ️ tests/fm-kimi-harness.test.sh - tests/fm-kimi-harness.test.sh fails on this host inside the changed-file lane with "fm-kimi-turnend-hook: refused: python3 with tomllib is required to validate config.toml" - the default python3 is 3.9.6 (tomllib landed in 3.11). Both the test and bin/fm-kimi-turnend-hook.sh are untouched by this range, and the suite passes with python3.13 on PATH, so it is a host-environment red unrelated to this change; no repo change is needed.
• bin/fm-test-run.sh tests/fm-crew-state.test.sh - all cases pass, including the 4 new gate-repo head-resolution cases and the file:// case I added
• bin/fm-test-run.sh tests/fm-test-run.test.sh - passes, including the new test_timing_survives_broken_interpreters
• bin/fm-test-run.sh tests/fm-calm-pi-extension.test.sh - passes; the .ts-import subtest skips on this host's node 23.3.0
• NODE_OPTIONS="--experimental-strip-types --no-warnings" bin/fm-test-run.sh tests/fm-calm-pi-extension.test.sh - the same subtest runs and passes when node can import .ts
• bin/fm-test-run.sh --changed --base a53ffc17fe5b4c6c2ba0de80e82adbbf16ec9253 - 25 scripts, 1 unrelated failure (fm-kimi-harness, host python3 lacks tomllib)
• bin/fm-test-run.sh tests/fm-kimi-harness.test.sh with python3.13 on PATH - passes, confirming the red is environmental
• Manual E2E: gate-head-e2e.sh builds a real worktree + bare gate repo + gate-only fix commit and runs bin/fm-crew-state.sh from base a53ffc1 vs target for the incident, a diverged gate-only head, a locally advanced tip, and a file:// gate remote
• Manual E2E: crew_absorb_class / crew_is_provably_working (bin/fm-classify-lib.sh, via FM_CREW_STATE_BIN) over the same incident topology with both helper versions
• Manual: runner-timing-evidence.sh runs test_jobs_parallel_scheduler_and_failure_propagation for base/target test-file x base/target runner under a ~630ms python3 shim, 3 repeats of the shipped pair, plus the failure-diagnostics comparison of test_timing_survives_broken_interpreters
• Fixture-integrity checks: git -C <worktree> cat-file -e <fix-head>^{commit} proves the run head is not an object in the crew worktree, and git merge-base proves the diverged control has no shared ancestor

⚠️ **Document** - 1 info

• ℹ️ docs/fm-test-portable-shards.md:8 - Follow-up, deliberately out of scope here: the 2026-07-29 isolation-proof durations in docs/fm-test-isolation-proof.md / .json feed the duration-balanced lane partition documented at docs/fm-test-portable-shards.md:8-48, and this change lengthened tests/fm-test-run.test.sh (jobs-scheduler fixture 0.5s -> 2s plus a new clock-fallthrough test) and tests/fm-crew-state.test.sh (new gate-repo cases). The recorded tables are dated evidence of a specific proof run, so I did not overwrite them with local numbers from different hardware; the measured imbalance (318 ms across ~162 s lanes) stays immaterial. Refresh both records from bin/fm-test-isolation-proof.sh --jobs 4 --json ... the next time the lane memberships in bin/fm-test-run.sh are re-derived, since that step edits executable code.

⚠️ **Lint** - 1 warning

• ⚠️ linter found issues (exit code 127)

✅ **Push** - passed

✅ No issues found.

[kunchenguid]

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

_{Noted for firstmate#1307 at f48dce8f.}

[devin-ai-integration]

Closed as superseded — this work already landed on main via #3681.

— Kun's Firstmate