Skip to content

fix(bin): let a live rebased run outrank the previous failed one in crew state - #4590

Open
dsantosg1103 wants to merge 7 commits into
kunchenguid:mainfrom
dsantosg1103:fm/fm-crew-state-reporta-corrida-vieja
Open

dsantosg1103 wants to merge 7 commits into
kunchenguid:mainfrom
dsantosg1103:fm/fm-crew-state-reporta-corrida-vieja

Conversation

@dsantosg1103

@dsantosg1103 dsantosg1103 commented Sep 16, 2026 •

Copy link
Copy Markdown

Intent

bin/fm-crew-state.sh reporto state: failed - source: run-step - run failed para una tarea cuya corrida de no-mistakes estaba VIVA y parada en su compuerta. Observado el 2026-09-07 en la tarea nutrifam-cerrar-allow-authenticated, copia ~/.treehouse/Nutrifam-fb5f91/1/Nutrifam:

  • corrida viva: 01M1YR324J8R91FYXYZC36ZPEM, status running, awaiting_agent parked 23m57s, compuerta en el paso review con status fix_review y 2 hallazgos, uno ask-user.
  • corrida que SI fallo: la ANTERIOR, 01M1Y8WFNKZVWKTT544KRS4GXQ, muerta en review por limite de cuota. Despues de esa, el trabajador recupero custodia y arranco la corrida nueva, y el clasificador se quedo leyendo el registro de la vieja.

Por que importa: el vigilante emitio un wake check: inactive-outcome ... state=failed que pide presentarle al capitan un resultado terminal que NO ocurrio. Es la misma clase de falso positivo que ya esta documentada en data/learnings.md para el falso "termino" al disparar /no-mistakes, y es peor que aquella, porque aquella venia de una linea de status ambigua y esta viene del clasificador que existe justamente para ser la fuente de verdad del estado actual. Un aviso equivocado que llega hasta el capitan erosiona la confianza en los avisos de verdad.

Criterio de aceptacion: con una corrida viva parada en compuerta y una corrida anterior fallida en la misma copia de trabajo, bin/fm-crew-state.sh debe reportar el estado de la corrida VIVA, y no debe emitirse un wake de resultado terminal. Con cobertura de prueba de ese caso concreto, porque es reproducible.

What Changed

  • fm_nm_runs_status_for_worktree in bin/fm-nm-run-lib.sh now treats a newest ledger row whose head diverges from the worktree HEAD the same as one that never resolved locally: if the row is LIVE it reaches the pipeline-continuation anchor (the immediately older row for the same branch must resolve to exactly the worktree HEAD), so a run the pipeline rebased onto an advanced upstream is recognized instead of falling through to the previous failed run's status.
  • Rows that do not bind are now split by ancestry: a head that resolves as a strict ancestor of the worktree HEAD is superseded local history and ends the scan outright, and only absent or unshared-line heads reach the anchor, which stays exact head equality. Terminal and unclassifiable unbindable rows still print nothing.
  • tests/fm-crew-state.test.sh adds three cases built on real git fixtures: the incident shape (rebased live row + terminal row at the worktree commit, pinning the whole state: failed · source: run-step · run failed line as a regression literal), a negative control where the anchor row is a descendant rather than the exact commit and nothing binds, and a strict-ancestor live newest row that the anchor must not rescue. Header comments in bin/fm-crew-state.sh, bin/fm-nm-run-lib.sh, and docs/architecture.md are updated to state the widened unbindable-head rule and its limits.

Risk Assessment

✅ Low: The change is tightly scoped and conservative - it widens exactly one ledger shape (live row with a diverged head) into the existing exact-head anchor, adds a strict-ancestor guard that closes the round-4 regression, leaves the sibling scan, the anchor rule, and teardown's call path untouched, and pins all four shapes with behavioral tests that drive the real script.

Testing

Reproduced the 2026-09-07 false-failed report against the real product before testing the fix: with the dead run bound by axi status at the worktree's exact commit and the live rebased run newest on the runs ledger, the base-commit binaries print the incident's exact line and the target-commit binaries print the live run instead, with the real watcher scan queueing nothing. A no-live-successor control on the same fixture still reports failed and still queues the terminal-outcome wake, so the silence is specific to the incident shape rather than a disabled wake path. Three adversarial shapes aimed at the widened anchor - a strict-ancestor live newest row, a diverged terminal newest row, and a rebased live row anchored only by a descendant - each refuse to bind a run and fall to the pane, and the deliberately reverted sibling rule still lets a terminal row at the worktree commit stand. Evidence is CLI transcripts because this change has no rendered UI surface; bin/fm-crew-state.sh's single emitted line is the end-user surface and is pinned verbatim in the transcripts. One unrelated bounded-scan failure in the inactive-reconcile suite reproduces identically at the base commit on this host and is reported informationally.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Incident repro: with a live rebased run and the previous failed run in the same task copy, fm-crew-state.sh reports the LIVE run, not state: failed ✅ pass live drive-incident.sh S1 against the real bin/fm-crew-state.sh: base-commit tree prints state: failed · source: run-step · run failed (the incident's own line), target-commit tree prints `state: working…
No terminal-outcome wake: the real watcher scan over the incident shape queues nothing ✅ pass live drive-incident.sh S2 runs the real bin/fm-inactive-reconcile.sh scan with the real crew-state; state/.wake-queue and state/terminal-outcomes stay empty (incident-transcript.txt)
Control: a genuinely failed run with no live successor is still reported and still wakes the captain ✅ pass live drive-incident.sh S3: same fixture with the live ledger row removed prints state: failed · source: run-step · run failed and queues inactive-outcome:… child=nutrifamcerrar state=failed (incident-t…
Adversarial: a live newest ledger row whose head is a strict ANCESTOR of the worktree HEAD, with a perfect exact-head terminal anchor behind it, binds nothing ✅ pass live drive-guards.sh G1: real bin/fm-crew-state.sh answers state: working · source: pane · harness busy (claude-hook) - never source: run-step (guards-transcript.txt)
Adversarial: a diverged TERMINAL newest row with a perfect exact-head anchor behind it is never anchored into a terminal verdict ✅ pass live drive-guards.sh G2: output is source: pane, with no state: failed and no source: run-step (guards-transcript.txt)
Adversarial: a rebased live newest row whose anchor row is only a DESCENDANT (not the exact worktree commit) binds nothing ✅ pass live drive-guards.sh G3: output is state: working · source: pane · harness busy (claude-hook), so the anchor stayed exact-equality only (guards-transcript.txt)
Decided boundary: the reverted sibling widening - a diverged live sibling behind a terminal row at the worktree commit does not displace the terminal answer ✅ pass live drive-guards.sh G4: pre-fix and fixed trees both print state: failed · source: run-step · run failed, matching the round-4 revert decision (guards-transcript.txt)
Regression suite for the changed classifier stays green end to end ✅ pass live bash tests/fm-crew-state.test.sh - all cases pass, including the four new ledger/ancestry cases (crew-state-targeted-cases.txt)
Evidence: Incident end-to-end transcript (pre-fix vs fixed crew-state, watcher scan, control)

Source: Incident end-to-end transcript (pre-fix vs fixed crew-state, watcher scan, control)

worktree HEAD (submitted, where the DEAD run died): 042a5e4... live run head (rebased onto advanced upstream): 56d9ecf... neither commit is an ancestor of the other: confirmed === S1: the incident shape, PRE-FIX product (base bdcacb9) === state: failed · source: run-step · run failed === S1: the incident shape, FIXED product (HEAD) === state: working · source: run-step · validating (background run) === S2: the watcher (bin/fm-inactive-reconcile.sh scan) over the same shape === scan exit: 0 wake/outcome records emitted: [] === S3 control: the SAME dead run with no live successor still reports and wakes === state: failed · source: run-step · run failed actionable: inactive terminal outcome awaiting captain presentation: child=nutrifamcerrar state=failed scan exit: 0 wake/outcome records emitted: [inactive-outcome:dd0bb0d1524ff1478b40c4f872232aa6 inactive state=failed ]

worktree HEAD (submitted, where the DEAD run died): 042a5e4da9935de7e8e50deb00e6fdc9fa5369ed
live run head  (rebased onto advanced upstream):    56d9ecf840e33e1909606e0b48c7982dc8b859d8
neither commit is an ancestor of the other: confirmed

=== S1: the incident shape, PRE-FIX product (base bdcacb9fed45266ec4476b95d2290794cfa501bb) ===
state: failed · source: run-step · run failed

=== S1: the incident shape, FIXED product (HEAD) ===
state: working · source: run-step · validating (background run)

=== S2: the watcher (bin/fm-inactive-reconcile.sh scan) over the same shape ===
scan exit: 0
wake/outcome records emitted: []

=== S3 control: the SAME dead run with no live successor still reports and wakes ===
state: failed · source: run-step · run failed
actionable: inactive terminal outcome awaiting captain presentation: child=nutrifamcerrar state=failed
scan exit: 0
wake/outcome records emitted: [inactive-outcome:dd0bb0d1524ff1478b40c4f872232aa6	inactive state=failed ]
Evidence: Adversarial guard transcript (ancestor / diverged-terminal / unanchored / reverted sibling)

Source: Adversarial guard transcript (ancestor / diverged-terminal / unanchored / reverted sibling)

=== G1 strict-ancestor live newest row (head bfd3c97 is an ancestor of HEAD 6683856) === pre-fix : state: working · source: pane · harness busy (claude-hook) fixed : state: working · source: pane · harness busy (claude-hook) === G2 diverged TERMINAL newest row + perfect exact-head anchor behind it === fixed : state: working · source: pane · harness busy (claude-hook) === G3 rebased live row whose anchor is a DESCENDANT, not the exact commit === fixed : state: working · source: pane · harness busy (claude-hook) === G4 terminal row at the worktree commit, diverged live sibling behind it === pre-fix : state: failed · source: run-step · run failed fixed : state: failed · source: run-step · run failed

=== G1 strict-ancestor live newest row (head bfd3c97 is an ancestor of HEAD 6683856) ===
  pre-fix : state: working · source: pane · harness busy (claude-hook)
  fixed   : state: working · source: pane · harness busy (claude-hook)

=== G2 diverged TERMINAL newest row + perfect exact-head anchor behind it ===
  fixed   : state: working · source: pane · harness busy (claude-hook)

=== G3 rebased live row whose anchor is a DESCENDANT, not the exact commit ===
  fixed   : state: working · source: pane · harness busy (claude-hook)

=== G4 terminal row at the worktree commit, diverged live sibling behind it ===
  pre-fix : state: failed · source: run-step · run failed
  fixed   : state: failed · source: run-step · run failed
Evidence: Incident driver script (reproducible)

Source: Incident driver script (reproducible)

#!/usr/bin/env bash
# Live driver for the nutrifam-cerrar-allow-authenticated incident (2026-09-07).
#
# Stands up a real git worktree, a real firstmate state home, and a fake
# `no-mistakes` CLI on PATH that answers exactly as the real one did during the
# incident, then drives the REAL bin/fm-crew-state.sh and the REAL
# bin/fm-inactive-reconcile.sh (the watcher path that emitted the false wake).
#
# usage: drive-incident.sh <repo-root> <scratch-dir>
set -u
ROOT=$1
SCRATCH=$2
BRANCH=fm/nutrifam-cerrar-allow-authenticated
ID=nutrifamcerrar

rm -rf "$SCRATCH"; mkdir -p "$SCRATCH"
export GIT_AUTHOR_NAME=fmtest GIT_AUTHOR_EMAIL=fm@test.invalid
export GIT_COMMITTER_NAME=fmtest GIT_COMMITTER_EMAIL=fm@test.invalid

# --- the crew's task copy -----------------------------------------------------
WT="$SCRATCH/Nutrifam"
mkdir -p "$WT"
git -C "$WT" init -q
git -C "$WT" commit -q --allow-empty -m 'baseline'
git -C "$WT" checkout -q -b "$BRANCH"
git -C "$WT" commit -q --allow-empty -m 'cerrar allow-authenticated'
SUBMITTED=$(git -C "$WT" rev-parse HEAD)

# The live run's head: the pipeline replayed the branch onto an advanced
# upstream, so its head resolves here (the pipeline pushed it) but neither
# commit descends from the other.
git -C "$WT" checkout -q --detach "$(git -C "$WT" rev-list --max-parents=0 HEAD)"
git -C "$WT" commit -q --allow-empty -m 'upstream advanced'
git -C "$WT" commit -q --allow-empty -m 'replayed onto the advanced upstream'
REBASED=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q "$BRANCH"

git -C "$WT" merge-base --is-ancestor "$SUBMITTED" "$REBASED" && { echo "FIXTURE BROKEN: rebased descends"; exit 1; }
git -C "$WT" merge-base --is-ancestor "$REBASED" "$SUBMITTED" && { echo "FIXTURE BROKEN: submitted descends"; exit 1; }
echo "worktree HEAD (submitted, where the DEAD run died): $SUBMITTED"
echo "live run head  (rebased onto advanced upstream):    $REBASED"
echo "neither commit is an ancestor of the other: confirmed"
echo

# --- the fake no-mistakes CLI, answering as it did on 2026-09-07 --------------
FB="$SCRATCH/fakebin"; mkdir -p "$FB"
cat > "$FB/no-mistakes" <<'SH'
#!/usr/bin/env bash
set -u
case "${1:-}" in
  axi) shift
    case "${1:-}" in
      status) shift
        if [ "${1:-}" = --run ]; then printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}"
        else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;;
      logs) printf '%s\n' "${FM_FAKE_CI_LOGS:-}" ;;
    esac ;;
  runs) printf '%s\n' "${FM_FAKE_RUNS_LIST:-}" ;;
  daemon) printf 'daemon running (pid 4242)\n'; exit 0 ;;
esac
exit 0
SH
cat > "$FB/tmux" <<'SH'
#!/usr/bin/env bash
set -u
case "${1:-}" in
  display-message) printf '%%1\n' ;;
  capture-pane) printf 'all quiet\n> \n' ;;
esac
exit 0
SH
chmod +x "$FB/no-mistakes" "$FB/tmux"

# `axi status` answers with the PREVIOUS run: it died at this worktree's exact
# commit when review hit the quota limit.
export FM_FAKE_AXI_STATUS="run:
  id: \"01M1Y8WFNKZVWKTT544KRS4GXQ\"
  branch: $BRANCH
  status: completed
  head: \"$SUBMITTED\"
  pr: \"\"
  findings: none
outcome: failed"
export FM_FAKE_AXI_STATUS_RUN="" FM_FAKE_CI_LOGS=""

SHORT_SUB=$(git -C "$WT" rev-parse --short=7 "$SUBMITTED")
SHORT_REB=$(git -C "$WT" rev-parse --short=7 "$REBASED")
# The runs ledger: the live run newest, the dead one immediately older.
LEDGER_WITH_LIVE="  running    $BRANCH $SHORT_REB  2026-09-07 14:14
  failed     $BRANCH $SHORT_SUB  2026-09-07 09:48"
# Control ledger: the same dead run with NO live successor.
LEDGER_DEAD_ONLY="  failed     $BRANCH $SHORT_SUB  2026-09-07 09:48"

# --- the firstmate home the watcher scans ------------------------------------
HOME_DIR="$SCRATCH/home"
mkdir -p "$HOME_DIR"/{state,data,config,projects} "$SCRATCH/root"
"$ROOT/bin/fm-meta-write.sh" 2>/dev/null || true
cat > "$HOME_DIR/state/$ID.meta" <<EOF
window=firstmate:fm-$ID
worktree=$WT
project=Nutrifam
harness=claude
kind=ship
mode=no-mistakes
yolo=off
spawn_gen=s$$.1
EOF
printf 'working: cerrar allow-authenticated\n' > "$HOME_DIR/state/$ID.status"
: > "$HOME_DIR/state/$ID.turn-ended"
OLD=$(( $(date +%s) - 600 ))
STAMP=$(date -r "$OLD" +%Y%m%d%H%M.%S)
touch -t "$STAMP" "$HOME_DIR/state/$ID.meta" "$HOME_DIR/state/$ID.status" "$HOME_DIR/state/$ID.turn-ended"

crew_state() { # <bin-dir>
  PATH="$FB:$PATH" FM_STATE_OVERRIDE="$HOME_DIR/state" "$1/fm-crew-state.sh" "$ID"
}
reconcile() {
  rm -f "$HOME_DIR/state/.inactive-outcome-reconcile"*
  PATH="$FB:$PATH" FM_ROOT_OVERRIDE="$SCRATCH/root" FM_HOME="$HOME_DIR" \
    FM_STATE_OVERRIDE="$HOME_DIR/state" FM_DATA_OVERRIDE="$HOME_DIR/data" \
    FM_CONFIG_OVERRIDE="$HOME_DIR/config" FM_INACTIVE_RECONCILE_SECS=60 \
    "$ROOT/bin/fm-inactive-reconcile.sh" scan
}
wakes() {
  grep -o 'inactive-outcome[^ ]*' "$HOME_DIR/state/.wake-queue" 2>/dev/null || true
  grep -rho 'state=[a-z]*' "$HOME_DIR/state/terminal-outcomes" 2>/dev/null || true
}
clear_wakes() { rm -rf "$HOME_DIR/state/.wake-queue" "$HOME_DIR/state/terminal-outcomes"; }

# --- a pre-fix copy of the product, to prove the incident reproduces ---------
PRE="$SCRATCH/prefix-bin"
cp -R "$ROOT/bin" "$PRE"
git -C "$ROOT" show "${BASE_COMMIT:?}:bin/fm-nm-run-lib.sh" > "$PRE/fm-nm-run-lib.sh"
git -C "$ROOT" show "${BASE_COMMIT}:bin/fm-crew-state.sh" > "$PRE/fm-crew-state.sh"
chmod +x "$PRE/fm-crew-state.sh"

echo "=== S1: the incident shape, PRE-FIX product (base $BASE_COMMIT) ==="
export FM_FAKE_RUNS_LIST="$LEDGER_WITH_LIVE"
crew_state "$PRE"
echo
echo "=== S1: the incident shape, FIXED product (HEAD) ==="
crew_state "$ROOT/bin"
echo
echo "=== S2: the watcher (bin/fm-inactive-reconcile.sh scan) over the same shape ==="
clear_wakes
reconcile; echo "scan exit: $?"
echo "wake/outcome records emitted: [$(wakes | tr '\n' ' ')]"
echo
echo "=== S3 control: the SAME dead run with no live successor still reports and wakes ==="
export FM_FAKE_RUNS_LIST="$LEDGER_DEAD_ONLY"
crew_state "$ROOT/bin"
clear_wakes
reconcile; echo "scan exit: $?"
echo "wake/outcome records emitted: [$(wakes | tr '\n' ' ')]"
Evidence: Adversarial guard driver script (reproducible)

Source: Adversarial guard driver script (reproducible)

#!/usr/bin/env bash
# Adversarial driver: the boundaries the incident fix must NOT cross.
# Drives the REAL bin/fm-crew-state.sh over real git repos + a fake
# `no-mistakes` CLI, for the ledger shapes that try to abuse the new
# "unbindable live newest row reaches the anchor" rule.
# usage: drive-guards.sh <repo-root> <scratch-dir>
set -u
ROOT=$1; SCRATCH=$2
rm -rf "$SCRATCH"; mkdir -p "$SCRATCH"
export GIT_AUTHOR_NAME=fmtest GIT_AUTHOR_EMAIL=fm@test.invalid
export GIT_COMMITTER_NAME=fmtest GIT_COMMITTER_EMAIL=fm@test.invalid

FB="$SCRATCH/fakebin"; mkdir -p "$FB"
cat > "$FB/no-mistakes" <<'SH'
#!/usr/bin/env bash
set -u
case "${1:-}" in
  axi) shift; case "${1:-}" in
      status) shift; if [ "${1:-}" = --run ]; then printf '\n'; else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;;
    esac ;;
  runs) printf '%s\n' "${FM_FAKE_RUNS_LIST:-}" ;;
  daemon) printf 'daemon running (pid 4242)\n'; exit 0 ;;
esac
exit 0
SH
cat > "$FB/tmux" <<'SH'
#!/usr/bin/env bash
set -u
case "${1:-}" in
  display-message) printf '%%1\n' ;;
  capture-pane) printf 'work in progress\nesc to interrupt\n' ;;
esac
exit 0
SH
chmod +x "$FB/no-mistakes" "$FB/tmux"

setup_case() { # <name> <branch> -> sets WT, HOME_DIR, ID
  NAME=$1; BR=$2; ID=$1
  WT="$SCRATCH/$NAME/wt"; HOME_DIR="$SCRATCH/$NAME/home"
  mkdir -p "$WT" "$HOME_DIR/state"
  git -C "$WT" init -q
  git -C "$WT" commit -q --allow-empty -m baseline
  git -C "$WT" checkout -q -b "$BR"
  cat > "$HOME_DIR/state/$ID.meta" <<EOF
window=firstmate:fm-$ID
worktree=$WT
project=p
harness=claude
kind=ship
EOF
}
arm_busy() { # make the pane a definitive busy answer, so a non-binding run row
  # falls to an unambiguous pane verdict instead of "harness unavailable".
  local gen
  gen=$("$ROOT/bin/fm-busy-event.sh" arm "$HOME_DIR/state" "$ID")
  "$ROOT/bin/fm-busy-event.sh" apply "$HOME_DIR/state" "$ID" busy --gen "$gen" \
    --source claude-hook --event user-prompt-submit
}
crew_state() { PATH="$FB:$PATH" FM_STATE_OVERRIDE="$HOME_DIR/state" "$ROOT/bin/fm-crew-state.sh" "$ID"; }
prefix_crew_state() { PATH="$FB:$PATH" FM_STATE_OVERRIDE="$HOME_DIR/state" "$SCRATCH/prefix-bin/fm-crew-state.sh" "$ID"; }
short() { git -C "$WT" rev-parse --short=7 "$1"; }

PRE="$SCRATCH/prefix-bin"; cp -R "$ROOT/bin" "$PRE"
git -C "$ROOT" show "${BASE_COMMIT:?}:bin/fm-nm-run-lib.sh" > "$PRE/fm-nm-run-lib.sh"

# --- G1: live NEWEST row whose head is a STRICT ANCESTOR of the worktree HEAD,
# with a perfect exact-head terminal anchor immediately behind it. That run is
# superseded local history, not an unprovable pipeline continuation.
setup_case ancestorlive fm/g-ancestor
git -C "$WT" commit -q --allow-empty -m 'abandoned run launched here'
OLDER=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" commit -q --allow-empty -m 'local work advanced past it'
HEADC=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" merge-base --is-ancestor "$OLDER" "$HEADC" || { echo "FIXTURE BROKEN"; exit 1; }
export FM_FAKE_AXI_STATUS="run:
  id: \"01OTHER\"
  branch: fm/some-other-crew
  status: running
  head: \"aaaaaaa\"
  pr: \"\"
  findings: none"
export FM_FAKE_RUNS_LIST="  running    fm/other aaaaaaa  2026-09-07 15:00
  running    fm/g-ancestor $(short "$OLDER")  2026-09-07 14:14
  failed     fm/g-ancestor $(short "$HEADC")  2026-09-07 09:48"
arm_busy
echo "=== G1 strict-ancestor live newest row (head $(short "$OLDER") is an ancestor of HEAD $(short "$HEADC")) ==="
echo "  pre-fix : $(prefix_crew_state)"
echo "  fixed   : $(crew_state)"
echo

# --- G2: DIVERGED TERMINAL newest row with a perfect exact-head anchor behind.
setup_case divterm fm/g-divterm
git -C "$WT" commit -q --allow-empty -m 'the work this crew submitted'
HEADC=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q --detach "$(git -C "$WT" rev-list --max-parents=0 HEAD)"
git -C "$WT" commit -q --allow-empty -m 'upstream advanced'
git -C "$WT" commit -q --allow-empty -m 'another task replayed onto it'
DIV=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q fm/g-divterm
export FM_FAKE_RUNS_LIST="  running    fm/other aaaaaaa  2026-09-07 15:00
  failed     fm/g-divterm $(short "$DIV")  2026-09-07 14:14
  completed  fm/g-divterm $(short "$HEADC")  2026-09-07 09:48"
arm_busy
echo "=== G2 diverged TERMINAL newest row + perfect exact-head anchor behind it ==="
echo "  fixed   : $(crew_state)"
echo

# --- G3: rebased live newest row whose anchor row is a DESCENDANT, not the
# exact worktree commit. The anchor is exact-equality only.
setup_case noanchor fm/g-noanchor
git -C "$WT" commit -q --allow-empty -m 'the work this crew submitted'
HEADC=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" commit -q --allow-empty -m 'a later commit'
DESC=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" reset -q --hard "$HEADC"
git -C "$WT" checkout -q --detach "$(git -C "$WT" rev-list --max-parents=0 HEAD)"
git -C "$WT" commit -q --allow-empty -m 'upstream advanced'
git -C "$WT" commit -q --allow-empty -m 'replayed onto it'
REB=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q fm/g-noanchor
export FM_FAKE_RUNS_LIST="  running    fm/other aaaaaaa  2026-09-07 15:00
  running    fm/g-noanchor $(short "$REB")  2026-09-07 14:14
  failed     fm/g-noanchor $(short "$DESC")  2026-09-07 09:48"
arm_busy
echo "=== G3 rebased live row whose anchor is a DESCENDANT, not the exact commit ==="
echo "  fixed   : $(crew_state)"
echo

# --- G4: the reverted sibling rule. Terminal newest row AT the worktree commit
# (binds), live sibling behind it with a DIVERGED head. Round-4 decision: the
# sibling widening was reverted, so the terminal answer stands.
setup_case sibling fm/g-sibling
git -C "$WT" commit -q --allow-empty -m 'the work this crew submitted'
HEADC=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q --detach "$(git -C "$WT" rev-list --max-parents=0 HEAD)"
git -C "$WT" commit -q --allow-empty -m 'upstream advanced'
git -C "$WT" commit -q --allow-empty -m 'replayed onto it'
REB=$(git -C "$WT" rev-parse HEAD)
git -C "$WT" checkout -q fm/g-sibling
export FM_FAKE_RUNS_LIST="  failed     fm/g-sibling $(short "$HEADC")  2026-09-07 14:14
  running    fm/g-sibling $(short "$REB")  2026-09-07 09:48"
echo "=== G4 terminal row at the worktree commit, diverged live sibling behind it ==="
echo "  pre-fix : $(prefix_crew_state)"
echo "  fixed   : $(crew_state)"
Evidence: Targeted crew-state suite cases for this change

Source: Targeted crew-state suite cases for this change

ok - an unfetched live sibling outranks a terminal row at the worktree's exact commit ok - a live run whose head was rebased outranks a terminal row at the worktree's commit ok - a rebased live row without the exact anchor still binds nothing ok - a diverged terminal newest row is never anchored by the row behind it ok - a strict-ancestor live newest row is never anchored ok - a genuinely failed run with no later run is not hidden ok - active fix round with an unfetched pipeline head reads working all fm-crew-state tests passed

ok - an unfetched live sibling outranks a terminal row at the worktree's exact commit
ok - a live run whose head was rebased outranks a terminal row at the worktree's commit
ok - a rebased live row without the exact anchor still binds nothing
ok - a diverged terminal newest row is never anchored by the row behind it
ok - a strict-ancestor live newest row is never anchored
ok - a genuinely failed run with no later run is not hidden
ok - active fix round with an unfetched pipeline head reads working
all fm-crew-state tests passed
- Outcome: ⚠️ 1 warning across 1 run (25m34s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ bin/fm-teardown.sh:1778 - Informational, pre-existing and outside this change's intent - no action expected here. This change taught the READ path (crew-state) that a rebased, resolvable-but-diverged run head is as unprovable as an unfetched one, but teardown's WRITE path still gates the shared ledger proof on the head being absent entirely: [ -z &#34;$(fm_nm_resolve_commit &#34;$wt&#34; &#34;$run_head&#34;)&#34; ] || return 1 runs before fm_nm_runs_status_for_worktree is ever consulted. So for the incident's own shape - a live run parked at its gate whose rebased head the task copy CAN resolve (the premise tests/fm-crew-state.test.sh:1263 states) - crew-state now correctly reports the live run, while task_status_is_own_parked_run still returns 1 and teardown orphans that parked run instead of concluding it. The library header at bin/fm-nm-run-lib.sh:8-13 names both callers as sharing one recognition rule, so the two paths now disagree about what "cannot be bound" means. This is unchanged behavior, not a regression introduced by the diff, and closing it would extend the change beyond the stated intent (which is about fm-crew-state.sh reporting only); recorded here only so the asymmetry is a deliberate choice rather than an oversight.
⚠️ **Test** - 1 warning
  • ⚠️ tests/fm-inactive-reconcile.test.sh:735 - Pre-existing, unrelated test failure on this host: tests/fm-inactive-reconcile.test.sh fails at test_stalled_state_read_is_bounded_and_scan_progresses ("next bounded scan did not resume with the following child", tests/fm-inactive-reconcile.test.sh:735). It fails identically when run from a clean tree at the base commit bdcacb9, so this change did not cause it. This machine has no GNU timeout/gtimeout, so fm_run_timed (bin/fm-timeout-lib.sh:134) takes the perl fork/setpgrp fallback; a manual replay of the bounded world shows the durable scan cursor advancing a -> b across the two budgeted scans while no child=b state=done wake is queued. Left unfixed because it is outside this change's surface and a fix would touch the bounded-scan path; remote CI should confirm whether it also fails where timeout exists.
  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Incident repro: with a live rebased run and the previous failed run in the same task copy, fm-crew-state.sh reports the LIVE run, not state: failed ✅ pass live drive-incident.sh S1 against the real bin/fm-crew-state.sh: base-commit tree prints state: failed · source: run-step · run failed (the incident's own line), target-commit tree prints `state: working…
No terminal-outcome wake: the real watcher scan over the incident shape queues nothing ✅ pass live drive-incident.sh S2 runs the real bin/fm-inactive-reconcile.sh scan with the real crew-state; state/.wake-queue and state/terminal-outcomes stay empty (incident-transcript.txt)
Control: a genuinely failed run with no live successor is still reported and still wakes the captain ✅ pass live drive-incident.sh S3: same fixture with the live ledger row removed prints state: failed · source: run-step · run failed and queues inactive-outcome:… child=nutrifamcerrar state=failed (incident-t…
Adversarial: a live newest ledger row whose head is a strict ANCESTOR of the worktree HEAD, with a perfect exact-head terminal anchor behind it, binds nothing ✅ pass live drive-guards.sh G1: real bin/fm-crew-state.sh answers state: working · source: pane · harness busy (claude-hook) - never source: run-step (guards-transcript.txt)
Adversarial: a diverged TERMINAL newest row with a perfect exact-head anchor behind it is never anchored into a terminal verdict ✅ pass live drive-guards.sh G2: output is source: pane, with no state: failed and no source: run-step (guards-transcript.txt)
Adversarial: a rebased live newest row whose anchor row is only a DESCENDANT (not the exact worktree commit) binds nothing ✅ pass live drive-guards.sh G3: output is state: working · source: pane · harness busy (claude-hook), so the anchor stayed exact-equality only (guards-transcript.txt)
Decided boundary: the reverted sibling widening - a diverged live sibling behind a terminal row at the worktree commit does not displace the terminal answer ✅ pass live drive-guards.sh G4: pre-fix and fixed trees both print state: failed · source: run-step · run failed, matching the round-4 revert decision (guards-transcript.txt)
Regression suite for the changed classifier stays green end to end ✅ pass live bash tests/fm-crew-state.test.sh - all cases pass, including the four new ledger/ancestry cases (crew-state-targeted-cases.txt)
  • bash tests/fm-crew-state.test.sh (full targeted suite for the changed classifier, all cases pass, including the four new ones)
  • bash tests/fm-gotmp.test.sh (other consumer of bin/fm-nm-run-lib.sh)
  • bash tests/fm-inactive-reconcile.test.sh (consumer of the wake path; 1 pre-existing failure, reproduced at base commit bdcacb9 from a clean git archive tree)
  • Live incident driver: BASE_COMMIT=bdcacb9 bash drive-incident.sh &lt;repo&gt; &lt;scratch&gt; - builds a real git task copy with a rebased (bidirectionally non-ancestral) live head, serves the dead previous run through a fake no-mistakes axi status and the live row through no-mistakes runs, then runs the real bin/fm-crew-state.sh under pre-fix and fixed bin trees and the real bin/fm-inactive-reconcile.sh scan
  • Live guard driver: BASE_COMMIT=bdcacb9 bash drive-guards.sh &lt;repo&gt; &lt;scratch&gt; - drives the real bin/fm-crew-state.sh over four adversarial ledger/ancestry shapes (strict-ancestor live newest row, diverged terminal newest row, rebased live row with descendant-only anchor, reverted diverged live sibling)
  • Manual replay of the bounded-scan case to characterise the pre-existing fm-inactive-reconcile failure (cursor advances a -> b, no wake queued; host has no GNU timeout)
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.


Checks are waiting on a maintainer to authorize workflow runs - this PR comes from a fork, so GitHub reports both runs as action_required with zero jobs started rather than failing them. Local validation is complete: review passed with no gate, the test step reproduced the 2026-09-07 incident end-to-end against the real bin/fm-inactive-reconcile.sh watcher path and confirmed no terminal-outcome wake is emitted (with a control proving that path still fires when a run genuinely failed), documentation passed, and ShellCheck 0.11.0 is clean on all three changed shell files.

bin/fm-crew-state.sh reported "state: failed - source: run-step - run
failed" for a task whose no-mistakes run was alive and parked at its
gate, and the watcher turned that into a terminal-outcome wake for an
outcome that never happened.

The ledger reader ended its scan at any newest row whose head the head
rule could not bind. That is correct for a terminal or unclassifiable
row, but a LIVE row whose head the pipeline replayed onto an advanced
upstream shares no ancestry with the worktree HEAD in either direction,
so it was discarded exactly like a foreign run - leaving the previous
run's terminal record standing as the present.

A rebased head is exactly as unprovable as an unfetched one, so both now
reach the same anchored pipeline-continuation recognition. The anchor
itself is unchanged: the immediately older row for the same branch must
still resolve to EXACTLY the worktree HEAD, so branch-name coincidence
and other tasks' runs still never match.

Observed 2026-09-07 on nutrifam-cerrar-allow-authenticated.
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

Outcome: waiting-ci — tip 878de07a78f9f8a7a325bb80abe1237b42dc13b6 attestation MATCH. Fork CI/NM approved after diff review (runs 35042884904 CI, 35043390250/35042884902 NM). Do not merge until green.

contract-class: restore — crew-state / fm_nm_runs_status_for_worktree must report the live rebased run, not a previous failed row at the worktree commit; restores the classifier as the honest source of current state (false state: failed → captain wake). Strict-ancestor and non-exact-anchor guards keep the scan fail-closed. No Fixes/Closes in body.

VISION: One captain/one interface — aligns (no false terminal wakes). Authority explicit — aligns (no new autonomy). Scripts own mechanics — aligns (ledger ancestry rules). Restart non-event — aligns (durable run ledger). Delegation spine — aligns (evidence honesty). Fleet outlives vendor — aligns. Scope — aligns.

Security: clean (no workflows/secrets). Review note on teardown vs read-path asymmetry is informational / out of intent — not a merge gate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants