fix: surface green PRs during CI monitoring

[kunchenguid]

Intent

Fix the fm-crew-state green-PR detection gap (backlog crewstate-green-pr-gap-p6). A no-mistakes run in its CI-monitor phase whose PR is already green was reading as 'working - validating (running)' from bin/fm-crew-state.sh instead of surfacing done/checks-passed, because axi status's steps[] table cannot distinguish 'still waiting on checks' from 'checks green, waiting on merge' - both read as plain ci,running. Root-caused this empirically against the real PR #252 run's ci.log (verified via ~/.no-mistakes/state.sqlite and ~/.no-mistakes/logs//ci.log on the installed v1.32.2 binary), which literally logs 'all CI checks passed - still monitoring until merged or closed' while axi status kept showing ci,running with no outcome. Fixed bin/fm-crew-state.sh to read the ci step's log tail via 'axi logs --step ci' and scan for the most recent recognized marker (green vs a later relapse into issues-detected/fixing), overriding working -> done once checks read green. As a safety net (since the watcher's wedge timer can re-escalate the same wedged pane forever, each escalation looking identical and easy to keep dismissing as 'still validating'), added a per-pane consecutive wedge-escalation counter to bin/fm-watch.sh that appends a demand-deep-inspection marker to the wake payload once a configurable threshold (default 3) is crossed; the counter resets when the pane's hash changes or becomes busy again. Also lightly clarified one line in bin/fm-brief.sh's ship-brief wording: the crew's own final 'done: PR {url} checks green' status line is owed at the CI-ready return point (when axi run first reports CI green), not after the background monitor-until-merge loop fully finishes - since in the incident the crew's foreground axi run call may never have returned promptly due to base-branch churn extending the ci step. Extended tests/fm-crew-state.test.sh (three new cases: checks-green-surfaces-done, still-waiting-stays-working, a later relapse-after-green is not masked) and tests/fm-watch-triage.test.sh (two new cases: consecutive escalations accumulate and demand-deep-inspection at the threshold, and the counter resets when the pane becomes active again) following the existing hermetic-fake patterns in each file. All 44 tests/.test.sh files pass and shellcheck bin/.sh bin/backends/.sh tests/*.sh is clean.

What Changed

• Updated fm-crew-state to inspect no-mistakes CI step logs so green PRs surface as done while pending, relapsed, fixing, or rearmed CI states remain working.
• Added watcher tracking for repeated wedge escalations, including a threshold marker that calls for deeper inspection and resets when pane activity changes.
• Clarified supervision docs and ship-brief wording, and expanded crew-state and watch-triage coverage for green CI, stale green, relapse, and repeated-wedge cases.

Risk Assessment

⚠️ Medium: Captain, the change is well covered and I found no material issues, but the risk stays medium because the behavior relies on parsing no-mistakes CI log text in a subtle supervision state machine.

Testing

The pre-run full test baseline was already green; I reran the two focused behavior suites and captured a reviewer-visible CLI transcript proving green PRs surface as done, waiting or relapsed CI stays working, and repeated wedge escalation demands deep inspection. Overall result: pass.

Evidence: fm-crew-state green PR gap E2E transcript

Shows real bin/fm-crew-state.sh returning state: done for a green CI-monitor log, state: working while checks are still running or relapsed, and real bin/fm-watch.sh adding demand-deep-inspection on the third same-pane wedge escalation.

# fm-crew-state green PR gap end-to-end evidence
repo: /Users/kunchen/.no-mistakes/worktrees/016d88035d58/01KWW88CMAVCF1BE1ZK4FXTCF2
head: 089018f

This transcript drives the real bin/fm-crew-state.sh and bin/fm-watch.sh entrypoints with hermetic no-mistakes/tmux fixtures. It shows the user-facing CLI state lines and watcher wake payloads for the changed behavior.

$ FM_STATE_OVERRIDE=/var/folders/5x/4nqprlbx0518k3ybcb1sz6gr0000gn/T/no-mistakes-evidence/01KWW88CMAVCF1BE1ZK4FXTCF2/manual-e2e/crew/cigreen/state PATH=<fake no-mistakes/tmux> bin/fm-crew-state.sh cigreen
  ci log tail:
    CI checks running, waiting for results...
    all CI checks passed - still monitoring until merged or closed
  => state: done · source: run-step · checks green: PR ready for review (still monitoring for merge/close)

$ FM_STATE_OVERRIDE=/var/folders/5x/4nqprlbx0518k3ybcb1sz6gr0000gn/T/no-mistakes-evidence/01KWW88CMAVCF1BE1ZK4FXTCF2/manual-e2e/crew/ciwait/state PATH=<fake no-mistakes/tmux> bin/fm-crew-state.sh ciwait
  ci log tail:
    CI checks running, waiting for results...
  => state: working · source: run-step · validating (running)

$ FM_STATE_OVERRIDE=/var/folders/5x/4nqprlbx0518k3ybcb1sz6gr0000gn/T/no-mistakes-evidence/01KWW88CMAVCF1BE1ZK4FXTCF2/manual-e2e/crew/cirelapse/state PATH=<fake no-mistakes/tmux> bin/fm-crew-state.sh cirelapse
  ci log tail:
    all CI checks passed - still monitoring until merged or closed
    base branch advanced (aaaaaaa..bbbbbbb), re-arming CI monitor timeout
    issues detected: merge conflict - auto-fixing (attempt 2/10)...
  => state: working · source: run-step · validating (running)

# fm-watch same-pane wedge escalation evidence
$ prime bin/fm-watch.sh once to classify the static pane as provably-working and start the wedge timer
  stale timer started for test:fm-wedged with no wake yet

$ round 1: FM_STALE_ESCALATE_SECS=240 bin/fm-watch.sh
  => stale: test:fm-wedged (idle 500s, possible wedge, escalation 1)

$ round 2: FM_STALE_ESCALATE_SECS=240 bin/fm-watch.sh
  => stale: test:fm-wedged (idle 500s, possible wedge, escalation 2)

$ round 3: FM_STALE_ESCALATE_SECS=240 bin/fm-watch.sh
  => stale: test:fm-wedged (idle 500s, possible wedge, escalation 3, demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone)

Persisted wake queue tail:
  1783365284	1	stale	test:fm-wedged	stale: test:fm-wedged (idle 500s, possible wedge, escalation 1)
  1783365284	2	stale	test:fm-wedged	stale: test:fm-wedged (idle 500s, possible wedge, escalation 2)
  1783365284	3	stale	test:fm-wedged	stale: test:fm-wedged (idle 500s, possible wedge, escalation 3, demand-deep-inspection: same pane has wedge-escalated 3 times in a row - do not re-absorb on the run-step/pane state alone)

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (33m49s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (6) ✅

• 🚨 bin/fm-crew-state.sh:320 - The green marker accepts any line containing no CI checks reported, but the real no-mistakes binary also emits no CI checks reported yet, waiting for checks to register... for a still-pending CI phase. If that is the latest marker, this reports state: done before checks have even registered. Match only the terminal no CI checks reported - still monitoring... form, or treat the ...yet... form as waiting.

🔧 Fix: Captain, distinguish pending no-checks CI marker
2 errors still open:

• 🚨 bin/fm-crew-state.sh:476 - A stale done: ... checks green status still wins even after the new CI-log parser sees a later relapse. If a crew wrote the expected done line at first green and no-mistakes later logs issues detected or CI checks running, RUN_STATE stays working and this branch emits done from the status log anyway, so the new relapse guard is bypassed in the common reported-status path. Gate this override on the CI log still being green, or skip it when the active CI log has a newer non-green marker.
• 🚨 bin/fm-crew-state.sh:292 - ci,fixing is eligible for the green override, so a current CI fix phase can be converted to done if the log tail's last recognized marker is an earlier green line. A fixing step is already current evidence that the run is not PR-ready; require the CI row to be running for the green-monitor override, or treat ci,fixing as an explicit non-green marker.

🔧 Fix: Harden CI relapse handling
1 error still open:

• 🚨 bin/fm-crew-state.sh:499 - A stale done: PR ... checks green line can still override an active top-level status: fixing when the steps table is absent or the ci row is unparseable: CI_STEP_STATUS stays empty, CI_LOG_STATE stays empty, and this branch emits done. Treat top-level fixing as not-ready before allowing the status-log fallback.

🔧 Fix: Block stale done during fixing
2 issues (1 error, 1 warning) still open:

• 🚨 bin/fm-crew-state.sh:479 - A top-level status: fixing run can still be converted to done when the steps table still contains ci,running and the CI log's last recognized marker is an earlier green line. Since top-level fixing is current evidence that the run is not PR-ready, skip the CI-green override whenever RUN_STATUS=fixing, not only in the status-log fallback.
• ⚠️ bin/fm-crew-state.sh:477 - The green override only consults the CI log when a ci,running steps row is parsed. The same mapper still supports top-level status: ci, but that shape leaves CI_STEP_STATUS empty and continues to report ci running even if the CI log says checks are green. Treat RUN_STATUS=ci as a running CI step fallback.

🔧 Fix: Captain, tighten CI status gating
2 errors still open:

• 🚨 bin/fm-crew-state.sh:335 - The CI log scanner ignores the base branch advanced, re-arming CI monitor timeout marker that the new tests already model. If that line is the latest log event after an earlier green marker, the grep filters it out and tail -1 still sees the old green line, so fm-crew-state can report done while CI has been invalidated by base-branch churn. Treat the re-arm/base-advanced marker as not-ready.
• 🚨 bin/fm-crew-state.sh:511 - The status-log relapse guard can run while RUN_SOURCE=coarse, but in that mode RUN_OUT still belongs to the other branch returned by the initial axi status. Calling nm_effective_ci_step_status/nm_ci_checks_state here can suppress this crew's own done: PR ... checks green line based on another branch's CI log, and the same coarse path still cannot detect a green PR when the crew never wrote that done line. Avoid CI-log probing in coarse mode unless you first obtain branch-specific run details or a branch-specific run id.

🔧 Fix: Captain, harden stale CI green handling
1 error still open:

• 🚨 bin/fm-crew-state.sh:335 - The re-arm guard still misses the real no-mistakes v1.32.2 marker, which is base branch advanced (<old>..<new>), re-arming CI monitor timeout. This matcher only recognizes base branch advanced, re-arming..., so after an earlier green line and a later real base-advance line, the later line is filtered out and the stale green marker still wins. Match base branch advanced.*re-arming CI monitor timeout and update the test fixture to include the commit range.

🔧 Fix: Captain, recognize ranged CI rearm markers
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 tests failed with exit code 1
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"

🔧 Fix: Confirm test suite passes
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Baseline already completed before this validation: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• bash tests/fm-crew-state.test.sh
• bash tests/fm-watch-triage.test.sh
• Manual evidence run captured at /var/folders/5x/4nqprlbx0518k3ybcb1sz6gr0000gn/T/no-mistakes-evidence/01KWW88CMAVCF1BE1ZK4FXTCF2/fm-crew-state-green-pr-gap-e2e-transcript.txt: hermetic fake no-mistakes/tmux fixtures drove the real bin/fm-crew-state.sh and bin/fm-watch.sh entrypoints for green CI, waiting CI, relapsed CI, and three consecutive wedge escalations.
• git status --short

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.