Skip to content

fix(bin): keep newest run attribution in crew-state - #5954

Open
mehulbhagwani wants to merge 5 commits into
kunchenguid:mainfrom
mehulbhagwani:fm/fm-bug-crewstate-run-identity
Open

mehulbhagwani wants to merge 5 commits into
kunchenguid:mainfrom
mehulbhagwani:fm/fm-bug-crewstate-run-identity

Conversation

@mehulbhagwani

@mehulbhagwani mehulbhagwani commented Sep 27, 2026 •

Copy link
Copy Markdown

Intent

Make every open mehulbhagwani PR on kunchenguid/firstmate clean for Kun to merge.

PRs: #5954 #5953 #5952 #5951 #5919 #5709 #4932

For each: checkout that PR head branch on origin/fork, rebase onto current upstream main, fix red CI (behavior portable serial jobs), ensure no-mistakes attestation head_sha MATCHES PR head, push to the PR branch (update existing PR in place — do NOT open new PRs), confirm CI green and mergeable CLEAN.

Priority: 5954 and 5952 already green—verify still clean; then 5953, 5951 red serial jobs; 5919 conflicts; 5709 attestation mismatch; 4932 two serial fails.

Do not merge on Kun's repo (no authority). Leave PRs ready. Report final status table.

Fix all 7 fast.

This run validates and updates existing PR #5954 in place (branch fm/fm-bug-crewstate-run-identity on the mehulbhagwani fork): crew-state attributes the active no-mistakes run per task without hiding a newer completed or cancelled run behind an older still-live run, and two runs created in the same minute are not misattributed when ledger timestamps collide; it prefers the overview order with a tie-break that keeps the live run. tests/fm-crew-state.test.sh covers both overview orders, including newer-first, which recorded overviews use. The previous CI failure on this PR was the unrelated Pi Calm hidden-block geometry E2E (/reload viewport transition), which this change does not touch. The branch was rebased onto main, so the push must update the existing PR branch (force with lease) and the PR body attestation must bind to the new head. Do not open a new PR.

What Changed

  • bin/fm-crew-state.sh and bin/fm-nm-run-lib.sh: crew-state now attributes the active no-mistakes run per task using the overview order, with a tie-break that keeps the live run. A newer completed or cancelled run is no longer hidden behind an older still-live run, and runs created in the same minute are not misattributed when ledger timestamps collide.
  • docs/architecture.md: documents the live-run precedence rule in crew-state attribution.
  • tests/fm-crew-state.test.sh: adds coverage for stale-status versus newest-run precedence and for both overview orders (newer-first and older-first) with same-minute runs.

Risk Assessment

✅ Low: The change is a narrowly gated tie-break in crew-state run selection that falls back to the overview's choice whenever the runs ledger cannot uniquely prove ordering, and it is covered by new behavioral tests in both overview orders.

Testing

Ran tests/fm-crew-state.test.sh, which drives the real bin/fm-crew-state.sh against disposable fixture overviews. All cases passed. The suite uses fixtures, not a running Herdr or no-mistakes daemon, so no scenario was driven live and each is marked untested.

  • Live validation: ⚠️ inconclusive - 0 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Live current run beats stale terminal overview row ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; it did not establish a live result against a running no-mistakes daemon.
Older live run does not hide a newer completed run; newer failed run stays failed ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; the shell test cannot start a real daemon, so no live result was established.
Ambiguous, mismatched or unreadable run selection reports unknown with candidate ids ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; no live result was established.
Evidence: crew-state test log

Source: crew-state test log

ok - captured AXI replacement status replays through crew-state
ok - captured AXI parked status replays through crew-state
ok - captured AXI failed status replays through crew-state
ok - captured capped inventory replays selection, ambiguity, and unavailable lookup
ok - captured status formats reject a synthetic authority transition
ok - captured completed status yields to synthetic subsequent development
ok - captured cancelled run leaves fleet inventory unverified without a failure contradiction
ok - failed-outcome/github/open: terminal delivery uses current disposition
ok - failed-outcome/github/merged: terminal delivery uses current disposition
ok - failed-outcome/github/closed: terminal delivery uses current disposition
ok - failed-outcome/github/unreadable: terminal delivery uses current disposition
ok - failed-outcome/github/skipped: terminal delivery uses current disposition
ok - failed-outcome/github/no-identity: terminal delivery uses current disposition
ok - failed-outcome/gitlab/open: terminal delivery uses current disposition
ok - failed-outcome/gitlab/merged: terminal delivery uses current disposition
ok - failed-outcome/gitlab/closed: terminal delivery uses current disposition
ok - failed-outcome/gitlab/unreadable: terminal delivery uses current disposition
ok - failed-outcome/gitlab/skipped: terminal delivery uses current disposition
ok - failed-outcome/gitlab/no-identity: terminal delivery uses current disposition
ok - failed-outcome/gerrit/open: terminal delivery uses current disposition
ok - failed-outcome/gerrit/merged: terminal delivery uses current disposition
ok - failed-outcome/gerrit/closed: terminal delivery uses current disposition
ok - failed-outcome/gerrit/unreadable: terminal delivery uses current disposition
ok - failed-outcome/gerrit/skipped: terminal delivery uses current disposition
ok - failed-outcome/gerrit/no-identity: terminal delivery uses current disposition
ok - failed-status/github/open: terminal delivery uses current disposition
ok - failed-status/github/merged: terminal delivery uses current disposition
ok - failed-status/github/closed: terminal delivery uses current disposition
ok - failed-status/github/unreadable: terminal delivery uses current disposition
ok - failed-status/github/skipped: terminal delivery uses current disposition
ok - failed-status/github/no-identity: terminal delivery uses current disposition
ok - failed-status/gitlab/open: terminal delivery uses current disposition
ok - failed-status/gitlab/merged: terminal delivery uses current disposition
ok - failed-status/gitlab/closed: terminal delivery uses current disposition
ok - failed-status/gitlab/unreadable: terminal delivery uses current disposition
ok - failed-status/gitlab/skipped: terminal delivery uses current disposition
ok - failed-status/gitlab/no-identity: terminal delivery uses current disposition
ok - failed-status/gerrit/open: terminal delivery uses current disposition
ok - failed-status/gerrit/merged: terminal delivery uses current disposition
ok - failed-status/gerrit/closed: terminal delivery uses current disposition
ok - failed-status/gerrit/unreadable: terminal delivery uses current disposition
ok - failed-status/gerrit/skipped: terminal delivery uses current disposition
ok - failed-status/gerrit/no-identity: terminal delivery uses current disposition
ok - cancelled-outcome/github/open: terminal delivery uses current disposition
ok - cancelled-outcome/github/merged: terminal delivery uses current disposition
ok - cancelled-outcome/github/closed: terminal delivery uses current disposition
ok - cancelled-outcome/github/unreadable: terminal delivery uses current disposition
ok - cancelled-outcome/github/skipped: terminal delivery uses current disposition
ok - cancelled-outcome/github/no-identity: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/open: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/merged: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/closed: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/unreadable: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/skipped: terminal delivery uses current disposition
ok - cancelled-outcome/gitlab/no-identity: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/open: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/merged: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/closed: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/unreadable: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/skipped: terminal delivery uses current disposition
ok - cancelled-outcome/gerrit/no-identity: terminal delivery uses current disposition
ok - cancelled-status/github/open: terminal delivery uses current disposition
ok - cancelled-status/github/merged: terminal delivery uses current disposition
ok - cancelled-status/github/closed: terminal delivery uses current disposition
ok - cancelled-status/github/unreadable: terminal delivery uses current disposition
ok - cancelled-status/github/skipped: terminal delivery uses current disposition
ok - cancelled-status/github/no-identity: terminal delivery uses current disposition
ok - cancelled-status/gitlab/open: terminal delivery uses current disposition
ok - cancelled-status/gitlab/merged: terminal delivery uses current disposition
ok - cancelled-status/gitlab/closed: terminal delivery uses current disposition
ok - cancelled-status/gitlab/unreadable: terminal delivery uses current disposition
ok - cancelled-status/gitlab/skipped: terminal delivery uses current disposition
ok - cancelled-status/gitlab/no-identity: terminal delivery uses current disposition
ok - cancelled-status/gerrit/open: terminal delivery uses current disposition
ok - cancelled-status/gerrit/merged: terminal delivery uses current disposition
ok - cancelled-status/gerrit/closed: terminal delivery uses current disposition
ok - cancelled-status/gerrit/unreadable: terminal delivery uses current disposition
ok - cancelled-status/gerrit/skipped: terminal delivery uses current disposition
ok - cancelled-status/gerrit/no-identity: terminal delivery uses current disposition
ok - outcome: cancellation without delivery carries no verdict
ok - status: cancellation without delivery carries no verdict
ok - selected: cancellation without delivery carries no verdict
ok - coarse: cancellation without delivery carries no verdict
ok - no-ci-log: cancellation without delivery carries no verdict
ok - red-ci: cancellation without delivery carries no verdict
ok - cancelled-test: cancellation without delivery carries no verdict
ok - skipped-test: cancellation without delivery carries no verdict
ok - synthetic cancelled run leaves fleet inventory unverified without a failure contradiction
ok - cancelled-outcome: terminal delivery reports only observed evidence
ok - cancelled-status: terminal delivery reports only observed evidence
ok - skipped-rebase: terminal delivery reports only observed evidence
ok - cancelled-skipped-rebase: terminal delivery reports only observed evidence
ok - passed: terminal delivery reports only observed evidence
ok - active run-step is authoritative
ok - stale needs-decision over active run is superseded
ok - stale blocked over active run is superseded
ok - daemon/timeout blocked claim over a live fixing run reads as run alive
ok - socket refusal or missing socket over a stale fixing run reports blocked
ok - socket refusal over a terminal attributed run reports blocked
ok - socket-down evidence outranks a live run only while it is the log's latest event
ok - broken-pipe blocker over a live run keeps the plain superseded reading
ok - genuine daemon-down blocked line still reports blocked
ok - a busy secondmate keeps its open blocker until that exact key closes
ok - the most recently opened decision supplies the reported state and detail
ok - ship and scout terminal declarations supersede stale decisions
ok - latest status retains legacy completion events and shared captain matching
ok - latest status subprocess work stays bounded an

... [3570 bytes truncated] ...

 rendered-tail fallback
ok - herdr's native busy verdict reads working with no record present
ok - a herdr CLI that fails to answer reads unknown/unreachable, never gone
ok - an alive endpoint whose scrollback read failed stays working
ok - a husk pane (agent gone) still reads gone for reclaim
ok - a mid-tool-call crew stays working because its record outranks herdr's generation state
ok - an idle record with idle agent_status stays not-busy (no regression for a human-blocked agent)
ok - no run + idle pane uses the status-log verb
ok - no run + idle pane parses keyed status syntax
ok - no run + idle pane on a paused: status reports state: paused with its reason
ok - no run + idle pane honors the configured paused verb
ok - a trailing resolved: event does not corrupt state render (idle stays idle)
ok - dead window ignores stale status log
ok - a tmux that fails to answer reads unknown/unreachable, never gone
ok - closed pane still reports a terminal run-step
ok - closed pane still reports an active run-step
ok - no timeout command uses perl bound
ok - scout skips the run lookup
ok - torn-down worktree is handled gracefully
ok - fm-crew-state remote: alive endpoint falls through to the routed status log
ok - fm-crew-state remote: an idle alive endpoint reads alive, never gone or dead
ok - fm-crew-state remote: an unreachable host reads unknown-remote, never gone or dead
ok - fm-crew-state remote: the remote host's own dead verdict is reported truthfully
ok - missing meta is handled gracefully
ok - crew_is_provably_working absorbs a validating crew found only via the runs-list fallback
ok - crew_is_provably_working still surfaces a genuinely stopped crew (safety property preserved)
ok - usage error exits 2
ok - historical same-branch rewritten head is not attributed as current
ok - active run with valid descendant fix head remains current
ok - local work advanced past run head invalidates attribution
ok - pipeline-owned active run binds without head equality and beats the failed row
ok - a genuinely failed run with no later run is not hidden
ok - coarse scan anchors the unresolvable active row instead of falling to an older one
ok - coarse scan with a mismatched anchor stays unknown and lets the pane answer
ok - coarse terminal row at a foreign head is not attributed
ok - an executing run binds regardless of branch_sync state
ok - a parked run keeps the strict head rule without pipeline_owned
ok - run_parked_scalar_gate_running keeps the strict head rule despite its live status word
ok - run_parked_in_gate_block keeps the strict head rule despite its live status word
ok - the exemption never applies to a terminal run
ok - missing run head falls back instead of matching by branch
ok - active fix round with an unfetched pipeline head reads working
ok - unanchored unverifiable active row is attributed because it is live
ok - unresolvable terminal row never reads as current
ok - runs-list continuation attribution works when axi answers another branch
ok - herdr stale registration over a shell-only pane reads agent gone, not alive
ok - herdr stale working record never reports a shell-only pane busy
ok - capped overview retains both competing same-branch run ids
ok - same-branch identity survives both runs falling outside the overview
ok - a capped overview with zero same-branch rows reports absent, not unreadable
ok - no run for this branch beside a live run elsewhere reads absent, not unreadable
ok - the capped inventory reader is bounded by the crew read budget
ok - a repo spelling the inventory does not record reads unreadable
ok - a linked worktree green PR in merge monitoring reads held for merge
ok - complete inventory preserves the replacement gate without writes
ok - missing complete-inventory failure reports unknown
ok - corrupt complete-inventory failure reports unknown
ok - schema complete-inventory failure reports unknown
ok - repo complete-inventory failure reports unknown
ok - count complete-inventory failure reports unknown
ok - norepo complete-inventory failure reports unknown
ok - R6 complete selection ignores unrelated branch semantics
ok - R6 requested branches use exact identity without a whitelist
ok - R6 capped inventory ignores unrelated semantics and names both ids
ok - R6 complete identity lookup precedes partial-row semantic rejection
ok - R6 capped inventory preserves quoted requested-branch identity
ok - R6 structural completeness and requested-run validation remain enforced
ok - R5 complete inventory without Python keeps the replacement gate
ok - R5 complete ambiguity without Python names both ids
ok - R5 capped lookup without Python preserves available ids
ok - R5 capped lookup without SQLite support preserves available ids
ok - R1 both directions of inventory liveness disagreement read unknown
ok - R2 uninitialized busy workers retain pane reporting
ok - R2 uninitialized idle workers retain status reporting
ok - R3 historical inventory yields to the current busy pane
ok - R3 historical inventory yields to current worker status
ok - superseded cancelled run preserves the replacement review gate
ok - a live rebased run beats an older failed run at the local head
ok - pending run with a rebased head reads working
ok - running run with a rebased head reads working
ok - legacy live rebased run is authoritative over an older failed row
ok - legacy surface binds a fixing run at a rebased head
ok - legacy surface binds a ci run at a rebased head
ok - an unproven record at a diverged head does not answer for the crew
ok - an unproven record with a dead daemon never overrides a busy pane
ok - an ordinary blocked tip over a coarse live row keeps the superseded reading
ok - a socket-refused blocker survives the dead-daemon verdict
ok - an ordinary blocked tip survives the dead-daemon verdict
ok - a parked gate survives a dead daemon with its findings intact
ok - an unproven record at a diverged head does not answer on the selected route
ok - a live record at a diverged head binds while the daemon answers
ok - the anchored continuation binds while the daemon answers
ok - a head-tied coarse record keeps its working reading and its original note
ok - a coarse failed record with a dead daemon reads unknown
ok - the selected-run anchored continuation reports the dead daemon, not an identity failure
ok - the selected-run anchored continuation binds while the daemon answers
ok - the selected route keeps an anchored parked run's gate with a dead daemon
ok - an open decision survives the dead-daemon verdict on the selected route
ok - a coarse pending ledger word reads unknown
ok - an unanswered probe never turns a failed coarse record into a gate
ok - the selected-route dead-daemon verdict names the run once
ok - a head-tied row reads working when axi names the self run
ok - a head-tied row reads working when axi names the other run
ok - a head-tied coarse row is exempt even when the record head diverged
ok - an unrecognised ledger word keeps the ordinary supersede note
ok - an unanswered daemon probe leaves a live rebased run bound
ok - a head-tied coarse live row is exempt from the dead-daemon verdict
ok - a coarse live row over an open decision keeps the original supersede note
ok - a coarse live row at a rebased head is not attributed
ok - a terminal run at a diverged head keeps the strict head rule
ok - competing live runs report unknown with both run ids
ok - active current run beats a stale overview row
ok - older live status does not hide a newer completed run
ok - newer failed run remains failed beside an older live run
ok - missing run selection reports unknown with candidate ids
ok - wrong-id run selection reports unknown with candidate ids
ok - wrong-branch run selection reports unknown with candidate ids
ok - wrong-head run selection reports unknown with candidate ids
ok - missing-status run selection reports unknown with candidate ids
ok - malformed-table run selection reports unknown with candidate ids
ok - inventory-error run selection reports unknown with candidate ids
ok - selected-error run selection reports unknown with candidate ids
ok - legacy conflicting run records report unknown
all fm-crew-state tests passed
- Outcome: ⚠️ 1 warning across 1 run (5m33s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 warning
  • ⚠️ live validation verdict: inconclusive (0 of 3 scenarios were driven live against the product); untested: Live current run beats stale terminal overview row, Older live run does not hide a newer completed run; newer failed run stays failed, Ambiguous, mismatched or unreadable run selection reports unknown with candidate ids
  • Live validation: ⚠️ inconclusive - 0 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Live current run beats stale terminal overview row ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; it did not establish a live result against a running no-mistakes daemon.
Older live run does not hide a newer completed run; newer failed run stays failed ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; the shell test cannot start a real daemon, so no live result was established.
Ambiguous, mismatched or unreadable run selection reports unknown with candidate ids ⏸️ untested no The prior payload only ran fixture-driven shell tests with live=false; no live result was established.
  • bash tests/fm-crew-state.test.sh
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Changes run selection logic in the crew state system.

The PR appears safe to merge; no outstanding review findings were identified.

Reviews (6) · Last reviewed commit: "test: cover both overview orders for sam..."

Comment thread bin/fm-crew-state.sh
Comment thread tests/fm-crew-state.test.sh
Comment thread bin/fm-nm-run-lib.sh Outdated
@greptile-apps

greptile-apps Bot commented Sep 28, 2026

Copy link
Copy Markdown

Want your agent to iterate on Greptile's feedback? Try greploops.

@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-crewstate-run-identity branch from 7b5aacd to 7d28337 Compare September 29, 2026 10:45
@mehulbhagwani mehulbhagwani changed the title fix: prefer current run over stale overview row fix(bin): use ledger order for same-branch run precedence in crew-state Sep 29, 2026
@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-crewstate-run-identity branch from 7d28337 to cc232ce Compare October 1, 2026 11:47
mehulbhagwani and others added 5 commits October 2, 2026 18:31
Document and pin newer-first (recorded AXI) and historical-first overview
orders. Same-minute ledger timestamps use row order as the tie-break so an
older live bare status cannot hide a newer completed run.
@mehulbhagwani
mehulbhagwani force-pushed the fm/fm-bug-crewstate-run-identity branch from cc232ce to f33f82f Compare October 2, 2026 13:13
@mehulbhagwani mehulbhagwani changed the title fix(bin): use ledger order for same-branch run precedence in crew-state fix(bin): keep newest run attribution in crew-state Oct 2, 2026
@mehulbhagwani

Copy link
Copy Markdown
Author

The one red job here (Behavior portable serial 4) is tests/fm-remote-delta-read.test.sh, the timing-dependent test from #6363 tracked in #6404; this PR does not touch it. Could you re-run that failed job? I don't have rerun rights on this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant