fix(bin): stop wedge alarms on settled terminal tasks - #40
Merged
Merged
Conversation
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
sbracewell64
added a commit
that referenced
this pull request
Aug 9, 2026
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
sbracewell64
added a commit
that referenced
this pull request
Aug 9, 2026
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
sbracewell64
added a commit
that referenced
this pull request
Aug 10, 2026
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
sbracewell64
added a commit
that referenced
this pull request
Aug 11, 2026
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
sbracewell64
added a commit
that referenced
this pull request
Aug 11, 2026
A task whose reconciled current state is terminal kept producing
possible-wedge stale wakes, because the stale path decided from the status
line's TEXT rather than the reconciled state. Seventeen such wakes were
recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task); every one found nothing to do, because an idle
pane is the correct condition for those tasks: a done task with its PR open
and green, a task parked on a captain decision with its work committed, a
task blocked on a credential.
fm-classify-lib.sh's crew_absorb_class already performs the one
fm-crew-state.sh read on that path, so the settled verdict is a fourth token
from that same read rather than a new call:
done nothing is left for the crew to do on its own.
parked, blocked the next move belongs above the crew, but ONLY while a
durable open decision (status_open_decisions) proves it.
Both supervisors consume that one verdict. The watcher absorbs a settled
stale with no wedge timer and records .settled-<key> so unchanged polls of
the same pane hash absorb without re-reading the crew state; any pane change
drops the marker and reclassifies. The away daemon self-handles the wake,
drops rather than records wedge tracking, and re-checks the reconciled state
immediately before escalating an aged marker, which is where a task that
settles after going stale would otherwise still alarm.
Genuine wedges are unaffected:
- failed is deliberately NOT settled, because it also reconciles a CANCELLED
run, the mid-supersession state in which a crew must recover custody and
resume, so an idle pane there is a real stall.
- unknown never settles, so a dead endpoint or torn-down worktree keeps aging.
- a run parked at a gate the crew must answer ITSELF opens no decision, so it
keeps aging and still escalates.
- reading the reconciled state rather than the log text preserves the earlier
precedence fix: a leftover pre-validation done: line under an active run
reconciles as working and keeps its wedge timer.
Nothing rots as a result: an absorb does not mark the status surfaced, so the
terminal status still reaches firstmate through the captain-relevant signal
path and, if that was missed, through the heartbeat catch-all scan, while a
parked or blocked task keeps its open decision surfacing on every wake drain.
Every new test was witnessed failing on unmodified code first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Delivery disclosure
This shipped direct-PR without the no-mistakes pipeline and therefore carries NO attestation marker.
It has not been through automated code review, the pipeline's test/lint/docs gates, or any pipeline-driven fix cycle.
The evidence below is a local test run plus this PR's own CI, reported as-is, including every failure and where each one pre-exists on the base.
Problem
A task whose reconciled current state is terminal kept producing possible-wedge stale wakes, because the stale path decided from the status line's text rather than the reconciled state.
Seventeen such wakes were recorded on 2026-08-04 (joined ledger outcomes, defect
stale-fires-on-finished-task), the second most expensive supervision defect measured that day. Every one fired on a task that wasdone,parked, orpausedwith the work complete; supervision read the state, found nothing to do, and recordedfalse-positive. An idle pane is the correct condition for those tasks, so the wake carried no information: a done task with its PR open and green awaiting merge approval, a task parked on a captain decision with work committed, a task blocked on a credential.Fourteen of the seventeen are the reconciled-terminal shape this change addresses. The other three (
seq 3144,3238,3240) describe an idle pane with a live test shell or long-running tool call underneath; their own ledger notes attribute them towedge-detection-ignores-child-processes. Those still age and escalate here, correctly, and are out of scope.Change
bin/fm-classify-lib.sh'screw_absorb_classalready performs the onefm-crew-state.shread on that path, so the newsettledverdict is a fourth token from that same read rather than a new call:doneparked,blockedstatus_open_decisions) proves the next move belongs above the crewBoth supervisors consume that one verdict, so they cannot drift:
bin/fm-watch.sh): absorbs a settled stale with no wedge timer and recordsstate/.settled-<key>so unchanged polls of the same pane hash absorb without re-reading the crew state. Any pane change drops the marker and reclassifies.bin/fm-supervise-daemon.sh):classify_stalereturns a newsettledaction that self-handles and drops rather than records wedge tracking, andhousekeepingre-checks the reconciled state immediately before escalating an aged marker, which is where a task that settles after going stale would otherwise still alarm.Why this shape
The alternative was to extend the seen-marker/terminal test on the status text. That is the cheaper-looking option but the wrong one: status text is exactly what is already wrong here, and it cannot see the reconciled state. It would have either missed the parked cases or re-broken the precedence fix that the existing crew-state read exists to protect. Consulting the reconciled state costs zero additional
fm-crew-state.shcalls on the watcher path, because that read already happens there once per newly-classified stale hash.Daemon cost is bounded and stated in-code: one read per stale wake in
classify_stale(the watcher enqueues at most one stale wake per distinct pane hash, the same bound the always-on path pays), plus one read in housekeeping only at the moment of escalation.Genuine wedges are preserved
This is the load-bearing property; breaking it would trade cheap false alarms for a silent stall.
failedis deliberately NOT settled. It also reconciles a cancelled run, the mid-supersession state in which a crew must recover custody and resume, so an idle pane there is a real stall.unknownnever settles, so a dead endpoint or torn-down worktree keeps aging.done:line under an active run reconciles asworkingand keeps its wedge timer.paused:long-cadence recheck and the existing seen-marker dedupe are unchanged.Nothing rots: an absorb deliberately does not mark the status surfaced, so the terminal status still reaches supervision through the captain-relevant signal path and, if that was missed, through the heartbeat catch-all scan, while a parked or blocked task keeps its open decision surfacing on every wake drain via
scan_open_decisions.Verification
Every new test was witnessed failing first against unmodified code on this exact base (fresh
git archiveofad53e97plus the new tests):Results on this head:
bin/fm-lint.shclean (pinned ShellCheck 0.11.0),bin/fm-doc-audience-check.shclean.The two locally failing suites, reported as failures
Both are proven pre-existing on this base, not asserted. Each was run against a pristine
git archiveofad53e97, with the relevant production and test files verified byte-identical to the base commit, and each fails there with the identical assertion and exit 1.tests/fm-watcher-lock.test.sh-restart did not attach to the verified healthy peer. Verified pristine:bin/fm-watch.sh,bin/fm-classify-lib.sh,tests/fm-watcher-lock.test.sh.tests/fm-session-start.test.sh-MISSING diagnostic did not appear at all. Verified pristine:bin/fm-bootstrap.sh,tests/fm-session-start.test.sh.Both of these pass in CI (they run in the
portable-seriallane, in shards that passed), so their local failures are specific to the machine this ran on. Either way they are not caused by this change and are not fixed by it.CI on this PR
12 passed, 2 failed of 14. Both failures are accounted for:
PR must be raised via no-mistakes- FAIL, and expected. This is the attestation gate. It fails precisely because this shipped direct-PR without the pipeline and carries no attestation marker, as disclosed at the top. It is not a code failure.Behavior portable serial 2- FAIL, pre-existing on the base. The failing suite istests/fm-remote-secondmate-lifecycle-e2e.test.sh(FM_TEST_END ... exit=1,FM_TEST_SUMMARY_FAMILY family=secondmate ... failed=1). The base commit's own push run onmain(run30999882189, sameBehavior portable serial 2shard) fails on the identical suite with the identical family summary. Not caused by this change.The family covering every file this PR touches reports clean in the same shard:
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=5 ... failed=0, and this PR's own new classifier test is visible passing in the CI log.Behavioural coverage added:
donewith an idle pane produces no stale wedge alarm, on first sight and past the wedge thresholdparkedon an open captain decision produces nonepaused:external wait still self-handles on its long cadenceNotes for review
This branch was rebased across the trunk reconciliation.
AGENTS.mdanddocs/architecture.mdeach had a one-line collision with work already on the trunk; both were resolved keeping both behaviours (.pr-dirty-*alongside.settled-*; the conflicting-PR wake alongside the settled-terminal exclusion). Nothing on the trunk was reverted.While the fork and upstream trunks were still diverged, this branch briefly targeted a base whose
status_open_decisionshad no symlink guard and noscan_open_decisions, and the in-code rationale was narrowed accordingly so it would not claim a mechanism that base lacked. The reconciled base carries both again, so the fuller rationale has been restored and the symlink concern no longer applies.