Repository navigation
fix: scope open-work ledger to home-owned obligations - #56
Merged
Merged
Conversation
…e worker stop detection
…nvalidated coverage
…econciliation guidance
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.
Context: the open-work ledger from PR #45 (state/open-loops.json,
check: open-loop-ledger) woke every Vernant lane on 2026-10-08 to route dozens of rows it does not own, which cost lane time on the night meant to clear blockers to Vernant. Evidence is in the backlog item note for fm-open-loops-lane-scope.What Changed
Risk Assessment
✅ Low: The changes are bounded, preserve case-sensitive branch and exact-URL ownership, retain generation-invalidated coverage failures, and introduce no substantiated material defects or unnecessary components.
Testing
Both focused test files passed, and seven live scenarios passed through the real ledger, snapshot, watcher, wake-drain, and Bearings interfaces using disposable workspace-local homes and read-only public GitHub data. Evidence includes published ledgers, forge request traces, watcher output, Bearings output, and generation-race transitions; setup issues were corrected before final observations, and all fixtures were removed.
Evidence: Live validation evidence index and isolation details
Source: Live validation evidence index and isolation details
Evidence: Foreign-fork exclusion and absence of check requests
Source: Foreign-fork exclusion and absence of check requests
Evidence: Case-insensitive head repository ownership
Source: Case-insensitive head repository ownership
Evidence: Owned task failing checks retain diagnosis obligation
Source: Owned task failing checks retain diagnosis obligation
Evidence: Unowned PR lane produces no ledger wake
Source: Unowned PR lane produces no ledger wake
Evidence: Unselected backlog stays quiet while selected work wakes
Source: Unselected backlog stays quiet while selected work wakes
Evidence: Bearings exposes selected overdue work only
Source: Bearings exposes selected overdue work only
Evidence: Stopped workers versus resumed and silently lost workers
Source: Stopped workers versus resumed and silently lost workers
Evidence: Configured pause survives unrelated resolution
Source: Configured pause survives unrelated resolution
Evidence: Stopped ship retains unpublished commit obligation
Source: Stopped ship retains unpublished commit obligation
Evidence: Actual generation races retain coverage and coherent rescans recover
Source: Actual generation races retain coverage and coherent rescans recover
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 4 issues found → auto-fixed (3) ✅
bin/fm_open_loops.py:501- Branch ownership is matched globally across repositories. For example, a lane has a task in repository A on fm/fix and another task in repository B on fm/other; an unrelated PR in B on fm/fix passes owns_pr(), fetches checks, and produces an actionable open_pr/red_check obligation. Task identifiers and branches are home-local, so another lane can legitimately use the same branch name. This leaves the reported cross-lane wake failure reachable. Qualify branch matching by the task's repository identity at owns_pr(), while retaining exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:494 stores unqualified branches; bin/fm_open_loops.py:589 applies the match to every discovered repository; docs/configuration.md:593 documents the overly broad rule.bin/fm_open_loops.py:260- A historical stop event overrides an authoritative non-stopped current state. A worker can append paused while awaiting its validation run, have current_state=working from that active run, and then lose its agent. recorded_stop() still returns true from the historical paused event, suppressing missing_worker and potentially publishing complete:true without a recovery obligation. The existing current-state contract explicitly gives active run/pane evidence precedence over stale status events (bin/fm-crew-state.sh:4-12 and bin/fm-classify-lib.sh:809-811). Make known working/blocked verdicts override historical stop events in this shared predicate. Related changed sites: bin/fm_open_loops.py:270 suppresses the backlog obligation; bin/fm_open_loops.py:299-304 suppresses liveness failures; bin/fm_open_loops.py:307 and :312 suppress current-state degradation and missing-worker rows; docs/configuration.md:599 repeats the unconditional historical-event rule.bin/fm_open_loops.py:259- Recorded-stop detection introduces a second, incomplete interpretation of status events. With FM_CLASSIFY_PAUSED_VERB=awaiting, the generated worker protocol emits awaiting, but STOP_EVENTS recognizes only literal paused. Likewise, a valid multiline paused declaration followed by continuation prose loses its stop classification because the snapshot projects the last physical nonempty line (bin/fm-fleet-snapshot.sh:372-373), not the logical event. Once the endpoint is gone and current_state becomes unknown, both cases incorrectly emit missing_worker. These are supported existing behaviors, covered by tests/fm-crew-state.test.sh:2808-2811 and :2960-2977. Reuse last_status_line/status_is_paused from the existing status-event owner rather than relying on the raw tail verb. Related changed sites: bin/fm_open_loops.py:38 hardcodes the vocabulary; :260 consumes it; :270 and :299-312 apply the resulting classification; docs/configuration.md:599 states the recorded-stop guarantee.bin/fm_open_loops.py:594- Simplification: the new main-home 'unowned project PRs' aggregation is not required to stop lanes receiving obligations they do not own; filtering actionable PR rows already satisfies that correction. It also labels PRs owned by secondmate lanes as having 'no fleet owner', although this collector intentionally reads only its own home and excludes secondmate task records. The smallest remedy is removing this introduced informational aggregation rather than adding cross-home ownership machinery. Related changed sites: bin/fm_open_loops.py:92 and :592 maintain the accumulator; :636 emits the group; docs/configuration.md:588 and :595 describe it; tests/fm-open-loops.test.sh:672-677 assert the new output. Removing this deliberate output component needs author approval.🔧 Fix applied.
6 issues (3 errors, 3 warnings) still open:
bin/fm_open_loops.py:501- Branch ownership is matched globally across repositories. For example, a lane has a task in repository A on fm/fix and another task in repository B on fm/other; an unrelated PR in B on fm/fix passes owns_pr(), fetches checks, and produces an actionable open_pr/red_check obligation. Task identifiers and branches are home-local, so another lane can legitimately use the same branch name. This leaves the reported cross-lane wake failure reachable. Qualify branch matching by the task's repository identity at owns_pr(), while retaining exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:494 stores unqualified branches; bin/fm_open_loops.py:589 applies the match to every discovered repository; docs/configuration.md:593 documents the overly broad rule.bin/fm_open_loops.py:260- A historical stop event overrides an authoritative non-stopped current state. A worker can append paused while awaiting its validation run, have current_state=working from that active run, and then lose its agent. recorded_stop() still returns true from the historical paused event, suppressing missing_worker and potentially publishing complete:true without a recovery obligation. The existing current-state contract explicitly gives active run/pane evidence precedence over stale status events (bin/fm-crew-state.sh:4-12 and bin/fm-classify-lib.sh:809-811). Make known working/blocked verdicts override historical stop events in this shared predicate. Related changed sites: bin/fm_open_loops.py:270 suppresses the backlog obligation; bin/fm_open_loops.py:299-304 suppresses liveness failures; bin/fm_open_loops.py:307 and :312 suppress current-state degradation and missing-worker rows; docs/configuration.md:599 repeats the unconditional historical-event rule.bin/fm_open_loops.py:259- Recorded-stop detection introduces a second, incomplete interpretation of status events. With FM_CLASSIFY_PAUSED_VERB=awaiting, the generated worker protocol emits awaiting, but STOP_EVENTS recognizes only literal paused. Likewise, a valid multiline paused declaration followed by continuation prose loses its stop classification because the snapshot projects the last physical nonempty line (bin/fm-fleet-snapshot.sh:372-373), not the logical event. Once the endpoint is gone and current_state becomes unknown, both cases incorrectly emit missing_worker. These are supported existing behaviors, covered by tests/fm-crew-state.test.sh:2808-2811 and :2960-2977. Reuse last_status_line/status_is_paused from the existing status-event owner rather than relying on the raw tail verb. Related changed sites: bin/fm_open_loops.py:38 hardcodes the vocabulary; :260 consumes it; :270 and :299-312 apply the resulting classification; docs/configuration.md:599 states the recorded-stop guarantee.bin/fm_open_loops.py:594- Simplification: the new main-home 'unowned project PRs' aggregation is not required to stop lanes receiving obligations they do not own; filtering actionable PR rows already satisfies that correction. It also labels PRs owned by secondmate lanes as having 'no fleet owner', although this collector intentionally reads only its own home and excludes secondmate task records. The smallest remedy is removing this introduced informational aggregation rather than adding cross-home ownership machinery. Related changed sites: bin/fm_open_loops.py:92 and :592 maintain the accumulator; :636 emits the group; docs/configuration.md:588 and :595 describe it; tests/fm-open-loops.test.sh:672-677 assert the new output. Removing this deliberate output component needs author approval.bin/fm_open_loops.py:501- Round 1's repository-scoping fix leaves fork PRs falsely owned because slug identifies the base repository, not the repository containing head.ref. Concrete case: this home has origin acme/project and branch fix/build; an unrelated contributor opens a PR into acme/project from contributor/project:fix/build. owns_pr() returns true, fetches its checks, and emits an actionable open_pr/red_check row despite no recorded URL or matching source branch. This remains reachable through the normal open-PR listing, including the supported third-party branch prefixes (bin/fm-brief.sh:60-65). Compare the task-origin identity with head.repo.full_name at the shared owns_pr() boundary, retaining exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:532 stores origin-qualified branches; :584 lists base-repository PRs; :591 applies ownership; docs/configuration.md:593 states the repository-and-branch guarantee; tests/fm-open-loops.test.sh:641 omits head repository identity from ownership fixtures. GitHub's response contract distinguishes head.repo from the requested repository: https://docs.github.com/en/rest/pulls/pulls#list-pull-requests.bin/fm_open_loops.py:258- Round 1's status-parser fix leaves the standing-pause sibling unresolved. A worker records needs-decision [key=choice], then paused while awaiting an independent condition; firstmate answers with fm-send --resolve-key choice, appending resolved [key=choice]. That answer does not retract the pause under the shared event contract (bin/fm-status-event-lib.sh:250-273). If the endpoint subsequently disappears, crew-state reports unknown and recorded_stop() examines only the latest resolved event, returning false and producing a spurious missing_worker obligation. The existing executable scenarios explicitly preserve pauses behind unrelated answers (tests/fm-daemon.test.sh:925-938). Use status_declared_wait_line/status_is_paused for the pause fallback, while retaining latest-event handling for done and known-current-state precedence. Related changed sites: bin/fm_open_loops.py:271 applies the predicate to in-flight backlog rows; :300 obtains it for worker rows; :308 and :313 use it for coverage and recovery obligations; docs/configuration.md:599 documents only latest-event handling; tests/fm-open-loops.test.sh:724-725 covers continuation and resumption but not unrelated resolution.🔧 Fix applied.
8 issues (3 errors, 5 warnings) still open:
bin/fm_open_loops.py:501- Branch ownership is matched globally across repositories. For example, a lane has a task in repository A on fm/fix and another task in repository B on fm/other; an unrelated PR in B on fm/fix passes owns_pr(), fetches checks, and produces an actionable open_pr/red_check obligation. Task identifiers and branches are home-local, so another lane can legitimately use the same branch name. This leaves the reported cross-lane wake failure reachable. Qualify branch matching by the task's repository identity at owns_pr(), while retaining exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:494 stores unqualified branches; bin/fm_open_loops.py:589 applies the match to every discovered repository; docs/configuration.md:593 documents the overly broad rule.bin/fm_open_loops.py:260- A historical stop event overrides an authoritative non-stopped current state. A worker can append paused while awaiting its validation run, have current_state=working from that active run, and then lose its agent. recorded_stop() still returns true from the historical paused event, suppressing missing_worker and potentially publishing complete:true without a recovery obligation. The existing current-state contract explicitly gives active run/pane evidence precedence over stale status events (bin/fm-crew-state.sh:4-12 and bin/fm-classify-lib.sh:809-811). Make known working/blocked verdicts override historical stop events in this shared predicate. Related changed sites: bin/fm_open_loops.py:270 suppresses the backlog obligation; bin/fm_open_loops.py:299-304 suppresses liveness failures; bin/fm_open_loops.py:307 and :312 suppress current-state degradation and missing-worker rows; docs/configuration.md:599 repeats the unconditional historical-event rule.bin/fm_open_loops.py:259- Recorded-stop detection introduces a second, incomplete interpretation of status events. With FM_CLASSIFY_PAUSED_VERB=awaiting, the generated worker protocol emits awaiting, but STOP_EVENTS recognizes only literal paused. Likewise, a valid multiline paused declaration followed by continuation prose loses its stop classification because the snapshot projects the last physical nonempty line (bin/fm-fleet-snapshot.sh:372-373), not the logical event. Once the endpoint is gone and current_state becomes unknown, both cases incorrectly emit missing_worker. These are supported existing behaviors, covered by tests/fm-crew-state.test.sh:2808-2811 and :2960-2977. Reuse last_status_line/status_is_paused from the existing status-event owner rather than relying on the raw tail verb. Related changed sites: bin/fm_open_loops.py:38 hardcodes the vocabulary; :260 consumes it; :270 and :299-312 apply the resulting classification; docs/configuration.md:599 states the recorded-stop guarantee.bin/fm_open_loops.py:594- Simplification: the new main-home 'unowned project PRs' aggregation is not required to stop lanes receiving obligations they do not own; filtering actionable PR rows already satisfies that correction. It also labels PRs owned by secondmate lanes as having 'no fleet owner', although this collector intentionally reads only its own home and excludes secondmate task records. The smallest remedy is removing this introduced informational aggregation rather than adding cross-home ownership machinery. Related changed sites: bin/fm_open_loops.py:92 and :592 maintain the accumulator; :636 emits the group; docs/configuration.md:588 and :595 describe it; tests/fm-open-loops.test.sh:672-677 assert the new output. Removing this deliberate output component needs author approval.bin/fm_open_loops.py:501- Round 1's repository-scoping fix leaves fork PRs falsely owned because slug identifies the base repository, not the repository containing head.ref. Concrete case: this home has origin acme/project and branch fix/build; an unrelated contributor opens a PR into acme/project from contributor/project:fix/build. owns_pr() returns true, fetches its checks, and emits an actionable open_pr/red_check row despite no recorded URL or matching source branch. This remains reachable through the normal open-PR listing, including the supported third-party branch prefixes (bin/fm-brief.sh:60-65). Compare the task-origin identity with head.repo.full_name at the shared owns_pr() boundary, retaining exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:532 stores origin-qualified branches; :584 lists base-repository PRs; :591 applies ownership; docs/configuration.md:593 states the repository-and-branch guarantee; tests/fm-open-loops.test.sh:641 omits head repository identity from ownership fixtures. GitHub's response contract distinguishes head.repo from the requested repository: https://docs.github.com/en/rest/pulls/pulls#list-pull-requests.bin/fm_open_loops.py:258- Round 1's status-parser fix leaves the standing-pause sibling unresolved. A worker records needs-decision [key=choice], then paused while awaiting an independent condition; firstmate answers with fm-send --resolve-key choice, appending resolved [key=choice]. That answer does not retract the pause under the shared event contract (bin/fm-status-event-lib.sh:250-273). If the endpoint subsequently disappears, crew-state reports unknown and recorded_stop() examines only the latest resolved event, returning false and producing a spurious missing_worker obligation. The existing executable scenarios explicitly preserve pauses behind unrelated answers (tests/fm-daemon.test.sh:925-938). Use status_declared_wait_line/status_is_paused for the pause fallback, while retaining latest-event handling for done and known-current-state precedence. Related changed sites: bin/fm_open_loops.py:271 applies the predicate to in-flight backlog rows; :300 obtains it for worker rows; :308 and :313 use it for coverage and recovery obligations; docs/configuration.md:599 documents only latest-event handling; tests/fm-open-loops.test.sh:724-725 covers continuation and resumption but not unrelated resolution.bin/fm_open_loops.py:503- Repository identity matching is case-sensitive although GitHub owner/repository names are not. With task origin https://github.com/acme/vernant.git, branch fm/build, and no recorded PR URL, GitHub can return head.repo.full_name=Acme/Vernant and head.ref=fm/build. owns_pr() then rejects this home's PR, silently omitting its open_pr/red_check obligation and check requests. Round 1's repository-scoping fix introduced this false-negative path, and Round 2's head-repository fix retains it. Normalize only the repository identity on both sides of branch ownership matching; preserve case-sensitive branch matching and exact recorded-URL ownership. Related changed sites: bin/fm_open_loops.py:534 stores the origin spelling; bin/fm_open_loops.py:593 filters classification; docs/configuration.md:593 promises repository-and-branch ownership. GitHub documents both names as case-insensitive: https://docs.github.com/en/rest/repos/repos#get-a-repository.bin/fm_open_loops.py:258- The live status fallback can erase the snapshot's explicit generation-invalidated observation. During a supported relaunch, spawn_gen changes while the fleet snapshot reads a task; the snapshot deliberately returns current_state=unknown with detail='task generation changed during snapshot', endpoint.exists=null, and agent_alive=unknown, discarding mutable observations (bin/fm-fleet-snapshot.sh:692-699). If the retained status log contains a standing pause or the replacement appends paused/done, recorded_stop() rereads that live log and returns true. worker_rows() then suppresses both inconclusive liveness and the generation-change diagnostic, potentially publishing complete:true without a coherent worker observation. Round 1's parser fallback introduced this cross-generation path; Round 2 preserves it. At the shared recorded_stop() boundary, reject stop inference for an explicitly generation-invalidated sample so its coverage failure remains visible. Related changed sites: bin/fm_open_loops.py:271 uses the predicate for backlog recovery; bin/fm_open_loops.py:300-305 suppresses liveness errors; bin/fm_open_loops.py:308 suppresses unknown-state coverage; bin/fm_open_loops.py:313 gates worker recovery.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-open-loops.test.sh— passed the focused ledger contract suite; mock-based cases were supplemental, not counted as live evidence.bash tests/fm-watch-open-loops.test.sh— passed the focused watcher integration suite.bin/fm-lab-home.sh create <workspace-local-home>andFM_HOME=<home> bin/fm-open-loops.sh --heartbeat --json— exercised isolated homes and verified atomic publication against command output.Read-only public GitHub scenarios through the installedgh-axi: foreign-fork exclusion, exact task/backlog URL ownership, repository-case normalization, branch-case sensitivity, owned failing checks, and main-home filtering. A transparent request tracer recorded requests without altering responses.FM_HOME=<home> bin/fm-watch.sh— observed quiet cycles for non-actionable backlog and unowned PRs, and one durable ledger wake for overdue dispatch-started work.FM_HOME=<home> bin/fm-wake-drain.shfollowed by its emitted acknowledgment command — handled the fixture's initial worker signal before observing subsequent ledger silence.FM_HOME=<home> bin/fm-bearings-snapshot.sh --json— confirmed informational backlog was omitted from overdue open-loop obligations.FM_HOME=<home> bin/fm-fleet-snapshot.sh --home-inputand the real ledger — exercised missing endpoints, durable stops, configured multiline pauses, unrelated resolutions, resumed workers, and unpublished Git commits.Changed disposable task metadata generations concurrently with real ledger scans for done, paused, and unrelated-resolution status streams; verified degraded coverage during the race and complete coherent rescans afterward.Removed all disposable fixtures and confirmed no fixture-addressed processes remained. No source files were changed.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.