Repository navigation
fix: reduce home-summary refresh overhead - #61
Merged
Merged
Conversation
…stages The home-summary refresh overran its 60-second deadline on a busy host because the producer (fm-fleet-snapshot.sh --secondmate-home-summary) spends its time starting processes, not computing. On the live fleet (27 tasks, 53 contribution records) a run costs about 12 s user and 25 s system CPU, and the system share is process creation. Stages whose cost grows with the fleet, and the fix for each: - Per-task record composition ran about 16 jq processes and about 25 metadata subshells per task, serially. Each task is now one jq program fed by in-shell reads (metadata keys, last status line, first PR URL, path tests), and records are composed in a rolling window. - Observation prefetch waited for a whole batch of 8 before starting the next, so one slow task idled seven slots. It now keeps a rolling window of the same size. - Contribution coverage validated each record with three jq runs and a wc. It now reads each record with one jq run, finds oversized records with one find, and runs beside the observation stages because it depends only on the backlog and task metadata. - The contribution task list resolved merge authority and ran jq once per task. Merge authority is a property of the away record, so it is resolved once, and the list is built by one jq. Every published field and its meaning are unchanged: the summary, home input, contribution input and full snapshot are byte-identical to the previous producer on a fixed fixture fleet. Measured on the live fleet, producer only, old vs new wall time: load 103-115: 29.7 s -> 19.4 s load 129-159: 33.1 s -> 23.0 s load 166-177: 35.4 s -> 22.0 s, 33.3 s -> 19.4 s A fixture fleet of the same size starts 518 jq processes on the previous producer and 115 now; tests/fm-fleet-snapshot-fleet-cost.test.sh bounds that and fails on the previous producer.
… inventory comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I would like all bugs to be fixed so tomorrow can be focused entirely to Vernant and not problems preventning Vernant from getting built.
Context: the captain earlier reported that session start took over 120 s and that the home-summary refresh had failed hundreds of times with nothing done. PR #51 (merged 2026-10-08 16:44Z) stopped session start, spawn and teardown from waiting on the refresh and added an alarm for repeated failures. The refresh itself still exceeds its 60-second deadline: 5 failures between 16:45Z and 16:55Z, so state/home-summary.json is still not republished. Main timed one run at 51 s while host load was 170 on 18 CPUs; an earlier measurement was 117 s at load 54.
What Changed
Timing: before and after
The 60-second deadline in
bin/fm-home-summary-refresh.shis unchanged. This change cuts the cost of the producer (bin/fm-fleet-snapshot.sh --secondmate-home-summary) so it finishes well inside that deadline at the current fleet size. Every published field and meaning is kept: the summary bytes are identical between the two versions on the same live fleet (39,893 bytes in all six runs below).Live fleet (27 tasks, 53 contribution records, 18 CPUs). Each pair ran back to back, main first, on the same host in the same minute. "Load" is the 1-minute load average read just before each run.
origin/main, 5ea3f5b): wallEarlier live runs on the same fleet at heavier load: main 29.7 to 35.4 s and this change 19.4 to 23.0 s at load 103 to 177. At the heaviest load the wall time rises for both, because the producer's cost on a busy host is process creation (system CPU is about twice user CPU), not computation.
Process starts on the fixture fleet of the same size (
tests/fm-fleet-snapshot-fleet-cost.test.sh): 518 jq starts on main, 115 here. The test bounds jq starts at 200 and fails on main.Reading these numbers honestly:
fm-crew-state.shstill makesno-mistakes axicalls for each task (about 60% of the remaining CPU); done ships still make one GitHub read each; and a cold fold of the two large status logs costs 17 to 24 s if a decision checkpoint (.open-decisions-cursor) becomes invalid.Risk Assessment
✅ Low: The change is a bounded optimization with targeted correctness fixes, and the full source review found no substantiated material defects or intent conflicts.
Testing
Base/target comparison, targeted executable regressions, and isolated live CLI scenarios demonstrated reduced snapshot cost, timely valid publication, prompt detached triggering, completion-order scheduling, safe contribution reads, and owned-child cleanup. Disposable-driver setup and assertion issues were documented and corrected or isolated. CLI transcripts and public JSON outputs were retained; no visual UI changed, so screenshots were not applicable.
Evidence: Measured base/target cost and refresh results
Source: Measured base/target cost and refresh results
Evidence: Fleet benchmark, publication, and detached-trigger transcript
Source: Fleet benchmark, publication, and detached-trigger transcript
Evidence: Valid published active-fleet summary
Source: Valid published active-fleet summary
Evidence: Active-fleet refresh under CPU pressure
Source: Active-fleet refresh under CPU pressure
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
bin/fm-fleet-snapshot.sh- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 1 issue found → auto-fixed (4) ✅
bin/fm-fleet-snapshot.sh:2196- The contribution PID remains tracked after wait completes. During intended execution, contributions can finish before the slower observations; if its PID is subsequently recycled, the EXIT cleanup sends SIGTERM to an unrelated process, potentially another fleet worker. Clear CONTRIBUTIONS_PID after consuming the child's exit status, and restrict early-error cleanup to the still-running contribution job rather than an unchecked saved PID. Related changed sites: bin/fm-fleet-snapshot.sh:2168 stores the handle; bin/fm-fleet-snapshot.sh:2196 consumes it without clearing it; bin/fm-fleet-snapshot.sh:1635 unconditionally signals the retained number.🔧 Fix applied.
2 warnings still open:
bin/fm-fleet-snapshot.sh:2196- The contribution PID remains tracked after wait completes. During intended execution, contributions can finish before the slower observations; if its PID is subsequently recycled, the EXIT cleanup sends SIGTERM to an unrelated process, potentially another fleet worker. Clear CONTRIBUTIONS_PID after consuming the child's exit status, and restrict early-error cleanup to the still-running contribution job rather than an unchecked saved PID. Related changed sites: bin/fm-fleet-snapshot.sh:2168 stores the handle; bin/fm-fleet-snapshot.sh:2196 consumes it without clearing it; bin/fm-fleet-snapshot.sh:1635 unconditionally signals the retained number.bin/fm-fleet-snapshot.sh:817- The rolling windows still block on the oldest child rather than replenish a freed slot. With concurrency 8, if the first crew-state read reaches its 10-second timeout while reads 2–8 finish after one second, task 9 cannot start for another nine seconds despite seven idle slots. Slow reads at successive window boundaries repeat this delay during the intended fleet refresh, retaining the slow-task slot-idling problem this optimization claims to remove. The same policy appears in task composition at bin/fm-fleet-snapshot.sh:1053. Collect completed children independently of launch order and replenish available slots while preserving the concurrency bound, exit-status handling, and final ID ordering.🔧 Fix applied.
1 warning still open:
bin/fm-contributions.sh:150- The bulk size scan can silently bypass the existing 1 MiB record cap. For example, a searchable but non-readable task directory (mode 0111) containing a readable, valid contributions.json padded beyond the cap is reachable through the glob and readable by jq, but find cannot enumerate it. Suppressing find's failure leaves the record absent from oversized, so pending succeeds instead of refusing and snapshot unreadable_records/complete/proven_clear can report falsely clean coverage. The same bypass occurs with a trailing-slash symlink data root: normalization removes the slash, physical find does not traverse the symlink, but the glob does. Related changed sites: bin/fm-contributions.sh:149 normalizes the root after the symlink guard at :144; :151 independently discovers records; :159-160 treats absence from the scan as size approval; :164 slurps those records without another size check. Preserve scan failure status and fall back to per-file size checks when enumeration is incomplete; apply the existing root-symlink guard to the normalized path.🔧 Fix applied.
2 warnings still open:
bin/fm-contributions.sh:150- The bulk size scan can silently bypass the existing 1 MiB record cap. For example, a searchable but non-readable task directory (mode 0111) containing a readable, valid contributions.json padded beyond the cap is reachable through the glob and readable by jq, but find cannot enumerate it. Suppressing find's failure leaves the record absent from oversized, so pending succeeds instead of refusing and snapshot unreadable_records/complete/proven_clear can report falsely clean coverage. The same bypass occurs with a trailing-slash symlink data root: normalization removes the slash, physical find does not traverse the symlink, but the glob does. Related changed sites: bin/fm-contributions.sh:149 normalizes the root after the symlink guard at :144; :151 independently discovers records; :159-160 treats absence from the scan as size approval; :164 slurps those records without another size check. Preserve scan failure status and fall back to per-file size checks when enumeration is incomplete; apply the existing root-symlink guard to the normalized path.bin/fm-contributions.sh:161- Round 3 fixes incomplete enumeration but leaves stale size approval in the successful-scan path. During the supported overlap between a detached refresh and contribution polling, find can observe a record below 1 MiB, then poll atomically replaces it with an oversized record before read_saved reaches that file. This is reachable because poll retains unbounded review arrays (:296), updates records (:439), and publishes without a size cap (:226). Since find succeeded and the path is absent from oversized, the replacement is slurped and accepted without a size check; pending can succeed where the previous per-file check would refuse, and snapshot omits the required unreadable-record disclosure. Enforce the cap on the record actually consumed at the shared read_saved boundary, rather than using an earlier scan's absence as approval. Related changed sites: bin/fm-contributions.sh:148 captures sizes before processing; :159 rejects only previously oversized paths; :169 consumes the potentially replaced record. docs/architecture.md:147 claims the cap is enforced.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
/bin/bash tests/fm-fleet-snapshot-fleet-cost.test.sh/bin/bash tests/fm-fleet-snapshot-rolling-window.test.sh/bin/bash tests/fm-fleet-snapshot-contribution-cleanup.test.sh/bin/bash tests/fm-contributions-read-saved.test.sh/bin/bash tests/fm-fleet-snapshot-view.test.shpython3 .lv/drive.py: compared base and target producers against 27 real private-tmux windows and 53 contribution records; exercisedfm-home-summary-refresh.sh --best-effortand--detach. Its initial tracing assertion was not counted as passing and was replaced by the focused scheduling proof.python3 .lv/rolling.py: exercised slow-first scheduling, serial/parallel/base output equivalence, ID ordering, and both child-failure paths; passed with the default crew-state timeout.python3 .lv/contributions.py: exercised inclusive size boundaries, permission-limited reads, symlink roots, trailing slashes, and oversized replacement immediately before consumption.python3 .lv/cleanup.py: exercised real contribution-job cleanup on active-child failure, completed-child failure, and successful snapshot publication.python3 .lv/active.py: published a valid 27-active-child summary under four disposable CPU workers. An incorrect throwaway assertion expected the state nameworking; the captured product output correctly reportedactive_child_work, and its validity, counts, and deadline were verified directly.Stopped all private tmux servers and disposable CPU workers, removed.lv, and confirmed the fixture directory was absent.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.