Repository navigation
fix: speed up session startup and escalate home-summary failures - #51
Merged
Merged
Conversation
MrGTV-love
force-pushed
the
fm/fm-session-start-fast
branch
from
October 8, 2026 13:28
ea05f11 to
38fe9c2
Compare
…escalate repeating failures A locked session start spent its whole 60-second refresh deadline on a publication that needed 117 seconds, so every start paid the deadline, the refresh never published, and 425 identical failures reached nobody. - Session start, spawn, and teardown run the refresh with the new --detach: single-flight, detached, never waited on. A trigger that finds a refresh in flight leaves a pending marker and the in-flight run refreshes once more, so bursts coalesce into one follow-up and none is lost. - Three consecutive refresh failures append one check: home-summary-refresh wake naming the reason and measured duration; it is raised again only when the reason changes or after a success ends the streak. - The refresh itself folds each task's status log from the checkpoint the wake drain keeps beside it instead of from line 1 (hunks from the lifecycle checkpoint work, reused as written). - Bootstrap reads backlog row states once instead of once per owned record, canonicalizes paths and checks control bytes in the shell, and the herdr journal reader no longer forks per field; the wake drain drops per-line tr and dirname/basename forks. - FM_SESSION_START_STAGE_TIMES_FILE times a session start stage by stage.
…move optional corpus mode
…watcher test timing
… regression oracle independent
…cklog-atomicity.test.sh. Its aggregate show-call limit incorrectly rejected two legitimate reads of the queued task: an eligibility probe and a full-body read for drop-provenance handling. The test now records command and task ID and asserts that none of the five nonqueued tasks is read individually, in both ordinary and comma-containing home paths. Existing one-list, queued-healing, state, and metadata assertions remain intact; production behavior is unchanged. Verification: the complete backlog atomicity suite passed locally with tasks-axi 0.2.6, including the previously failing scenario. An initial run passed that scenario but hit the final secondmate fixture's repository-containment guard; rerunning with the harness's normal ephemeral temp location passed the full suite. Temporary verification fixtures were removed. No other pipeline phase or push was invoked
… documentation ownership
MrGTV-love
force-pushed
the
fm/fm-session-start-fast
branch
from
October 8, 2026 16:14
ecf4abd to
a3dea5c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
we also need to fix an issue with the session start hooks. you increased the time to 180s, but even 120s was too long to get a session started. here is an analysis. HOME_SUMMARY failed 295x and never did anythign about it. another exaple of why ignoring and working around problems instead of solving them is the wrong thing to do.
What Changed
Risk Assessment
Testing
Live validation passed across startup latency, checkpoint migration, detached publication, lifecycle triggers, failure escalation, recovery, and concurrency. Focused regression runs exposed two fixture assumptions; the corrected scenarios, ownership checks, and checkpoint regressions passed. Evidence consists of real CLI transcripts and persisted ledgers; this change has no visual-layout surface requiring screenshots.
Evidence: Live scenario manifest and validation setup
Source: Live scenario manifest and validation setup
Evidence: Live refresh outcomes, streaks, wakes, and published ledgers
Source: Live refresh outcomes, streaks, wakes, and published ledgers
Evidence: Upgraded-startup timing
Source: Upgraded-startup timing
Evidence: Blocked-summary startup timing
Source: Blocked-summary startup timing
Evidence: Detached ledger published after releasing validation
Source: Detached ledger published after releasing validation
Evidence: Batched backlog reconciliation and concurrent lock acquisition
Source: Batched backlog reconciliation and concurrent lock acquisition
Evidence: Partial-checkpoint drain and publication behavior
Source: Partial-checkpoint drain and publication behavior
Pipeline
Updates from git push no-mistakes
... (12 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)
🔧 **Review** - 6 issues found → auto-fixed (5) ✅
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
🔧 **Test** - 4 issues found → auto-fixed ✅
bin/fm-classify-lib.sh:358- First-upgrade startup still reproduces the reported latency problem. A real Claude primary with a valid version-9 checkpoint and an 840,058-byte status log containing 24,000 routine lines hit the 120-second startup bound at 120.784s in wake-queue. The buried decision and remaining digest sections were not delivered. Version 10 correctly rejects the old checkpoint, but the incremental cold fold still invokes a subprocess for every status line. Optimize cold/refused-checkpoint folding without trusting obsolete checkpoints or increasing the startup timeout, and retain an executable large-history migration regression.tests/fm-home-summary-refresh.test.sh:708- The focused refresh script remains failing at its accumulated-home publication scenario: 300 wide working lines plus the cost-gate decision exceeded the configured 30-second deadline and produced no ledger. Fixture-root isolation was corrected, and the preceding modified timeout, initialization, and logging checks passed. A separate real no-endpoint variant published the same history shape in 12.006s, so that narrower live result does not resolve the fuller owned-worker case. Diagnose the classification/fixture difference rather than increasing the deadline.bin/fm-home-summary-refresh.sh:217- A directory at state/home-summary.json produces false publication success. The real writer moved its temporary JSON inside that directory, returned zero, and left no failure streak while the public ledger path remained unreadable as JSON. This bypasses the new failure accounting. Attribution to this change has not been established, but the exercised publication boundary should reject a directory destination and account for the acquired failure.Scoped the change withgit diff --stat 90172034c69544cbdf8e37ece135976a6e8f16b9 04b38188cc1dc715a5e07ad71829d70b8a0c8b13; read the trusted Herdr lab contracts and checked required tools on PATH.TMPDIR=$WORKTREE/.validation-summary/tmp bash tests/fm-home-summary-refresh.test.shandbash tests/fm-home-summary-refresh-ownership.test.sh; initial runs exposed fixture-root isolation and incidental timing assumptions.TMPDIR=$WORKTREE/.validation-primary/tmp bash tests/fm-wake-drain-open-decisions-cursor.test.sh— passed.Ran the focused summary scripts from an unchanged disposable code copy with sibling temporary homes inside the worktree, avoiding the secondmate-inside-code-root refusal.bash .validation-primary/focused-test-world/code/tests/fm-home-summary-refresh-ownership.test.sh— passed after the test-only deadline correction.bash .validation-primary/focused-test-world/code/tests/fm-home-summary-refresh.test.sh— modified failure-handling checks passed; the final run failed accumulated-home publication at its 30-second deadline.Provisioned and tore downfm-lab-start-fast-77979-1475andfm-lab-upgrade-cold-53699-6156only throughbin/fm-herdr-lab.sh; both teardown tripwires verified the default session remained unchanged.From real Claude primaries, executedbash -c 'TIMEFORMAT="STARTUP_RUNTIME_SECONDS=%R"; time ./bin/fm-session-start.sh'with a held summary lock and with a previous-version long-history checkpoint.FM_BOOTSTRAP_NETWORK=skip bin/fm-bootstrap.shagainst a marked comma-containing home, using a transparent tracer delegating to installed tasks-axi; observed one list and one show while healing the queued record.Executed realbin/fm-wake-drain.shandbin/fm-fleet-snapshot.sh --secondmate-home-summaryacross cached tails, partial appends, invalid boundaries, log replacement, terminal closure, and reopening.Executed realbin/fm-home-summary-refresh.sh --best-effortwith an immutable ledger for three-failure escalation, fourth-failure deduplication, recovery, and re-escalation; also exercised an unavailable logger with a full stderr pipe.Executed five real--detachtriggers during successive refreshes; verified the same owner drained all pending transitions.Paused an actual refresh worker, killed its diagnostic parent, published a newer ledger, then resumed the stale worker; verified the newer ledger remained byte-identical.Executed the real writer against a directory ledger destination and a separate 300-wide-line marked-home fixture without backend mocks; preserved resulting product state and removed all disposable fixtures.🔧 Fix applied.
✅ Re-checked - no issues remain.
bash tests/fm-home-summary-refresh-ownership.test.sh— passed.bash tests/fm-wake-drain-open-decisions-cursor.test.sh— passed.bash tests/fm-home-summary-refresh.test.sh— initially failed because workspace-local temporary homes were nested under the code root; the unchanged test passed asbash .live-validation-emr992ci/code-root/tests/fm-home-summary-refresh.test.shusing a disposable sibling code root.Launched real Claude primaries on private, non-zero-sized tmux sessions; disposable SessionStart settings invokedbin/fm-session-start.sh --source startupandbin/fm-session-start.sh --reemit --source clear.Drovebin/fm-home-summary-refresh.sh,--best-effort, and--detachagainst accumulated histories, directory destinations, held refresh locks, blocked logging, and actual one-second deadlines.Killed a real refresh parent while its worker was suspended, published newer state, recorded a newer failure, and resumed the orphaned worker.Drovebin/fm-wake-drain.shthrough decision opening, terminal completion, reopening, authoritative-log replacement, and partial-line completion.Used realtasks-axi add,list, andshowcommands plusbin/fm-bootstrap.shto recover an owned queued task with numeric-looking IDs and a comma-containing home path.Provisioned a named Herdr lab throughbin/fm-herdr-lab.sh; scaffolded withbin/fm-brief.sh lifecycle batch-project --mode local-only --herdr-lab; drove real long-running task spawn and teardown while summary publication was blocked.Ran the real summary writer from Claude SessionStart with the pane's actual private-socket TMUX environment, an owned endpoint, 300 wide status lines, and real no-mistakes state reads.Tore down Herdr with its default-fleet tripwire intact, stopped private tmux servers, removed their helper-owned socket directories, destroyed the exact test-created Treehouse worktree, and removed disposable worktree fixtures.✅ No issues found.
Inspected the runtime changes against base commitcb7205a65d14a1d45338c26a030ae551ed6ec5dband read the trusted Herdr lab contracts.Provisioned two named non-default sessions throughbin/fm-herdr-lab.sh provision; drove their real Claude processes through the helper and removed both through guardedteardown.A real Claude primary ranFM_HOME=<marked lab> bin/fm-session-start.sh --source startupwith a version-9 checkpoint, 24,000 routine status lines, and a held summary-refresh lock.Drovebin/fm-home-summary-refresh.sh,--best-effort, and--detachagainst real isolated producers for accumulated history, appended decisions, directory destinations, failure escalation, recovery, overlapping triggers, blocked logging, and parent cancellation.Drovebin/fm-wake-drain.shacross an incomplete status line, its completion, resolution, and a changed resolution-verb override.Used the installedtasks-axiCLI and realbin/fm-bootstrap.shwith numeric/boolean-looking IDs and a comma-containing home path; acquired lifecycle locks concurrently while bootstrap remained running.Ran realbin/fm-spawn.sh live-scout <disposable project> --scout --harness claude --backend herdr, thenbin/fm-captain-hold.sh complete live-scout --noneandbin/fm-teardown.sh live-scout, while summary-refresh ownership remained held.bash tests/fm-home-summary-refresh-ownership.test.shpassed.bash tests/fm-home-summary-refresh.test.shinitially failed because repository-local temporary secondmate homes violated the supported isolation guard; after collecting the actual failed-state diagnostic,TMPDIR=/tmp/ bash tests/fm-home-summary-refresh.test.shpassed.bash tests/fm-wake-drain-open-decisions-cursor.test.shinitially exceeded an external 300-second runner limit; the unchanged focused script subsequently completed successfully in 469.16 seconds with an 1800-second external limit. The product startup deadline was not increased.Preserved product digests, JSON ledgers, wake/streak state, lifecycle transcripts, and focused regression logs in the evidence directory; removed transient fixtures and generated startup state.✅ No issues found.
Launched real Claude primaries withtmux -L fm-lab new-sessionon disposable private sockets, with 120×40 terminal dimensions.Drovebin/fm-session-start.sh --source startupthrough Claude against a version-9 checkpoint and a 936,060-byte status history containing 24,000 routine lines; startup completed in 10.494s.Drovebin/fm-session-start.sh --reemit --source clearthrough Claude; completed in 8.281s with the buried decision retained.Drove startup through another real Claude primary while detached summary validation was blocked; startup completed in 8.563s, and publication succeeded after releasing the barrier.Executedbin/fm-home-summary-refresh.sh,--best-effort, and--detachagainst disposable homes to exercise directory-destination failures, escalation, recovery, blocked FIFO logging, five pending follow-ups, parent cancellation, and acquired validation timeouts.Usedbin/fm-herdr-lab.shto provision and tear down named non-default labs; exercised realbin/fm-spawn.shandbin/fm-teardown.shcalls while summary publication was blocked.Executedbin/fm-bootstrap.shwith real tasks-axi records named123,true,false, andnullin a comma-containing home; verified queued-row healing and concurrent lifecycle-lock availability.Executedbin/fm-wake-drain.shand the real summary writer across a partial-line checkpoint, completed append, resolution, and later decision append.Ranbash tests/fm-home-summary-refresh-ownership.test.shandbash tests/fm-wake-drain-open-decisions-cursor.test.sh; both passed.Ran focusedbash tests/fm-home-summary-refresh.test.shchecks. Diagnosed and corrected competing-watcher and short-deadline fixture assumptions, then executed the corrected escalation, recovery, and large-history sections verbatim through a disposable targeted driver; those sections passed.Stopped private primary sockets, tore down Herdr labs, returned disposable worker slots, and removed repository-local fixtures and diagnostic drivers. No full repository suite, linters, formatters, or static analysis ran.✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
🔧 **Lint** - 1 issue found → auto-fixed (2) ✅
🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.