Repository navigation
fix(worktrees): sweep archived merged and no-PR worker worktrees - #103
Merged
Merged
Conversation
Archived workers whose PR merged waited the full 14 days instead of the 10-minute merged grace: the publisher drops an archived thread's badge once GitHub answers merged, so the bound PR read as unknown and so as open. The thread's own completion on a merge now stands in, and GitHub is still asked afresh before removal. Archived workers with no PR of their own (reviewers, research and release workers) were only ever freed, never removed. They now go a day after their last activity once clean, pushed and with no PR open for the branch; a failed lookup keeps them.
Before freeing node_modules or removing a worktree, the sweep now lists every process's working folder (lsof on macOS, /proc on Linux) and leaves the worktree be when any is inside it, or when that can't be told. The thread state alone can't see an agent still running after an archive, a dev server or an outside shell.
…n unknown PR Review findings 1–3 on #103: - Every thread, a worker adopting a user-made worktree included, is swept only in the repo's own `.worktrees/` folder. - The sweep's clean check, Settings › Storage's probe and the shared `worktreeCleanliness` run `git status --porcelain --untracked-files=all --ignore-submodules=none --ignored=no`, and `git worktree remove` without --force runs with `status.showUntrackedFiles=all`, so a user's `status.showUntrackedFiles=no` no longer hides untracked files that the removal then deletes. - A worker with no PR is removed only on gh's own successful answer listing no open PR for its branch (or detached HEAD). `Pr.status` returns `undefined` when gh isn't ready, so it no longer feeds that decision; a failed, signed-out or missing gh keeps the worktree.
…p detached heads Review findings 4–5 on #103, and two of its lower-severity notes: - A thread with a pending wake is never removed: closing it deletes `thread_wakes`. The sweep reads the wakes when it judges a thread and again right before the close; freeing node_modules still goes ahead. - Threads record `parkedAt`, stamped by the store's `updateThread` when an edit archives them and dropped on unarchive. A no-PR worker's one-day grace counts from max(last activity, parkedAt); one archived before this was recorded waits the full removeAfterDays from its last activity. - Before a detached-HEAD worktree is removed, its HEAD is kept under refs/gram/swept/<thread id>, so a commit held only by a remote branch that's later deleted stays reachable. If that can't be written, the worktree stays. - A free step scans processes once, under its lock, not twice. Storage copy loses its stray apostrophe.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
Desktop release metadata proposal only. Merging this PR does not publish or tag Gram; follow docs/RELEASING.md to release. --- ## [1.5.0](v1.4.0...v1.5.0) (2026-11-03) ### Features * **voice:** a normal Voice setting with a push-to-talk key picker ([#98](#98)) ([a49acdf](a49acdf)) * **voice:** recognise this project's names ([#102](#102)) ([d306e35](d306e35)) * **voice:** spoken symbols, formatting and filler clean-up ([#99](#99)) ([912b5e5](912b5e5)) * **voice:** voice input on Linux with a TTS-free sherpa-onnx build ([#100](#100)) ([4606b96](4606b96)) ### Bug Fixes * **release:** stop the release notes saying voice is missing on Linux ([#107](#107)) ([fbde4f2](fbde4f2)) * **worktrees:** sweep archived merged and no-PR worker worktrees ([#103](#103)) ([b9ed8d4](b9ed8d4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: gram-release-please[bot] <351321620+gram-release-please[bot]@users.noreply.github.com> Co-authored-by: Johan <32440065+ANT1OFF@users.noreply.github.com>
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
…#111) ## What Two new condition kinds for `schedule_wake`, built on the mechanism from #108, plus a fix to that mechanism's engine. - **`ciStuck`** `{ repo, pr? | runId?, queuedMinutes? }`: fires when a hosted Actions job of the PR or run has sat queued with no runner for `queuedMinutes` (default 15). It also fires when a run awaits a maintainer's approval, as a fork PR's does (#76 ran no CI and nobody noticed). `pr` defaults to the thread's bound PR in that repo. A run that completes ends the wait, and the notice says so. Facts: `job`, `queued` ("27 min, no runner"), `url`, and `alsoQueued` when more jobs are queued. For an approval: `state: awaits approval` and the run's URL. - **`machine`**, two shapes: - `{ metric: "load", below, forMinutes? }`: the 1-minute load average stays below `below` for `forMinutes` (default 5). The number is **absolute**, as `uptime` shows it, not per CPU. It is what people read off `uptime`/`top`, and today's numbers ("timed out at load ~28") were absolute. Use it so heavy checks wait for a quiet Mac. - `{ metric: "diskFree", belowGB, path? }`: free space on the volume holding `path`, by default the thread's repo, drops below `belowGB`. GB here means 10^9 bytes, as Finder and `df -H` count. Use it so cleanup starts before the disk fills. Chip labels: `CI job queued 15+ min`, `quieter (load < 8)` and `disk < 20 GB`, next to the chip's alarm icon. The `when` lines: `a CI job of acme/app#7 sits queued 15+ min, or a run awaits approval`, `this machine's load stays below 8 for 5 min` and `free disk space at /path drops below 20 GB`. ### Engine fix (`wakes/conditions.ts`) Each kind's sweep cleared the debounce state (`metSince` and the pending debounce look) of every wake it hadn't just seen. But a sweep only sees its own kind. So any other kind's sweep inside a debounce wiped it, and a debounced wake fired only if no other kind happened to sweep during its debounce. The new kinds' looks run every 30 s, which made this constant. Now a sweep clears only its own kind's entries. Reproduction: `a run awaiting approval fires…` in `service-condition-wakes-ci-machine.test.ts` sweeps `machine` and `branchMoved` inside a `ciStuck` debounce. It fails with #108's `conditions.ts` and passes with this one. I ran both. ## Polling and API cost | Kind | Poll | Debounce / hold | Calls per poll | | --- | --- | --- | --- | | `ciStuck` on a PR | 2 min | 1 min debounce | **0 or 1 GraphQL** (`Pr.checks`, the read the PR monitors share; reused if anyone read it in the last minute, and its 10 s cache answers every wake on the PR). Only when the PR has **no Actions check at all**: +1 REST (`actions/runs?status=action_required`), and +1 REST (`pulls/N` for its head) only if that list isn't empty. | | `ciStuck` on a run | 2 min | 1 min debounce | **1 REST** (`runs/{id}/jobs`), reused for 20 s across wakes on the run. +1 REST (`runs/{id}`) only once nothing of the run is in flight, to tell an approval from a finish. | | `machine` load | 30 s | `forMinutes` hold (in memory) | none: `os.loadavg()` in the server process | | `machine` diskFree | 30 s, but each volume is read at most every 5 min | none | none: `fs.statfs()` in the server process | Every GitHub call is admitted against the background share of the allowance (`GhBudget.admit(…, "background")`) and runs under `asBackground`. If `gh` can't be reached, is logged out, or is out of allowance, the look returns "couldn't tell": it neither fires nor throws. As before, a kind with nothing pending doesn't poll. The run monitor reads `runs/{id}` and never the jobs list of a running run, so it doesn't duplicate the run case. For PRs, `Pr.seenChecks` and `Pr.checks` are the shared path. When GitHub gives a queued job no time, its queued time counts from the first look that saw it. That time is kept in the wake's `seen`, so a restart doesn't reset it (tested). ## Arm-time refusals - `ciStuck`: refused when given both `pr` and `runId`; with neither and no PR bound in that repo; for a run GitHub doesn't have; when a job is already queued past the limit or a run already awaits approval; and for non-integer or out-of-range numbers (`queuedMinutes` must be 1–1440). - `machine` load: refused when the 1-, 5- and 15-minute averages, as far as they cover `forMinutes`, already show the load below. Beyond 15 minutes the averages can't tell, so it doesn't refuse. - `machine` diskFree: refused when free space is already below, or `path` isn't absolute or can't be read. ## Worktree sweep (#103) on low disk: not done, follow-up Running the sweep early would only do what is already due, and the sweep runs every 15 min anyway, so it gains at most 15 minutes. Freeing space that matters under disk pressure would mean shortening `freeAfterDays`/`removeAfterDays` (removing `node_modules` or worktrees sooner). That changes a user setting's meaning and needs consent, so it isn't a small safe change. Suggested follow-up: a disk-pressure mode, with the user's opt-in, that frees finished threads' `node_modules` early. Meanwhile an agent can set `{kind:"machine", metric:"diskFree", belowGB}` and act on the wake. ## Prompt budget (#89) The two kinds add about 980 characters per role: about 600 of schema and 380 of description, after trimming. To save room: - numeric fields use `Schema.Finite`, whose JSON schema is `{"type":"number"}`. `Int`'s bounds and `Number`'s Infinity/NaN `anyOf` are each ~65 characters. The source refuses a fraction. - `schedule_wake`'s `inMinutes` now uses `Schema.Finite`, which rejects Infinity and NaN at decode. - the condition sentence names fields instead of repeating object literals. Even so, worker (35,201) and plain (24,370) went over, so I **raised their limits by 1,000**, to 35,700 and 24,900, as the test's header asks ("raise one only on purpose, in the change that needs the room, and say so there"). The orchestrator stays under 62,400 at 61,307. The merge-freeze and release-version workers add kinds in parallel and will need room of their own. ## Code layout New files, to stay out of the parallel kinds' way: `wakes/ci-stuck.ts` (the `CiJobs` service, `CiJobsLive`, `judgeCiStuck`, `ciStuckSource`) and `wakes/machine.ts` (the `MachineState` service, `MachineStateLive`, `machineSource`). `folderOf` moved to `wakes/condition-folders.ts` so the sources share it. The test harness fakes `CiJobs` (`world.ciStates`) and `MachineState` (`world.machineLoad`, `world.diskSpace`). ## Verification - `bun run test` on `service-condition-wakes-ci-machine` (6), `wakes/ci-stuck` (7), contracts `wake-conditions` (3), and `service-condition-wakes`, `prompt-budget`, `service-wakes`, `service-digests`, `service-restarts`, `tool-ergonomics`, `rules`, `tools`, `http/server`, contracts `orchestration`: 13 files, 603 tests pass. The web `thread-view` test passes (16). - Each kind fires once: a stuck PR job, a fork run awaiting approval, a completed run, load, disk. - Debounce: a job picked up inside the debounce doesn't fire. `forMinutes`: a load spike restarts the hold. - Unreachable gh: `CiJobsLive` over a `gh` that won't start, and `Pr.checks` failing, returns `undefined` and makes no finding. In the engine, an unreadable PR neither fires nor fails. - Restart: a job's first-seen queued time survives a restart and fires after it. - API calls: the number of `gh` calls per read is asserted, including zero when the shared checks are under a minute old. - Labels and `when` lines are tested in contracts. - `bun run typecheck`, `bun run lint`, `bun run fmt`, `bun run check:docs`, `bun run check:scaffold`: clean. - I didn't run a full `bun run check` locally; CI gates it. ## Known limits - All GitHub behaviour is behind `CiJobs`, which the tests fake (`CiJobsLive` is unit-tested over a fake `gh` and `Pr`). There was no live-provider qualification. Two GitHub details are assumed: that GraphQL `CheckRun.startedAt` is the queue time for a queued Actions job, and that a fork's approval-pending run has `status`/`conclusion` `action_required` with no check runs on the PR. If `startedAt` is absent, the first-seen time is used, which only delays the wake. - For a PR, "queued" is the check's phase, which also covers `WAITING` (environment approval) and `PENDING` (concurrency). Those count as stuck too, after the same minutes. Non-Actions statuses are ignored. - A `ciStuck` wake on a PR that merges or closes waits until its fallback time, polling one GraphQL point every 2 min. A run's completion does end its wait. - The load hold lives in memory, so a restart starts it again. The app can't know the load while it was down. - Windows has no load average, so a load wake is refused there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The disk was nearly full and 66 Gram worktrees had piled up, mostly from finished, merged and archived workers. The sweep had already freed most of their
node_modules, but it never removed them. Two causes showed up on a real machine:badgeForthen returnsundefined,workerPrBadgesgives[undefined],anyPrOpenreads the unknown PR as open, andmergedAndDoneis false. So the 10-minuteMERGED_GRACE_MSnever applied, and every merged archived worker waited the 14-dayremoveAfterDays. All of the Gram 1.x leftovers were 7 days old or less.removeneedshasPr, so these were only ever freed. A test named it a rule: "never removes a worker with no PR".Not gaps, checked and left alone:
local-checksis already pruned bylocal-checks.tsretention: passes are kept 30 days and other checks 7, logs are capped at 256 MB, and abandoned work folders are reclaimed after an hour. On this machine it was 57 MB withwork/empty.What changes
completedReason: "prMerged",completedAtset) that follows no extra PRs and has no badge still open counts as merged-done. It goes afterMERGED_GRACE_MS.checkWorktreestill asks GitHub about the bound PR afresh and needs it merged or closed, the worktree clean, and HEAD on a remote.removea day (ARCHIVED_GRACE_MS) after its last activity, markedwithoutPr.sweepStepremoves it only when it's clean, HEAD is on a remote, and GitHub answered that no PR is open for its branch (noPrOpen). A failed lookup keeps it. Plain (non-worker) threads with no PR still aren't removed, and neither are unarchived workers.node_modulesor removes a worktree, the sweep lists every process's working folder:lsof -d cwdon macOS,/proc/*/cwdon Linux. If any is inside the worktree, it does nothing (inUse/freeUnlessInUse). If it can't tell (Windows, a failed scan), it also does nothing. Thread state alone can't see an agent still running after an archive, a dev server or an outside shell.Nothing else changed: removal still goes through
closeAtRest, which refuses a dirty tree, a busy thread or a running terminal. Threads held by others, threads waiting on the user and orchestrators are still skipped, and no branch is ever deleted.Independent review fixes (findings 1–5)
.worktreesgot removed.worktrees/status.showUntrackedFiles=nohid untracked files from the clean check and fromgit worktree removeSTRICT_STATUS_ARGS(--untracked-files=all --ignore-submodules=none --ignored=no) in the sweep, the Storage probe andworktreeCleanliness;git worktree removeruns with-c status.showUntrackedFiles=allremoveWorktreeDirrefuses, the file survivesPr.statusreturnsundefinedwhenghisn't ready, which read as "no PR open"gh pr list --state openitself to succeed and list none (by branch, or by commit when detached).Pr.statusno longer feeds itThread.parkedAt, stamped by the store when an edit archives a thread and dropped on unarchive; grace counts from max(activity, parkedAt). Threads archived before this wait the full M daysAlso from the review's lower-severity notes: a detached HEAD is kept under
refs/gram/swept/<thread id>before its worktree goes (if that can't be written, the worktree stays), a free step scans processes once instead of twice, and the Storage copy is fixed. For each finding, putting the old line back makes its new test fail.Verification
At 1a656a2:
bunx vitest run packages/contracts packages/git packages/server/src/git packages/server/src/persistence apps/web/src/settings: 179 files, 1935 passed, 16 skipped. lint, typecheck, fmt:check, check:scaffold pass.bunx vitest run packages/server/src/git/: 33 files, 326 passed, 13 skipped.worktree-gc.test.tsis 19/19.Live-process test: a real
sleepruns inside a temp worktree'snode_modules. The sweep's free step leaves every file in place, and frees the folder once the process exits. With the guard line taken out, this test fails.A worker in the same state as
feat-voice-vocab(open, cold between turns, no PR, never completed) is asserted never due.bunx vitest run packages/server/src/git/ apps/web/src/settings: 76 files, 682 passed, 13 skippedThe new tests run against the old
worktree-gc.tsfail 6 of 15, including "removes an archived merged worker after the grace, though its badge was dropped". With the fix, 15/15 pass.bun run typecheck,bun run lint,bun run fmt,bun run check:scaffoldpass.Full
bun run checkis left to CI and the merge gate. The machine is shared.Known limits
.env.local, notes in ignored folders) aren't counted by any clean check and go withgit worktree remove, as they already did for the close dialog and Storage cleanup.bun --cwdrun). An idle thread's live agent session or a shell left at its prompt keeps its worktree until the session is evicted.refs/gram/swept/.restore_worktreecan't bring back a worktree that had no branch.gh.