Skip to content

fix(worktrees): sweep archived merged and no-PR worker worktrees - #103

Merged
ANT1OFF merged 5 commits into
mainfrom
feat/worktree-cleanup
Nov 2, 2026
Merged

ANT1OFF merged 5 commits into
mainfrom
feat/worktree-cleanup

Conversation

@ANT1OFF

@ANT1OFF ANT1OFF commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Why

The disk was nearly full and 66 Gram worktrees had piled up, mostly from finished, merged and archived workers. The sweep had already freed most of their node_modules, but it never removed them. Two causes showed up on a real machine:

Leftovers Why the sweep kept them
~40 archived workers whose PR merged (e.g. #60, #61, #98, #99) Once GitHub answers that an archived thread's PR merged, the publisher drops that thread's state. badgeFor then returns undefined, workerPrBadges gives [undefined], anyPrOpen reads the unknown PR as open, and mergedAndDone is false. So the 10-minute MERGED_GRACE_MS never applied, and every merged archived worker waited the 14-day removeAfterDays. All of the Gram 1.x leftovers were 7 days old or less.
~15 archived workers with no PR of their own (reviewers, research, release cuts on a detached HEAD) remove needs hasPr, so these were only ever freed. A test named it a rule: "never removes a worker with no PR".

Not gaps, checked and left alone:

  • local-checks is already pruned by local-checks.ts retention: passes are kept 30 days and other checks 7, logs are capped at 256 MB, and abandoned work folders are reclaimed after an hour. On this machine it was 57 MB with work/ empty.
  • Worktrees from the 0.x app (gram-private) belong to another app's store. The 1.x sweep never knew about them.

What changes

  • Archived merged workers: a parked thread completed on a merge (completedReason: "prMerged", completedAt set) that follows no extra PRs and has no badge still open counts as merged-done. It goes after MERGED_GRACE_MS. checkWorktree still asks GitHub about the bound PR afresh and needs it merged or closed, the worktree clean, and HEAD on a remote.
  • Archived workers with no PR: a parked worker with no bound PR, no badge and no extra PRs gets remove a day (ARCHIVED_GRACE_MS) after its last activity, marked withoutPr. sweepStep removes it only when it's clean, HEAD is on a remote, and GitHub answered that no PR is open for its branch (noPrOpen). A failed lookup keeps it. Plain (non-worker) threads with no PR still aren't removed, and neither are unarchived workers.
  • A process working in the worktree keeps it. Right before it frees node_modules or removes a worktree, the sweep lists every process's working folder: lsof -d cwd on macOS, /proc/*/cwd on Linux. If any is inside the worktree, it does nothing (inUse / freeUnlessInUse). If it can't tell (Windows, a failed scan), it also does nothing. Thread state alone can't see an agent still running after an archive, a dev server or an outside shell.
  • The Settings › Storage toggle copy names the new rule.

Nothing else changed: removal still goes through closeAtRest, which refuses a dirty tree, a busy thread or a running terminal. Threads held by others, threads waiting on the user and orchestrators are still skipped, and no branch is ever deleted.

Independent review fixes (findings 1–5)

# Finding Fix Refusal test
1 An adopted worker in a user-made worktree outside .worktrees got removed Every thread, workers included, is swept only directly inside the repo's own .worktrees/ Adopted worker outside, merged worker elsewhere, nested folder and unknown repo are never due
2 status.showUntrackedFiles=no hid untracked files from the clean check and from git worktree remove STRICT_STATUS_ARGS (--untracked-files=all --ignore-submodules=none --ignored=no) in the sweep, the Storage probe and worktreeCleanliness; git worktree remove runs with -c status.showUntrackedFiles=all Real repo with the config set: the sweep reads it dirty, removeWorktreeDir refuses, the file survives
3 Pr.status returns undefined when gh isn't ready, which read as "no PR open" No-PR removal needs gh pr list --state open itself to succeed and list none (by branch, or by commit when detached). Pr.status no longer feeds it Signed out, missing, timed out, garbage or open PR: kept
4 Closing the thread deleted its pending wakes A thread with a wake isn't removed (checked when judged and again right before the close); freeing still runs No-PR and merged threads with a wake: removal not due
5 The grace counted from last activity, not from the archive Thread.parkedAt, stamped by the store when an edit archives a thread and dropped on unarchive; grace counts from max(activity, parkedAt). Threads archived before this wait the full M days Quiet 3 days, archived 1 h ago: not due. Store test for the stamp

Also from the review's lower-severity notes: a detached HEAD is kept under refs/gram/swept/<thread id> before its worktree goes (if that can't be written, the worktree stays), a free step scans processes once instead of twice, and the Storage copy is fixed. For each finding, putting the old line back makes its new test fail.

Verification

  • At 1a656a2: bunx vitest run packages/contracts packages/git packages/server/src/git packages/server/src/persistence apps/web/src/settings: 179 files, 1935 passed, 16 skipped. lint, typecheck, fmt:check, check:scaffold pass.

  • bunx vitest run packages/server/src/git/: 33 files, 326 passed, 13 skipped. worktree-gc.test.ts is 19/19.

  • Live-process test: a real sleep runs inside a temp worktree's node_modules. The sweep's free step leaves every file in place, and frees the folder once the process exits. With the guard line taken out, this test fails.

  • A worker in the same state as feat-voice-vocab (open, cold between turns, no PR, never completed) is asserted never due.

  • bunx vitest run packages/server/src/git/ apps/web/src/settings: 76 files, 682 passed, 13 skipped

  • The new tests run against the old worktree-gc.ts fail 6 of 15, including "removes an archived merged worker after the grace, though its badge was dropped". With the fix, 15/15 pass.

  • bun run typecheck, bun run lint, bun run fmt, bun run check:scaffold pass.

  • Full bun run check is left to CI and the merge gate. The machine is shared.

Known limits

  • A reviewer worktree on a detached HEAD whose PR branch was deleted and pruned from the remote reads as unpushed and is kept. That's the safe side. It didn't happen on this machine, so it isn't handled here.
  • Ignored files (e.g. .env.local, notes in ignored folders) aren't counted by any clean check and go with git worktree remove, as they already did for the close dialog and Storage cleanup.
  • The process guard sees only processes whose working folder is inside the worktree, not ones holding files open there from elsewhere (an editor, a bun --cwd run). An idle thread's live agent session or a shell left at its prompt keeps its worktree until the session is evicted.
  • Removal closes the thread. A branch it had stays, and a detached HEAD is kept under refs/gram/swept/. restore_worktree can't bring back a worktree that had no branch.
  • On Windows the process guard can't tell, so the sweep acts on nothing there. No Windows releases are published.
  • The process check, like the thread check, runs right before the step and isn't held across the deletion (the existing "known and accepted gap").
  • Not live-qualified against a running app. Covered by unit tests and git-backed tests with a stubbed gh.

Archived workers whose PR merged waited the full 14 days instead of the
10-minute merged grace: the publisher drops an archived thread's badge once
GitHub answers merged, so the bound PR read as unknown and so as open. The
thread's own completion on a merge now stands in, and GitHub is still
asked afresh before removal.

Archived workers with no PR of their own (reviewers, research and release
workers) were only ever freed, never removed. They now go a day after
their last activity once clean, pushed and with no PR open for the branch;
a failed lookup keeps them.
Before freeing node_modules or removing a worktree, the sweep now lists
every process's working folder (lsof on macOS, /proc on Linux) and leaves
the worktree be when any is inside it, or when that can't be told. The
thread state alone can't see an agent still running after an archive, a
dev server or an outside shell.
…n unknown PR

Review findings 1–3 on #103:

- Every thread, a worker adopting a user-made worktree included, is swept
  only in the repo's own `.worktrees/` folder.
- The sweep's clean check, Settings › Storage's probe and the shared
  `worktreeCleanliness` run `git status --porcelain --untracked-files=all
  --ignore-submodules=none --ignored=no`, and `git worktree remove` without
  --force runs with `status.showUntrackedFiles=all`, so a user's
  `status.showUntrackedFiles=no` no longer hides untracked files that the
  removal then deletes.
- A worker with no PR is removed only on gh's own successful answer listing
  no open PR for its branch (or detached HEAD). `Pr.status` returns
  `undefined` when gh isn't ready, so it no longer feeds that decision; a
  failed, signed-out or missing gh keeps the worktree.
…p detached heads

Review findings 4–5 on #103, and two of its lower-severity notes:

- A thread with a pending wake is never removed: closing it deletes
  `thread_wakes`. The sweep reads the wakes when it judges a thread and
  again right before the close; freeing node_modules still goes ahead.
- Threads record `parkedAt`, stamped by the store's `updateThread` when an
  edit archives them and dropped on unarchive. A no-PR worker's one-day
  grace counts from max(last activity, parkedAt); one archived before this
  was recorded waits the full removeAfterDays from its last activity.
- Before a detached-HEAD worktree is removed, its HEAD is kept under
  refs/gram/swept/<thread id>, so a commit held only by a remote branch
  that's later deleted stays reachable. If that can't be written, the
  worktree stays.
- A free step scans processes once, under its lock, not twice. Storage copy
  loses its stray apostrophe.
@ANT1OFF
ANT1OFF merged commit b9ed8d4 into main Nov 2, 2026
23 checks passed
@ANT1OFF
ANT1OFF deleted the feat/worktree-cleanup branch November 2, 2026 23:08
ANT1OFF added a commit that referenced this pull request Nov 3, 2026
Desktop release metadata proposal only. Merging this PR does not publish
or tag Gram; follow docs/RELEASING.md to release.
---


## [1.5.0](v1.4.0...v1.5.0)
(2026-11-03)


### Features

* **voice:** a normal Voice setting with a push-to-talk key picker
([#98](#98))
([a49acdf](a49acdf))
* **voice:** recognise this project's names
([#102](#102))
([d306e35](d306e35))
* **voice:** spoken symbols, formatting and filler clean-up
([#99](#99))
([912b5e5](912b5e5))
* **voice:** voice input on Linux with a TTS-free sherpa-onnx build
([#100](#100))
([4606b96](4606b96))


### Bug Fixes

* **release:** stop the release notes saying voice is missing on Linux
([#107](#107))
([fbde4f2](fbde4f2))
* **worktrees:** sweep archived merged and no-PR worker worktrees
([#103](#103))
([b9ed8d4](b9ed8d4))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: gram-release-please[bot] <351321620+gram-release-please[bot]@users.noreply.github.com>
Co-authored-by: Johan <32440065+ANT1OFF@users.noreply.github.com>
ANT1OFF added a commit that referenced this pull request Nov 3, 2026
…#111)

## What

Two new condition kinds for `schedule_wake`, built on the mechanism from
#108, plus a fix to that mechanism's engine.

- **`ciStuck`** `{ repo, pr? | runId?, queuedMinutes? }`: fires when a
hosted Actions job of the PR or run has sat queued with no runner for
`queuedMinutes` (default 15). It also fires when a run awaits a
maintainer's approval, as a fork PR's does (#76 ran no CI and nobody
noticed). `pr` defaults to the thread's bound PR in that repo. A run
that completes ends the wait, and the notice says so. Facts: `job`,
`queued` ("27 min, no runner"), `url`, and `alsoQueued` when more jobs
are queued. For an approval: `state: awaits approval` and the run's URL.
- **`machine`**, two shapes:
- `{ metric: "load", below, forMinutes? }`: the 1-minute load average
stays below `below` for `forMinutes` (default 5). The number is
**absolute**, as `uptime` shows it, not per CPU. It is what people read
off `uptime`/`top`, and today's numbers ("timed out at load ~28") were
absolute. Use it so heavy checks wait for a quiet Mac.
- `{ metric: "diskFree", belowGB, path? }`: free space on the volume
holding `path`, by default the thread's repo, drops below `belowGB`. GB
here means 10^9 bytes, as Finder and `df -H` count. Use it so cleanup
starts before the disk fills.

Chip labels: `CI job queued 15+ min`, `quieter (load < 8)` and `disk <
20 GB`, next to the chip's alarm icon. The `when` lines: `a CI job of
acme/app#7 sits queued 15+ min, or a run awaits approval`, `this
machine's load stays below 8 for 5 min` and `free disk space at /path
drops below 20 GB`.

### Engine fix (`wakes/conditions.ts`)

Each kind's sweep cleared the debounce state (`metSince` and the pending
debounce look) of every wake it hadn't just seen. But a sweep only sees
its own kind. So any other kind's sweep inside a debounce wiped it, and
a debounced wake fired only if no other kind happened to sweep during
its debounce. The new kinds' looks run every 30 s, which made this
constant. Now a sweep clears only its own kind's entries.

Reproduction: `a run awaiting approval fires…` in
`service-condition-wakes-ci-machine.test.ts` sweeps `machine` and
`branchMoved` inside a `ciStuck` debounce. It fails with #108's
`conditions.ts` and passes with this one. I ran both.

## Polling and API cost

| Kind | Poll | Debounce / hold | Calls per poll |
| --- | --- | --- | --- |
| `ciStuck` on a PR | 2 min | 1 min debounce | **0 or 1 GraphQL**
(`Pr.checks`, the read the PR monitors share; reused if anyone read it
in the last minute, and its 10 s cache answers every wake on the PR).
Only when the PR has **no Actions check at all**: +1 REST
(`actions/runs?status=action_required`), and +1 REST (`pulls/N` for its
head) only if that list isn't empty. |
| `ciStuck` on a run | 2 min | 1 min debounce | **1 REST**
(`runs/{id}/jobs`), reused for 20 s across wakes on the run. +1 REST
(`runs/{id}`) only once nothing of the run is in flight, to tell an
approval from a finish. |
| `machine` load | 30 s | `forMinutes` hold (in memory) | none:
`os.loadavg()` in the server process |
| `machine` diskFree | 30 s, but each volume is read at most every 5 min
| none | none: `fs.statfs()` in the server process |

Every GitHub call is admitted against the background share of the
allowance (`GhBudget.admit(…, "background")`) and runs under
`asBackground`. If `gh` can't be reached, is logged out, or is out of
allowance, the look returns "couldn't tell": it neither fires nor
throws. As before, a kind with nothing pending doesn't poll. The run
monitor reads `runs/{id}` and never the jobs list of a running run, so
it doesn't duplicate the run case. For PRs, `Pr.seenChecks` and
`Pr.checks` are the shared path.

When GitHub gives a queued job no time, its queued time counts from the
first look that saw it. That time is kept in the wake's `seen`, so a
restart doesn't reset it (tested).

## Arm-time refusals

- `ciStuck`: refused when given both `pr` and `runId`; with neither and
no PR bound in that repo; for a run GitHub doesn't have; when a job is
already queued past the limit or a run already awaits approval; and for
non-integer or out-of-range numbers (`queuedMinutes` must be 1–1440).
- `machine` load: refused when the 1-, 5- and 15-minute averages, as far
as they cover `forMinutes`, already show the load below. Beyond 15
minutes the averages can't tell, so it doesn't refuse.
- `machine` diskFree: refused when free space is already below, or
`path` isn't absolute or can't be read.

## Worktree sweep (#103) on low disk: not done, follow-up

Running the sweep early would only do what is already due, and the sweep
runs every 15 min anyway, so it gains at most 15 minutes. Freeing space
that matters under disk pressure would mean shortening
`freeAfterDays`/`removeAfterDays` (removing `node_modules` or worktrees
sooner). That changes a user setting's meaning and needs consent, so it
isn't a small safe change. Suggested follow-up: a disk-pressure mode,
with the user's opt-in, that frees finished threads' `node_modules`
early. Meanwhile an agent can set `{kind:"machine", metric:"diskFree",
belowGB}` and act on the wake.

## Prompt budget (#89)

The two kinds add about 980 characters per role: about 600 of schema and
380 of description, after trimming. To save room:

- numeric fields use `Schema.Finite`, whose JSON schema is
`{"type":"number"}`. `Int`'s bounds and `Number`'s Infinity/NaN `anyOf`
are each ~65 characters. The source refuses a fraction.
- `schedule_wake`'s `inMinutes` now uses `Schema.Finite`, which rejects
Infinity and NaN at decode.
- the condition sentence names fields instead of repeating object
literals.

Even so, worker (35,201) and plain (24,370) went over, so I **raised
their limits by 1,000**, to 35,700 and 24,900, as the test's header asks
("raise one only on purpose, in the change that needs the room, and say
so there"). The orchestrator stays under 62,400 at 61,307. The
merge-freeze and release-version workers add kinds in parallel and will
need room of their own.

## Code layout

New files, to stay out of the parallel kinds' way: `wakes/ci-stuck.ts`
(the `CiJobs` service, `CiJobsLive`, `judgeCiStuck`, `ciStuckSource`)
and `wakes/machine.ts` (the `MachineState` service, `MachineStateLive`,
`machineSource`). `folderOf` moved to `wakes/condition-folders.ts` so
the sources share it. The test harness fakes `CiJobs` (`world.ciStates`)
and `MachineState` (`world.machineLoad`, `world.diskSpace`).

## Verification

- `bun run test` on `service-condition-wakes-ci-machine` (6),
`wakes/ci-stuck` (7), contracts `wake-conditions` (3), and
`service-condition-wakes`, `prompt-budget`, `service-wakes`,
`service-digests`, `service-restarts`, `tool-ergonomics`, `rules`,
`tools`, `http/server`, contracts `orchestration`: 13 files, 603 tests
pass. The web `thread-view` test passes (16).
- Each kind fires once: a stuck PR job, a fork run awaiting approval, a
completed run, load, disk.
- Debounce: a job picked up inside the debounce doesn't fire.
`forMinutes`: a load spike restarts the hold.
- Unreachable gh: `CiJobsLive` over a `gh` that won't start, and
`Pr.checks` failing, returns `undefined` and makes no finding. In the
engine, an unreadable PR neither fires nor fails.
- Restart: a job's first-seen queued time survives a restart and fires
after it.
- API calls: the number of `gh` calls per read is asserted, including
zero when the shared checks are under a minute old.
  - Labels and `when` lines are tested in contracts.
- `bun run typecheck`, `bun run lint`, `bun run fmt`, `bun run
check:docs`, `bun run check:scaffold`: clean.
- I didn't run a full `bun run check` locally; CI gates it.

## Known limits

- All GitHub behaviour is behind `CiJobs`, which the tests fake
(`CiJobsLive` is unit-tested over a fake `gh` and `Pr`). There was no
live-provider qualification. Two GitHub details are assumed: that
GraphQL `CheckRun.startedAt` is the queue time for a queued Actions job,
and that a fork's approval-pending run has `status`/`conclusion`
`action_required` with no check runs on the PR. If `startedAt` is
absent, the first-seen time is used, which only delays the wake.
- For a PR, "queued" is the check's phase, which also covers `WAITING`
(environment approval) and `PENDING` (concurrency). Those count as stuck
too, after the same minutes. Non-Actions statuses are ignored.
- A `ciStuck` wake on a PR that merges or closes waits until its
fallback time, polling one GraphQL point every 2 min. A run's completion
does end its wait.
- The load hold lives in memory, so a restart starts it again. The app
can't know the load while it was down.
- Windows has no load average, so a load wake is refused there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant