Repository navigation
feat(orchestration): wake on stuck CI and on machine load or low disk - #111
Merged
Merged
Conversation
Two condition-wake kinds on #108's mechanism: - ciStuck { repo, pr? | runId?, queuedMinutes? }: a hosted Actions job sits queued with no runner for queuedMinutes (default 15), or a run awaits a maintainer's approval (a fork PR's). pr defaults to the thread's bound PR. A run that completes ends the wait. - machine { metric: load, below, forMinutes? } and { metric: diskFree, belowGB, path? }: the 1-minute load average (absolute, as uptime shows it) stays below a number, or free space on the repo's volume drops below some GB. Read in-process with os.loadavg and fs.statfs. Also fixes the engine: a sweep of one kind dropped the pending debounce of every other kind's wakes, so a debounced wake fired only if no other kind swept inside its debounce.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
## What When a worker or an orchestrator is at rest waiting on something, the sidebar now says what, at a glance: an hourglass and a short line under its name, and the whole list on hover. The Board's chips and ⌘J's Workers list read the same selector, so the three surfaces say the same words. Presentation only. Nothing in `packages/` changes: everything comes from the ledger the web already gets (PRs, wakes, watched runs, limit waits) and the threads store. ### One "waiting on" model `apps/web/src/orchestration/waiting-on.ts` turns a calm worker's or orchestrator's waits into: - `label`: the row's line, e.g. `Waiting · checks on #69` - `reasons`: every wait, each with a kind, a short form and a full line - `hint`: the hover, i.e. the label with `+N` when more apply, then every wait's line | Kind | Label | Comes from | | --- | --- | --- | | checks | `Waiting · checks on #69`, `Waiting · local check #69` | required checks pending; Gram's local check running | | review | `Waiting · review on #69` | the review bot reviewing the head | | merge | `Waiting · merge of #69`, `Waiting · #69 held` | an armed `merge_when_ready`; a hold on an agent's work | | dependency | `Waiting · #102 to merge`, `Waiting · acme/api#7 to merge`, `Waiting · #645 to move` | `depends_on`; a wake on another PR | | run | `Waiting · Release run` | `run_monitor` | | condition | `Waiting · main to move`, `Waiting · thread to finish`; any other kind as its chip reads it (`wakeConditionLabel`) | #108's condition wakes; the hover names the thread and says `wakeConditionWhen` | | timer | `Waiting · until 08:10`, `Waiting · until Wed 09:00` | a timed wake | | limit | `Paused · limit until 08:10`, `Paused · retry at 07:05` | a usage-limit or rate-limit wait | | held | `Held · by you` | the user holds the worker | | frozen | (reserved) | nothing produces it yet; the merge freeze will | **Which wait the label names.** What holds everything else comes first (a limit, a frozen base, your hold). Then what clears soonest by its nature (CI, the review bot, Gram's merge), then the longer waits (another PR, a run, a condition). A timed wake comes last, since it's mostly the fallback beside the others. Within a kind, the soonest wins. **Waiting is never "needs you".** A wait is only said of a calm tone (waiting, idle, or finished with a run still watched), never beside work, an ask or a failure. Nothing here touches the Needs you list or counts. ### Surfaces - **Sidebar worker rows**: the hourglass line replaces the card's now line ("checks running", "you hold it") while the worker waits. The hover card lists every wait. - **Orchestrator line**: a second line under the workspace name for its own pending wakes, or a limit it waits out. The line's hover adds the list. It's hidden while the workspace is paused or parked. - **Board**: a chip's line is the same label, and its hover the same list. - **⌘J**: a row's now line is the same label, and its hover the same list. - The waiting tone's label is now "Waiting" rather than "Waiting on checks", since it covers more than checks. ### Behaviour change A worker waiting out a usage limit (`limitWait` in the ledger) now reads as waiting on automation, with the hollow dot. Before, its activity ignored the limit, so it showed idle (or attention). ### Labels kept to four words The Board keeps a chip's line to four words. Every label fits, with the ` · ` separator counted as punctuation; `board-rail.spec.ts` now counts it that way. ### Condition kinds still to come `conditionShort` special-cases `branchMoved` and `threadFinished` only. Any other kind falls back to contracts' `wakeConditionLabel`, and its hover line to `wakeConditionWhen`. So #110 (`freezeLifted`), #111 (`ciStuck`, machine load and disk) and #112 (`releasePublished`, `versionChanged`) show up without a change here. Whichever lands last adds any icon, words or priority of its own. The `frozen` kind is reserved for #110's frozen-main state. ## Screenshots From `pr-shots` on synthetic mock data. The "kinds" scene (`simulateWaitingOn`) adds a timed wake, a watched release run, a usage limit, an armed merge, a wake on another PR and one on `main` moving, and the orchestrator resting on two wakes of its own (a thread to finish, ranked first, and a timed one). **The sidebar over `casefiles`: workers waiting on CI, the review bot and a dependency say what they wait on** (dark) _After only: the base timed out staging it; the light pair below has the before._  **The sidebar over `casefiles`: workers waiting on CI, the review bot and a dependency say what they wait on** (light) | Before | After | | --- | --- | |  |  | **Every kind of wait in the sidebar, and the orchestrator resting on a wake of its own** (dark) _After only: the scene's mock scenario is new in this PR._  **Every kind of wait in the sidebar, and the orchestrator resting on a wake of its own** (light) _After only: the scene's mock scenario is new in this PR._  **A waiting worker's hover card lists every wait, the one its row names first** (dark) _After only: the scene's mock scenario is new in this PR._  **The same waits on the Board's chips, in the sidebar's words** (dark) _After only: the scene's mock scenario is new in this PR._  **The same waits on the Board's chips, in the sidebar's words** (light) _After only: the scene's mock scenario is new in this PR._  **⌘J's Workers list says the same** (dark) _After only: the scene's mock scenario is new in this PR._  **The same `casefiles` workers on the Board: the chips agree with Quiet's dots** (dark) | Before | After | | --- | --- | |  |  | **The same `casefiles` workers on the Board: the chips agree with Quiet's dots** (light) | Before | After | | --- | --- | |  |  | ## Verification On `fc8e877`, rebased onto `main` after #108: - `bun run typecheck`, `bun run lint`, `bun run fmt:check`, `bun run check:scaffold`: pass - `vitest run src/orchestration src/views src/sidebar src/thread src/shell src/api`: 1228 passed (on `767c1af`); `waiting-on.test.ts`: 30 passed on the head - Affected e2e, `--workers=2` (`overview-sidebar board-parity board-rail board-virtual feature-sheet orchestration workers-list quiet-view overview-extras workspace-density reaching-workers worker-questions one-thread-budgets ask-stack`): 135 passed, 3 failed. Two were tests to update for this change (the orchestrator's waits read in rank order; ⌘J's four-word cap counting the ` · `), fixed in `fc8e877`. The third, `overview-sidebar.spec.ts:851` (a Needs you count of 9 for 8), is untouched by this and passed alone. All three passed alone on the head. - Specs updated, not deleted: `board-rail` and `workers-list` count words without the separator; `overview-sidebar` reads a row's name with its wait, and tells the agent's glyph from the hourglass. - Not run: the full `bun run check`, left to the merge gate. ## Known limits - `frozen` has a label and a rank but no source until the merge freeze lands. - A thread-finished condition shows "thread to finish" on the row; the thread's title is in the hover only, to keep the line short. - Synthetic mock data only; not run against a live provider or real workspace.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
…ake (#110) Builds on #108 (condition wakes, merged). **Changes merge-gate behaviour: held for an independent review.** ## What **`merge_freeze`** (orchestrator and worker tool): a thread freezes Gram's own merges onto a repo's branch (the default branch unless `branch` is given) for a `reason`, for at most 60 minutes (`minutes`, default 60). Calling it again renews the freeze. `lift: true` ends it. - **Gate:** while a freeze stands, `merge_when_ready` waits instead of merging and records why on the PR, so the ledger shows `main is frozen by "Release 1.6" for release 1.6 cut until 14:05Z`. `set_auto_merge` refuses, because GitHub's auto-merge, once armed, would merge under a freeze Gram can't hold back. - **Race:** a merge's last check and its `gh pr merge` launch run under a per-repo fence that `freeze` also takes. A freeze is either seen by that last check or lands after the launch; it can't slip between them. The check also runs at evaluate and at the start of each attempt. - **Authority:** only the thread that froze a branch can renew it. That thread, or the orchestrator of the workspace it was in, can lift it. Anyone else is refused, with the expiry time. Freezing over another thread's freeze is refused and points at `freezeLifted`. - **Always expires:** every read compares `until` with the clock, so neither a timer that never fired nor a restart past the expiry keeps a freeze alive. Timers also end it on time and are re-armed at start. Rows are persisted in a new `merge_freezes` table (migration `0002_merge_freezes`), so a freeze survives a restart until it expires. A closed thread's freezes end with it; an archived or stopped thread's simply run out. - **Lift/expiry re-checks:** any change on `MergeFreezes.changes` re-checks every armed merge in that repo, so a held merge goes as soon as the freeze ends. - **Limits:** it governs only Gram's merges. A person or GitHub can still merge, and the tool description and docs say so. **`freezeLifted`** condition kind (on #108's interface): `{ kind: "freezeLifted", repo, branch? }` fires when the freeze ends, lifted or expired. Without `branch` it takes the freeze standing on the repo; it is refused if nothing is frozen. It is driven by `MergeFreezes.changes`, with a 1-minute poll fallback and a 5-second debounce. **Release flow:** [docs/RELEASING.md § Holding main during the cut](docs/RELEASING.md#holding-main-during-the-cut). The cutting thread freezes main just before merging the release PR, renews while needed, and lifts once `release:rc` has recorded the Linux dispatch, instead of asking the orchestrator to hold merges. The release-worker brief itself lives in orchestrator memory, outside the repo, so it should point at this section. ## Review fixes (c259b65, findings 1–9 from the independent review of 204a03c) 1. **Release flow no longer blocks itself:** `allow: [PR numbers]` (at most 4) lets the named PRs merge under the freeze. I chose this over exempting the owner's bound PR because it names exact PRs. RELEASING.md now freezes with `allow: [<release PR>]`. Test: the allowed PR merges while the freeze stands. 2. **GitHub auto-merge armed before the freeze:** on freeze, Gram turns off GitHub auto-merge on open PRs into the branch that belong to Gram threads (except allowed ones) and records them on the freeze with their method. When the freeze ends, by lift, expiry or after a restart, it arms them again, or hands them to a freeze standing on the branch by then. The result lists `autoMergeOff` and any `autoMergeStillArmed`. Not covered, and said so in the tool and docs: a person merging, auto-merge on a PR no Gram thread has, and merge-queue entries. Tests: taken down and re-armed on lift; re-armed on expiry after a restart; a PR no Gram thread has is left alone. 3. **Workspace scope:** a freeze is refused unless the repo is the origin of one of the caller's workspace projects. Refusal test. 4. **`set_auto_merge`:** the check and the launch run under the freeze fence, and when a freeze stands on the repo it reads the PR's base afresh (`gh pr view --json baseRefName`). Test: a stale badge base plus a fresh `main` is refused; an allowed PR merges. 5. **Branch validation:** branches are checked in the tool schema and on the resolved default branch, and the row is written before the in-memory map. Test: `release+1` and a default branch named `trunk+1` are refused, and nothing is left standing. 6–7. **Expiry:** one watcher per freeze re-reads the freeze and the clock at least every 60 s and ends it once expired. `lift` and the thread-closed path no longer touch timers, and ending can't be interrupted partway. Every read still compares against the clock. Test: lift, then a new freeze by the owner still ends on time and its end lets the held merge go. A macOS sleep that suspends timers can't be faithfully simulated under TestClock, so 7 rests on the 60-second re-read by design. 8. **Total cap:** a freeze stands at most 4 hours from when it was first set (`since`). Past that, renewal is refused and a fresh freeze is needed after it ends. Test included. 9. **Reason text:** collapsed to a single line, with control characters removed. Test included. - **Note:** the docs say a freeze can wait behind an in-flight merge's fresh `gh` read, since the fence covers it. ## Re-review fixes (8225a52, findings A and B from the re-review of c259b65; rebased on main after #111) - **A. Re-arm no longer overrides a deliberate choice:** a PR leaves the freeze's re-arm list the moment someone turns its auto-merge off (`set_auto_merge enabled:false`), holds its merge (`merge_when_ready hold`), unbinds it (`pr_unbind`), or Gram sees it closed or merged. The change is saved at once, so a restart can't bring it back. At the end, re-arm also checks each PR fresh: it skips one that no Gram thread binds any more, whose thread is archived, babysat or in a paused workspace, or whose merge is held. Tests cover turn-off and hold (both survive a restart), unbind and close, and archived and paused. With the fix disabled, both turn-off tests fail. - **B. A quit partway through no longer loses the record:** each PR goes on the freeze's record before its auto-merge is turned off, under the branch's lock, and comes off again if GitHub refuses. A quit partway can at worst leave a PR recorded that wasn't turned off, and arming that again is harmless. Test: GitHub hangs on the second PR and the app quits there; after the restart and expiry, both are armed again. - **Rebase:** the branch is rebased onto #111 (stuck-CI and machine kinds) and squashed. `freezeLifted` now sits beside those kinds in the union, the sources and `schedule_wake`'s description, and `wakeConditionsLayer` takes all the GitHub and machine readers as one layer. - **Prompt budget:** with #111's kinds and `merge_freeze` both in, I tightened the tool's text and then **raised orchestrator from 62,400 to 63,000 and worker from 35,700 to 36,600** (measured 62,424 and 36,318). Plain is unchanged at 24,642 of 24,900. - **Verification:** 22 related test files pass (348 tests), including #111's `wake-conditions`, `service-condition-wakes-ci-machine` and `ci-stuck`. Typecheck, lint, docs and scaffold checks are clean. ## Files - `packages/contracts/src/wake-conditions.ts`: `FreezeLiftedCondition`, `MergeFreeze`, `mergeFreezeText`, `MERGE_FREEZE_MAX_MINUTES` = 60, `MERGE_FREEZE_REASON_MAX` = 200 - `packages/server/src/git/merge-freezes.ts`: the `MergeFreezes` service (`freeze`, `lift`, `active`, `list`, `fence`, `changes`), plus `MergeFreezesNone` for test layers that don't exercise freezes - `packages/server/src/git/merge-when-ready.ts`: the freeze check at evaluate, at attempt start and inside the launch fence; the re-check on freeze changes - `packages/server/src/app-tools/pr-tools.ts`: the `merge_freeze` tool; `set_auto_merge` refusal - `packages/server/src/wakes/condition-sources.ts`: the `freezeLifted` source; `wakeConditionsLayer` - `packages/server/src/persistence/migrations.ts`: `0002_merge_freezes` - docs: `AGENTS_AND_FEATURES.md` (merge freezes, `freezeLifted` row), `RELEASING.md` ## Verification - `service-merge-freeze.test.ts` (real harness, full-autonomy workspace, PR armed with `merge_when_ready`): - **Gate and authority:** freezing the default branch with a reason; `minutes: 61` refused; the ready PR waits with the exact line `main is frozen by "Release 1.6" for the 1.6 release cut until HH:MMZ` and no `gh pr merge`; `set_auto_merge` refused naming the freeze. A non-owning worker can't freeze over it or lift it ("Only "Release 1.6" or its orchestrator…"). The owner renews. The orchestrator lifts, and the merge then launches pinned with `--match-head-commit`. - **Restart and expiry:** the freeze survives a restart and the ready PR still waits. The owning agent is interrupted and never returns, yet the freeze expires at its time (`active` is undefined) and the merge goes. A plain thread's `freezeLifted` wake fires once after its debounce; `freezeLifted` on an unfrozen branch is refused. - #108's `service-condition-wakes.test.ts` still passes (6 tests). - `merge-after-local-check`, `merge-base-lock`, `merge-moved-base`, `merge-ci-launch`, `pr-tools`, `pr-tools-merge`, `pr-tools-merge-launch`, `migrations`, `held-briefs`, `service-wakes`, `tools`, `tool-ergonomics`, `http/server`, `registry`: pass (16 files, 258 tests, with the 3 `migrations` cases updated for `0002`; re-run green). - `prompt-budget.test.ts`: **the worker limit is raised on purpose from 34,700 to 35,600**. `merge_freeze` plus #108's condition text bring the worker to 35,137 characters. The orchestrator is at 61,243 of 62,400; plain threads don't get the tool. - `bun run typecheck`, `bun run lint`, `bun run fmt`, `bun run check:docs`, `bun run check:scaffold`: clean. No full local `bun run check`; CI and the managed check gate it. ## Known limits - A PR whose badge doesn't name its base is held by a freeze anywhere on its repo (the conservative choice). - The fence is per repo, so two Gram merges onto different bases of one repo serialize their final launch. This only matters if they launch at the same moment. - The user has no "lift" button; the freeze is at most 60 minutes and the orchestrator can lift it. Adding a UI control would mean a new surface, which the brief rules out. - Migration `0002` is the first 1.x migration. A parallel PR adding one must renumber.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
…ion change (#112) ## What Two new condition kinds for `schedule_wake`, built on the engine from #108. Also a fix to that engine, which the new kinds exposed. - **`releasePublished`**: a package on npm, or a GitHub repo's releases, publishes a version newer than `after` that is at least `minAgeDays` old. Prereleases (and GitHub drafts) don't count unless `includePrerelease` is set. This is for dependency bumps that wait out a release-age window. - **`versionChanged`**: the installed version of Gram, `claude`, `codex` or `gh` changes from `from`. ## The spec types (`packages/contracts/src/wake-conditions.ts`) ```ts export const ReleaseVersion = Schema.String.check(Schema.isPattern(/^v?\d+\.\d+\.\d+(-[\w.-]{1,40})?$/)); export const ReleaseName = Schema.String.check(Schema.isPattern(/^@?[\w.~][\w.~-]*(\/[\w.~-]+)?$/)); // never a leading dash export const RELEASE_MIN_AGE_DAYS = 7; // Gram's own Dependabot cooldown export const ReleasePublishedCondition = Schema.Struct({ kind: Schema.Literal("releasePublished"), source: Schema.Literals(["npm", "github"]), name: ReleaseName, // @scope/pkg, or owner/repo after: Schema.optionalKey(ReleaseVersion), // filled with the latest release when set minAgeDays: Schema.optionalKey(Schema.Finite), // filled with 7; 0 = none; the source refuses one outside whole 0..90 includePrerelease: Schema.optionalKey(Schema.Boolean), }); export const VersionTarget = Schema.Literals(["gram", "claude", "codex", "gh"]); export const VersionChangedCondition = Schema.Struct({ kind: Schema.Literal("versionChanged"), target: VersionTarget, from: Schema.optionalKey(ReleaseVersion), // filled with the installed version when set }); ``` New kinds and optional fields, so `WAKE_CONDITION_VERSION` stays at 1. **Words:** - Chip label: `codex newer than 0.166.0, 7 days old`, `cli release` (no baseline yet), `Gram update`, `Codex update`. - `when`: `@openai/codex on npm has a release newer than 0.166.0, at least 7 days old`, `Gram is updated from 1.5.0`, `the installed Codex changes from 0.166.0`. **Facts on fire:** - `releasePublished`: `source`, `name`, `version`, `published` (ISO), `url`, `after`. - `versionChanged`: `target`, `version`, `previous`. **Refusals when set:** - No such package or repo. - A GitHub name that isn't `owner/repo` (including `.`/`..`). - A newer release that is old enough already. - A `from` that isn't what's installed. - A CLI the launch checks didn't find. - A Gram build that doesn't report its version (dev). ## Network calls Gram's existing policy (`agent-runtime/releases.ts`) is to reach packages only through the user's own npm, never a registry npm wasn't configured with. This follows it: - **npm:** `npm view --json <name> time versions`, from the home folder through `Proc`. npm applies its own registry, proxy and cache. Gram never reads `.npmrc`. npm's stderr is read only for `E404`, never kept or logged. Timeout 2 min, output cap 4 MiB. - **GitHub:** `gh api repos/<owner>/<repo>/releases?per_page=30 --hostname github.com`, with the user's gh, after `GhBudget.admit("core", "background")` and under `asBackground`, as `BranchHeads` does. - Nothing else is sent: only the name asked about, through the user's own tools. `versionChanged` makes **no** network calls and starts **no** process. ## Poll intervals | Kind | Looks | Network / probes | | --- | --- | --- | | `releasePublished` | every 1 h (cheap: over the kept answer); debounce 0 | one read per package, shared by every wake on it, kept 6 h; a read when a wake is set unless one is under 60 s old; a failed read is never met and is retried after 12 h, then every 24 h, and is reset by a success | | `versionChanged` | every 10 min, 10 s after start, and on each launch-check run; debounce 0 | none. Gram's version is `ServerConfig.appVersion`. A CLI's comes from the last `LaunchChecksCache` run (at start, on Re-check, after a Settings update). It is read through the new `latest` and `changes`, which never start a run. Setting a wake may wait for the cache's one first run, which `app.launchChecks` starts anyway | ## Rebased onto #111 #111 landed the same per-kind `metSince` fix in `wakes/conditions.ts`, so this PR now keeps main's version of that file. The regression test here (`another kind's sweep leaves a debounce under way alone`) stays as extra cover. The union, sources, layers and test fakes now include both PRs' kinds. ## Rebased onto #113 #113's waiting-on board keeps labels to four words. This PR gives the new kinds short board words: `Waiting · codex release` and `Waiting · Gram update`. The longer chip label stays (`codex newer than 0.166.0, 7 days old`). #113's test for "a kind with no words" used `releasePublished` as its stand-in future kind, with a made-up shape. Now that the kind is real, that broke `check tests 1/2` and `macos tests 2/3`. The test now uses a kind that doesn't exist, and a new test covers the real words. Web `orchestration` tests: 47 files, 463 passed. ## Rebased onto #110 #110 (merge freeze, `freezeLifted`) merged first. The union, the sources, `wakeConditionsLayer`'s inputs, the server's layers and the test harness now carry all seven kinds. ## Prompt budget The `schedule_wake` description lists all seven kinds. My sentence was cut to `releasePublished` (a release of `name` newer than `after`, `minAgeDays` old) and `versionChanged` (installed `target` leaves `from`). The defaults are in the schema, the docs and the tool's answer. The wording was also tightened elsewhere: no "release-age window" example, no "the message names the event", and a shorter fallback clause. The new kinds' input fields still push two roles over, so worker and plain are **raised by 500 on purpose**, as #111 and #110 did for theirs. The reason is in the test header. | Role | Measured | Limit | | --- | --- | --- | | orchestrator | 62,853 | 63,000 (unchanged) | | worker | 36,747 | 37,100 (was 36,600) | | plain | 25,071 | 25,400 (was 24,900) | `minAgeDays` is `Schema.Finite`, as #111's numbers are, to keep the schema short. The source refuses a fraction or a value outside 0..90 (tested). ## Verification - After the rebase onto #110, `bun run test` over 62 files (server wakes, condition, ci-machine, merge-freeze, release, restarts, tools, budget, launch-checks, http server, contracts, web `orchestration` and `thread-view`): 659 passed. Typecheck, lint, docs and scaffold: clean. - After the earlier rebase onto #111, `bun run test` over 23 files (the list below plus #111's `service-condition-wakes-ci-machine`, and the web `thread-view`): 698 passed, 1 skipped. - Before the rebase: `bun run test` over `service-release-wakes` (new, 4), `service-condition-wakes` (7), `prompt-budget`, `service-wakes`, `service-digests`, `service-restarts`, `rules`, `tools`, `tool-ergonomics`, `wakes/*` (including the new `upstream-releases` parser test), `launch-checks/*` (including the new `observeRuns` test), `agent-support`, `issues/tools`, `http/server`, `thread-store`, contracts `wake-conditions` (new) and `orchestration`: 20 files, 666 passed, 1 skipped. - Web `thread-view`: 16 passed. - What the new tests cover: - Each kind fires once. - Age gating: nothing at 6 days; fires at 7 from the hourly look without a new read. - Prereleases are skipped, and counted when asked for. - An already-due release is refused. - An unreadable registry is set without a baseline, never fires, and is read at 0, 12 h and 36 h (backoff); the first good read becomes the baseline, then reads every 6 h. - A CLI's change fires on a launch-check run. - A Gram update fires after a restart through the existing `ThreadWakes` host. - Refusals, and names shaped like flags or paths. - `bun run typecheck`, `bun run lint`, `bun run fmt`, `bun run check:docs`, `bun run check:scaffold`: clean. - Gram's managed full check of the pre-rebase head (06407b6) failed 24 tests. 22 were timeouts, and the other 2 were element lookups in web panel tests, all in files this PR doesn't touch, with machine load around 20–50. On the rebased head, `dreams-service-archive`, `dreams-service-adoption`, `codex-dream-protocol`, web `pr-panel` and `message-block` were rerun: 94 passed. The release-stable test was not rerun. ## Known limits - All npm, GitHub and launch-check behaviour is behind `UpstreamReleases` and `InstalledVersions`, which the tests fake. No live-provider qualification was run. - A CLI update made outside Gram (for example `npm i -g` in a terminal) is seen only at the next launch-check run: a restart, Re-check, or an update from Settings. This is deliberate: it reuses the existing probes and never adds one. - For `claude` and `codex`, `versionChanged` watches the PATH CLI the launch checks probe, not a managed runtime. - A GitHub release's version is the first `x.y.z` in its tag (`v1.2.3`, `rust-v0.166.0`). Tags naming no version are skipped. One repo publishing several packages under different tag prefixes isn't told apart. - Only github.com is read for `source: "github"`. - The kept reads and backoff are in memory, so a restart reads again once.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
Desktop release metadata proposal only. Merging this PR does not publish or tag Gram; follow docs/RELEASING.md to release. --- ## [1.6.0](v1.5.0...v1.6.0) (2026-11-03) ### Features * condition wakes for schedule_wake (branch moved, thread finished) ([#108](#108)) ([550545a](550545a)) * merge freeze that holds Gram's own merges, and a freezeLifted wake ([#110](#110)) ([5096a48](5096a48)) * **orchestration:** wake on an upstream release or an installed version change ([#112](#112)) ([efdd8fa](efdd8fa)) * **orchestration:** wake on stuck CI and on machine load or low disk ([#111](#111)) ([be9f98f](be9f98f)) * **web:** show what a worker or orchestrator is waiting on ([#113](#113)) ([4a2ae69](4a2ae69)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: gram-release-please[bot] <351321620+gram-release-please[bot]@users.noreply.github.com> Co-authored-by: Johan <32440065+ANT1OFF@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two new condition kinds for
schedule_wake, built on the mechanism from #108, plus a fix to that mechanism's engine.ciStuck{ repo, pr? | runId?, queuedMinutes? }: fires when a hosted Actions job of the PR or run has sat queued with no runner forqueuedMinutes(default 15). It also fires when a run awaits a maintainer's approval, as a fork PR's does (test(terminal): quiet expected non-Windows profile skip warnings (#73) #76 ran no CI and nobody noticed).prdefaults to the thread's bound PR in that repo. A run that completes ends the wait, and the notice says so. Facts:job,queued("27 min, no runner"),url, andalsoQueuedwhen more jobs are queued. For an approval:state: awaits approvaland the run's URL.machine, two shapes:{ metric: "load", below, forMinutes? }: the 1-minute load average stays belowbelowforforMinutes(default 5). The number is absolute, asuptimeshows it, not per CPU. It is what people read offuptime/top, and today's numbers ("timed out at load ~28") were absolute. Use it so heavy checks wait for a quiet Mac.{ metric: "diskFree", belowGB, path? }: free space on the volume holdingpath, by default the thread's repo, drops belowbelowGB. GB here means 10^9 bytes, as Finder anddf -Hcount. Use it so cleanup starts before the disk fills.Chip labels:
CI job queued 15+ min,quieter (load < 8)anddisk < 20 GB, next to the chip's alarm icon. Thewhenlines:a CI job of acme/app#7 sits queued 15+ min, or a run awaits approval,this machine's load stays below 8 for 5 minandfree disk space at /path drops below 20 GB.Engine fix (
wakes/conditions.ts)Each kind's sweep cleared the debounce state (
metSinceand the pending debounce look) of every wake it hadn't just seen. But a sweep only sees its own kind. So any other kind's sweep inside a debounce wiped it, and a debounced wake fired only if no other kind happened to sweep during its debounce. The new kinds' looks run every 30 s, which made this constant. Now a sweep clears only its own kind's entries.Reproduction:
a run awaiting approval fires…inservice-condition-wakes-ci-machine.test.tssweepsmachineandbranchMovedinside aciStuckdebounce. It fails with #108'sconditions.tsand passes with this one. I ran both.Polling and API cost
ciStuckon a PRPr.checks, the read the PR monitors share; reused if anyone read it in the last minute, and its 10 s cache answers every wake on the PR). Only when the PR has no Actions check at all: +1 REST (actions/runs?status=action_required), and +1 REST (pulls/Nfor its head) only if that list isn't empty.ciStuckon a runruns/{id}/jobs), reused for 20 s across wakes on the run. +1 REST (runs/{id}) only once nothing of the run is in flight, to tell an approval from a finish.machineloadforMinuteshold (in memory)os.loadavg()in the server processmachinediskFreefs.statfs()in the server processEvery GitHub call is admitted against the background share of the allowance (
GhBudget.admit(…, "background")) and runs underasBackground. Ifghcan't be reached, is logged out, or is out of allowance, the look returns "couldn't tell": it neither fires nor throws. As before, a kind with nothing pending doesn't poll. The run monitor readsruns/{id}and never the jobs list of a running run, so it doesn't duplicate the run case. For PRs,Pr.seenChecksandPr.checksare the shared path.When GitHub gives a queued job no time, its queued time counts from the first look that saw it. That time is kept in the wake's
seen, so a restart doesn't reset it (tested).Arm-time refusals
ciStuck: refused when given bothprandrunId; with neither and no PR bound in that repo; for a run GitHub doesn't have; when a job is already queued past the limit or a run already awaits approval; and for non-integer or out-of-range numbers (queuedMinutesmust be 1–1440).machineload: refused when the 1-, 5- and 15-minute averages, as far as they coverforMinutes, already show the load below. Beyond 15 minutes the averages can't tell, so it doesn't refuse.machinediskFree: refused when free space is already below, orpathisn't absolute or can't be read.Worktree sweep (#103) on low disk: not done, follow-up
Running the sweep early would only do what is already due, and the sweep runs every 15 min anyway, so it gains at most 15 minutes. Freeing space that matters under disk pressure would mean shortening
freeAfterDays/removeAfterDays(removingnode_modulesor worktrees sooner). That changes a user setting's meaning and needs consent, so it isn't a small safe change. Suggested follow-up: a disk-pressure mode, with the user's opt-in, that frees finished threads'node_modulesearly. Meanwhile an agent can set{kind:"machine", metric:"diskFree", belowGB}and act on the wake.Prompt budget (#89)
The two kinds add about 980 characters per role: about 600 of schema and 380 of description, after trimming. To save room:
Schema.Finite, whose JSON schema is{"type":"number"}.Int's bounds andNumber's Infinity/NaNanyOfare each ~65 characters. The source refuses a fraction.schedule_wake'sinMinutesnow usesSchema.Finite, which rejects Infinity and NaN at decode.Even so, worker (35,201) and plain (24,370) went over, so I raised their limits by 1,000, to 35,700 and 24,900, as the test's header asks ("raise one only on purpose, in the change that needs the room, and say so there"). The orchestrator stays under 62,400 at 61,307. The merge-freeze and release-version workers add kinds in parallel and will need room of their own.
Code layout
New files, to stay out of the parallel kinds' way:
wakes/ci-stuck.ts(theCiJobsservice,CiJobsLive,judgeCiStuck,ciStuckSource) andwakes/machine.ts(theMachineStateservice,MachineStateLive,machineSource).folderOfmoved towakes/condition-folders.tsso the sources share it. The test harness fakesCiJobs(world.ciStates) andMachineState(world.machineLoad,world.diskSpace).Verification
bun run testonservice-condition-wakes-ci-machine(6),wakes/ci-stuck(7), contractswake-conditions(3), andservice-condition-wakes,prompt-budget,service-wakes,service-digests,service-restarts,tool-ergonomics,rules,tools,http/server, contractsorchestration: 13 files, 603 tests pass. The webthread-viewtest passes (16).forMinutes: a load spike restarts the hold.CiJobsLiveover aghthat won't start, andPr.checksfailing, returnsundefinedand makes no finding. In the engine, an unreadable PR neither fires nor fails.ghcalls per read is asserted, including zero when the shared checks are under a minute old.whenlines are tested in contracts.bun run typecheck,bun run lint,bun run fmt,bun run check:docs,bun run check:scaffold: clean.bun run checklocally; CI gates it.Known limits
CiJobs, which the tests fake (CiJobsLiveis unit-tested over a fakeghandPr). There was no live-provider qualification. Two GitHub details are assumed: that GraphQLCheckRun.startedAtis the queue time for a queued Actions job, and that a fork's approval-pending run hasstatus/conclusionaction_requiredwith no check runs on the PR. IfstartedAtis absent, the first-seen time is used, which only delays the wake.WAITING(environment approval) andPENDING(concurrency). Those count as stuck too, after the same minutes. Non-Actions statuses are ignored.ciStuckwake on a PR that merges or closes waits until its fallback time, polling one GraphQL point every 2 min. A run's completion does end its wait.