Repository navigation
test(orchestration): a size budget for each role's prompt and tools - #89
Merged
Merged
Conversation
Adds up each role's system prompt and the Gram tool definitions its sessions are bound with, in characters, and fails above a limit about 5% over today's size: orchestrator 59,373, worker 33,022, plain 22,669.
ANT1OFF
added a commit
that referenced
this pull request
Nov 2, 2026
… repeating (#90) Stacked on #89, the prompt size budget; these additions stay within it. Merge after #89. ## What changes Small changes to role text that every orchestrator and worker reads. Each one is quoted below exactly as it now reads. Nothing else changes. ### (a) A worker's `report` says what it ruled out (`prompt.ts`, worker role) In the `report` line, "Keep the summary short and factual." now reads: > Keep the summary short and factual, and say what you tried and ruled out, not only what worked. ### (b) Check before repeating an outside action after a restart or cut-off New line in the worker role (`prompt.ts`), after "Report, then end your turn…": > - After a restart or a turn that was cut off, check your worktree, git history and GitHub before you repeat a push, a pull request or a comment: it may have gone through already. The cut-off notice a worker is sent when a restart continues it is `restartContinuation` in `rules.ts`, sent by the boot path (`service/boot.ts` → `noteRestart`). Its old phrase "Continue where you left off, after checking what state your work was left in." now reads: > Continue where you left off, after checking your worktree, git history and GitHub: don't repeat a push, a pull request or a comment that already went through. The orchestrator's away-digest note on cut-off workers (`restartNote` in `rules.ts`) gains one sentence after the list of workers. Whether to set a worker going again stays the orchestrator's decision: > Before you set one going again, have it check its worktree, git history and GitHub so no push, pull request or comment goes out twice. ### (c) Briefs attach the user's words and invent nothing (`prompt.ts`, orchestrator role) The `userMessages` line used to read: "When the user pasted something the worker needs as it is (a log, a spec, an error, a long list), don't retype it into the brief: pass `userMessages` … The brief still says what to do with them." It now reads: > - When a brief comes from what the user asked, attach their words rather than retyping them, above all a paste the worker needs as it is (a log, a spec, an error, a long list): pass `userMessages` with how many of the user's latest messages in this conversation to attach, and Gram copies them word for word after the brief. Do the same on `send_message` to a worker. The brief still says what to do with them, and adds no requirements, acceptance criteria or checklists the user didn't ask for. The brief-writing guidance before it ("A worker sees only its brief… Write the brief so it stands alone: …") is unchanged. The orchestrator still writes briefs, and a test now asserts that line remains. ## Size (characters, measured with the budget test's inputs) | Role | System prompt before | after | Total after (prompt + tools) | Limit from #89 | | --- | ---: | ---: | ---: | ---: | | orchestrator | 20,112 | 20,243 (+131) | 59,504 | 62,400 | | worker | 5,998 | 6,237 (+239) | 33,261 | 34,700 | | plain | 302 | 302 | 22,669 | 23,900 | Everything stays within budget, so nothing was trimmed. Tool definitions are unchanged. ## Tests - `rules.test.ts`: - the worker prompt carries (a) and (b); - the orchestrator prompt carries (c) and still says "Write the brief so it stands alone"; - `restartContinuation` carries the check; - `restartNote`'s exact string is updated. - `bun run test packages/server/src/orchestration/rules.test.ts packages/server/src/orchestration/service-restarts.test.ts packages/server/src/orchestration/prompt-budget.test.ts packages/server/src/orchestration/workspace-pause.test.ts`: all passed (437 before the last wording change to `restartNote`; 431 on the final run, which left out the budget and pause files). - `GOMEMLIMIT=2GiB bun run typecheck --singleThreaded`, `bun run lint`, `bun run fmt:check`, `bun run check:scaffold`: pass ## Known limits - Synthetic only: the text is asserted, not how live agents behave with it. - This is general harness behaviour, not Gram's own repo process: nothing here names a Gram-specific workflow.
ANT1OFF
added a commit
that referenced
this pull request
Nov 3, 2026
…#111) ## What Two new condition kinds for `schedule_wake`, built on the mechanism from #108, plus a fix to that mechanism's engine. - **`ciStuck`** `{ repo, pr? | runId?, queuedMinutes? }`: fires when a hosted Actions job of the PR or run has sat queued with no runner for `queuedMinutes` (default 15). It also fires when a run awaits a maintainer's approval, as a fork PR's does (#76 ran no CI and nobody noticed). `pr` defaults to the thread's bound PR in that repo. A run that completes ends the wait, and the notice says so. Facts: `job`, `queued` ("27 min, no runner"), `url`, and `alsoQueued` when more jobs are queued. For an approval: `state: awaits approval` and the run's URL. - **`machine`**, two shapes: - `{ metric: "load", below, forMinutes? }`: the 1-minute load average stays below `below` for `forMinutes` (default 5). The number is **absolute**, as `uptime` shows it, not per CPU. It is what people read off `uptime`/`top`, and today's numbers ("timed out at load ~28") were absolute. Use it so heavy checks wait for a quiet Mac. - `{ metric: "diskFree", belowGB, path? }`: free space on the volume holding `path`, by default the thread's repo, drops below `belowGB`. GB here means 10^9 bytes, as Finder and `df -H` count. Use it so cleanup starts before the disk fills. Chip labels: `CI job queued 15+ min`, `quieter (load < 8)` and `disk < 20 GB`, next to the chip's alarm icon. The `when` lines: `a CI job of acme/app#7 sits queued 15+ min, or a run awaits approval`, `this machine's load stays below 8 for 5 min` and `free disk space at /path drops below 20 GB`. ### Engine fix (`wakes/conditions.ts`) Each kind's sweep cleared the debounce state (`metSince` and the pending debounce look) of every wake it hadn't just seen. But a sweep only sees its own kind. So any other kind's sweep inside a debounce wiped it, and a debounced wake fired only if no other kind happened to sweep during its debounce. The new kinds' looks run every 30 s, which made this constant. Now a sweep clears only its own kind's entries. Reproduction: `a run awaiting approval fires…` in `service-condition-wakes-ci-machine.test.ts` sweeps `machine` and `branchMoved` inside a `ciStuck` debounce. It fails with #108's `conditions.ts` and passes with this one. I ran both. ## Polling and API cost | Kind | Poll | Debounce / hold | Calls per poll | | --- | --- | --- | --- | | `ciStuck` on a PR | 2 min | 1 min debounce | **0 or 1 GraphQL** (`Pr.checks`, the read the PR monitors share; reused if anyone read it in the last minute, and its 10 s cache answers every wake on the PR). Only when the PR has **no Actions check at all**: +1 REST (`actions/runs?status=action_required`), and +1 REST (`pulls/N` for its head) only if that list isn't empty. | | `ciStuck` on a run | 2 min | 1 min debounce | **1 REST** (`runs/{id}/jobs`), reused for 20 s across wakes on the run. +1 REST (`runs/{id}`) only once nothing of the run is in flight, to tell an approval from a finish. | | `machine` load | 30 s | `forMinutes` hold (in memory) | none: `os.loadavg()` in the server process | | `machine` diskFree | 30 s, but each volume is read at most every 5 min | none | none: `fs.statfs()` in the server process | Every GitHub call is admitted against the background share of the allowance (`GhBudget.admit(…, "background")`) and runs under `asBackground`. If `gh` can't be reached, is logged out, or is out of allowance, the look returns "couldn't tell": it neither fires nor throws. As before, a kind with nothing pending doesn't poll. The run monitor reads `runs/{id}` and never the jobs list of a running run, so it doesn't duplicate the run case. For PRs, `Pr.seenChecks` and `Pr.checks` are the shared path. When GitHub gives a queued job no time, its queued time counts from the first look that saw it. That time is kept in the wake's `seen`, so a restart doesn't reset it (tested). ## Arm-time refusals - `ciStuck`: refused when given both `pr` and `runId`; with neither and no PR bound in that repo; for a run GitHub doesn't have; when a job is already queued past the limit or a run already awaits approval; and for non-integer or out-of-range numbers (`queuedMinutes` must be 1–1440). - `machine` load: refused when the 1-, 5- and 15-minute averages, as far as they cover `forMinutes`, already show the load below. Beyond 15 minutes the averages can't tell, so it doesn't refuse. - `machine` diskFree: refused when free space is already below, or `path` isn't absolute or can't be read. ## Worktree sweep (#103) on low disk: not done, follow-up Running the sweep early would only do what is already due, and the sweep runs every 15 min anyway, so it gains at most 15 minutes. Freeing space that matters under disk pressure would mean shortening `freeAfterDays`/`removeAfterDays` (removing `node_modules` or worktrees sooner). That changes a user setting's meaning and needs consent, so it isn't a small safe change. Suggested follow-up: a disk-pressure mode, with the user's opt-in, that frees finished threads' `node_modules` early. Meanwhile an agent can set `{kind:"machine", metric:"diskFree", belowGB}` and act on the wake. ## Prompt budget (#89) The two kinds add about 980 characters per role: about 600 of schema and 380 of description, after trimming. To save room: - numeric fields use `Schema.Finite`, whose JSON schema is `{"type":"number"}`. `Int`'s bounds and `Number`'s Infinity/NaN `anyOf` are each ~65 characters. The source refuses a fraction. - `schedule_wake`'s `inMinutes` now uses `Schema.Finite`, which rejects Infinity and NaN at decode. - the condition sentence names fields instead of repeating object literals. Even so, worker (35,201) and plain (24,370) went over, so I **raised their limits by 1,000**, to 35,700 and 24,900, as the test's header asks ("raise one only on purpose, in the change that needs the room, and say so there"). The orchestrator stays under 62,400 at 61,307. The merge-freeze and release-version workers add kinds in parallel and will need room of their own. ## Code layout New files, to stay out of the parallel kinds' way: `wakes/ci-stuck.ts` (the `CiJobs` service, `CiJobsLive`, `judgeCiStuck`, `ciStuckSource`) and `wakes/machine.ts` (the `MachineState` service, `MachineStateLive`, `machineSource`). `folderOf` moved to `wakes/condition-folders.ts` so the sources share it. The test harness fakes `CiJobs` (`world.ciStates`) and `MachineState` (`world.machineLoad`, `world.diskSpace`). ## Verification - `bun run test` on `service-condition-wakes-ci-machine` (6), `wakes/ci-stuck` (7), contracts `wake-conditions` (3), and `service-condition-wakes`, `prompt-budget`, `service-wakes`, `service-digests`, `service-restarts`, `tool-ergonomics`, `rules`, `tools`, `http/server`, contracts `orchestration`: 13 files, 603 tests pass. The web `thread-view` test passes (16). - Each kind fires once: a stuck PR job, a fork run awaiting approval, a completed run, load, disk. - Debounce: a job picked up inside the debounce doesn't fire. `forMinutes`: a load spike restarts the hold. - Unreachable gh: `CiJobsLive` over a `gh` that won't start, and `Pr.checks` failing, returns `undefined` and makes no finding. In the engine, an unreadable PR neither fires nor fails. - Restart: a job's first-seen queued time survives a restart and fires after it. - API calls: the number of `gh` calls per read is asserted, including zero when the shared checks are under a minute old. - Labels and `when` lines are tested in contracts. - `bun run typecheck`, `bun run lint`, `bun run fmt`, `bun run check:docs`, `bun run check:scaffold`: clean. - I didn't run a full `bun run check` locally; CI gates it. ## Known limits - All GitHub behaviour is behind `CiJobs`, which the tests fake (`CiJobsLive` is unit-tested over a fake `gh` and `Pr`). There was no live-provider qualification. Two GitHub details are assumed: that GraphQL `CheckRun.startedAt` is the queue time for a queued Actions job, and that a fork's approval-pending run has `status`/`conclusion` `action_required` with no check runs on the PR. If `startedAt` is absent, the first-seen time is used, which only delays the wake. - For a PR, "queued" is the check's phase, which also covers `WAITING` (environment approval) and `PENDING` (concurrency). Those count as stuck too, after the same minutes. Non-Actions statuses are ignored. - A `ciStuck` wake on a PR that merges or closes waits until its fallback time, polling one GraphQL point every 2 min. A run's completion does end its wait. - The load hold lives in memory, so a restart starts it again. The app can't know the load while it was down. - Windows has no load average, so a load wake is refused there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
This adds a test,
packages/server/src/orchestration/prompt-budget.test.ts, that adds up each role's system prompt and the definitions of the Gram tools its sessions are bound with. It fails when a role goes over a fixed limit set about 5% above today's size. The test only reads: no prompt, tool or runtime behaviour changes.What is counted
supervised,publish,full) and harness (claude,codex), with synthetic names:rolePrompt, with two repos and gated;workerPrompt, with one extra repo and gated;plainPrompt.AppToolRegistry.forThread(…, role)on the orchestration test server, which is the list a session of that role is bound with. For each tool: name, description andJSON.stringifyof its input JSON schema.prompt.tsandtools.ts. The dream prompt (dream-prompt.ts) isn't counted.Measured on 2026-11-02 (origin/main 9f6e3b7), in characters
Tools counted: orchestrator 50, worker 41, plain 37.
Going over a limit fails with the role's prompt and tool sizes in the message. The file's header says to lower a limit when text shrinks, and to raise one only on purpose, in the change that needs the room.
Commands
bun run test packages/server/src/orchestration/prompt-budget.test.ts: 1 passed. With every limit set to 0 it failed with the sizes above, so the check does fail when a role is over.GOMEMLIMIT=2GiB bun run typecheck --singleThreaded: passbun run lint,bun run fmt:check,bun run check:scaffold: passKnown limits