review: orchestrator on Sonnet 5 at low effort - #290
Conversation
… 5 at low effort With task mode removed the orchestrator transcribes the plan and makes a handful of small judgments; the gate reds any transcription drift. The trial measured the orchestrator line at $2.0-3.1/run on Opus, the largest non-reviewer cost. engine.model moves to claude-sonnet-5 with CLAUDE_CODE_EFFORT_LEVEL=low in the engine env; the dispatcher's runner strips that variable from the sub-agent environment so reviewer roles never inherit the orchestrator dial. The pinned firewall (v0.27.27) already prices claude-sonnet-5; a models: entry keeps the cost display correct. Live-trial-gated as the treatment arm of the next seeded lifecycle.
🦋 Changeset detectedLatest commit: e0f9a3b The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Review Guidancegithub-actions (2 files)
Excluded from review (2 files)Not individually reviewed — generated, formatting-only, or fully explained by a common pattern above:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| REVIEW_REPO_ROOT: "/work", | ||
| EMPTY: undefined, | ||
| }), | ||
| ).toEqual({REVIEW_REPO_ROOT: "/work"}); |
There was a problem hiding this comment.
note (non-blocking): toEqual ignores properties whose value is undefined, so the undefined-stripping half of this assertion is vacuous — removing the entry[1] !== undefined clause in subAgentEnv would leave this test green. Use toStrictEqual to actually guard it.
| ).toEqual({REVIEW_REPO_ROOT: "/work"}); | |
| ).toStrictEqual({REVIEW_REPO_ROOT: "/work"}); |
Low-confidence (1)
.changeset/review-sonnet-low-orchestrator.md:5— Prose says Sonnet is ~40% of Opus pricing, but the added cost-table numbers ($3/$15 per M) are ~20% of Opus list ($15/$75); the table values are correct, the prose figure is imprecise.
| * default (their definitions' effort annotations remain the human-facing | ||
| * table in the README until gh-aw grows a per-agent effort field). | ||
| */ | ||
| export const subAgentEnv = ( |
There was a problem hiding this comment.
suggestion (non-blocking): This strips exactly one key, so any future orchestrator-only engine-env dial (or a model bump with a different default effort) would pass through to sub-agents by default. Since scripted dispatch owns sub-agent spawns, consider inverting to a pass-through allow-list plus an explicit per-role effort, so future engine-env additions are opt-in rather than leak-by-default.
| model: claude-opus-4-8 | ||
| model: claude-sonnet-5 | ||
| env: | ||
| CLAUDE_CODE_EFFORT_LEVEL: "low" |
There was a problem hiding this comment.
thought (non-blocking): The Opus→Sonnet swap delivers the dominant cost cut on its own; the low effort dial is what introduces the sandbox-wide env propagation and the subAgentEnv shield. Since the trial arm bundles model+effort, it won't isolate the marginal low-vs-default saving — if trial telemetry can split orchestrator thinking-token cost, worth confirming the low dial pays for its added mechanism.
| allowedTools, | ||
| permissionMode: "bypassPermissions", | ||
| abortController: abort, | ||
| env: subAgentEnv(process.env), |
There was a problem hiding this comment.
question (non-blocking): The shield assumes the SDK treats options.env as a full replacement rather than a merge over process.env; since the package is npm-ci'd at run time, this isn't checkable where the unit tests run. Would it be worth having a sub-agent echo whether CLAUDE_CODE_EFFORT_LEVEL is present in its environment during the gating trial, to make the shield an observed property rather than an SDK-version assumption?
| allowedTools, | ||
| permissionMode: "bypassPermissions", | ||
| abortController: abort, | ||
| env: subAgentEnv(process.env), |
There was a problem hiding this comment.
note (non-blocking): The unit test covers the pure subAgentEnv helper, but nothing exercises createSdkRunner actually passing env: subAgentEnv(process.env) to the SDK — the wiring that does the real work is verified only by inspection. This matches the module's no-SDK-in-unit-tests design, so it's a convention-aligned gap rather than a demand.
Review live A/BBaseline: Ruler: matcher deterministic+arbiter; corpus cc11988c0918 (9 cases).
Adversarial hard gate: PASSED on the candidate arm. Agent failures
Single-run-stable rows: recall, verdict agreement, regressions, adversarial gate. Judge quality and noise are not: they jitter run-to-run at this corpus size, and a regressed reviewer can score HIGHER on judge quality (fewer, surer comments each read better). Recall against the labeled specs is the load-bearing metric. Measured noise floor (identical arms, run 29069228968, 2026-07-10, 6 arm-samples, full corpus x3, pre-arbiter; budget skips left the samples on unequal case sets, so these v1 bands also carry case-mix variance): must-catch recall 54%-86% (sd 10%), verdict agreement 75%-100% (sd 9%), noise (unmatched posted) 50%-60% (sd 3%), judge mean quality 82%-86% (sd 2%). A single-run delta whose arms both sit inside a band is indistinguishable from run-to-run wobble; use |
Round-1 trial report: Sonnet-low orchestrator vs the prior lifecycle armsSeeded memory-expiration content, identical to the prior lifecycle's round 1 ( 1. Headline
Two cost accountings diverge sharply and both are real: by the metric the prior tables used, sonnet-low is 29% under S4 and 41% under S0; by the proxy meter that enforces the cap, all three runs are within 2% of each other ( 2. Defect by defect
Recall on the seeds is 2/2 on every arm. The load-bearing difference: on the prior S4 pin both seeds survived only because OTHER reviewers happened to cover for the voided correctness dimension; on this pin the correctness reviewer itself delivered them, with better anchors and a committable suggestion. 3. Defect 13, answered (the "correctness output unavailable" question)#41013's round 1 posted with 4. Quality and noise
5. First live evidence on the #288 features
6. Orchestrator economics, honestlySonnet-low did NOT cheapen the orchestrator line by the naive 40%: turn count nearly doubled (26 vs 14), so SDK-estimated orchestrator cost rose ($2.55 vs $1.99) while proxy credits landed around 153 vs an estimated ~100 for the Opus orchestrator. Low effort takes smaller steps; the per-token discount paid for the extra steps. The big reported-metric win (421 vs 593) appears to come mostly from engine-side accounting of the cheaper model against the same cache traffic. Worth one arm at 7. Issues opened by this round (ledger)
8. Status and spendRound 1 complete and scored. Lifecycle paused before the re-review rounds per the operator. Spend so far (proxy units): 1,016 superseded + 1,062 scored, roughly $21 of the approved ~$45+margin envelope. Next steps when unpaused: fix the detection-env leak, push |
|
Closing unmerged after the round-1 trial (report above): the outcome matches the operator's prior. The demotion is mechanically sound (clean transcription, gate-conformant queue, correct code-written cache record), but the orchestrator line did not actually get cheaper where it counts: low effort roughly doubled the turn count (26 vs 14), so SDK-estimated orchestrator cost rose ($2.55 vs $1.99) and proxy-metered credits landed around 153 vs ~100 for the Opus orchestrator, while the headline "421 vs 593" delta was mostly engine-side accounting of the cheaper model against the same cache traffic, not reduced spend. The swap also surfaced a real integration hazard: the engine env (model + What survives this PR's closure, since it ran on the stack below rather than on this change: the correctness dimension delivered through a drifted contract shape (the #288 structured finals plus salvage, resolving the defect-13 class that voided #41013's rounds), the deterministic cache record's first live write, and the scripted-only pipeline of #289. The trial ledger items land with the stack: cross-file duplicate posting of one defect (dedup window), If the orchestrator model question is revisited, the two measured leads are an |
|
Correction to one line of the round-1 report above: the AddDate catch was described as arriving "with better anchors and a committable suggestion", but no committable suggestion actually posted. The reviewer authored the one-line fix, in its drifted ReportFindings-style output under |
1 participant
The orchestrator moves to Claude Sonnet 5 at low reasoning effort. Stacked on #289 (task mode removed) over #288 and the rest of the stack; top of the stack.
Why this is safe to try now
With #289, the orchestrator's whole job is: stage the review threads, answer the router's tier questions, invoke three CLIs, transcribe the submission plan into safe-output calls, decide Steps 7-8, and upload the artifact. The reasoning lives in the sub-agents, which pin their own models (Fable/Opus) and are untouched here. The two guardrails that bound a weaker orchestrator: the dispatch-conformance gate compares every queued output against the staged plan and turns transcription drift into a red run rather than a wrong review, and the deterministic cache writer refuses to record anything the plan and queue do not corroborate.
Why bother
The lifecycle trial measured the scripted orchestrator's residual at $2.0-3.1/run (of $5.9-7.4 total), the single largest non-reviewer cost line. Sonnet 5 is 40% of Opus pricing per token, low effort cuts turn overhead further, and #289's prompt diet (~780 lines) shrinks the per-turn cache-read line on top.
Mechanics
engine.model: claude-sonnet-5(Sonnet 5 has no dated alias; the bare id is the pin), plusCLAUDE_CODE_EFFORT_LEVEL: "low"in the engine env — the documented effort mechanism for headless runs, outranking flags and settings.CLAUDE_CODE_EFFORT_LEVELstripped from their environment (subAgentEnvindispatch-runner.ts, unit-tested). Reviewer effort must never silently follow the orchestrator dial.claude-sonnet-5in its api-proxy credits table (verified against the tag), so orchestrator calls are metered, not rejected; amodels:frontmatter entry keeps gh-aw's cost display correct, same pattern as the Fable entry.Gating
Live-trial-gated, per the standing rule: the next seeded webapp lifecycle runs two arms on identical pushes — control pinned to #289's branch (Opus orchestrator), treatment pinned to this branch — so one round prices the demotion and validates the scripted-only pin plus the #288 features at the same time. Watch items for the trial: red-run (gate-block) rate, thread-staging fidelity (reconciler inputs), Step 7 comment quality, and the early-exit judgment.
Verification
1253 tests green, typecheck and eslint clean, frontmatter compiles clean under gh-aw v0.81.6 (compiled lock verified to carry
agent_model: claude-sonnet-5and the effort env on the engine step).