-
Notifications
You must be signed in to change notification settings - Fork 1
review: orchestrator on Sonnet 5 at low effort #290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| "review": minor | ||
| --- | ||
|
|
||
| The orchestrator moves to Claude Sonnet 5 at low reasoning effort (`engine.model: claude-sonnet-5` plus `CLAUDE_CODE_EFFORT_LEVEL: low` in the engine env). With task mode removed the orchestrator's job is transcription plus a handful of small judgments (thread staging, the router tier questions, Steps 7-8), the dispatch-conformance gate turns any transcription drift into a red run rather than a wrong review, and the lifecycle trial measured the orchestrator line at $2.0-3.1/run on Opus, the largest non-reviewer cost; Sonnet at roughly 40% of Opus pricing with a low-effort dial cuts most of it while keeping near-Opus instruction-following. Sub-agents are untouched: they pin their own models, and the dispatcher's runner strips `CLAUDE_CODE_EFFORT_LEVEL` from the sub-agent environment so the reviewer roles can never silently inherit the orchestrator's effort dial. The pinned firewall (gh-aw-firewall v0.27.27) already prices `claude-sonnet-5`; a `models:` entry keeps gh-aw's cost display correct. Live-trial-gated: the seeded lifecycle that validates the scripted-only pin runs this branch as its second arm against an Opus-orchestrator control. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,6 +11,27 @@ | |
|
|
||
| import type {AgentRequest, AgentResult, AgentRunner} from "./dispatch"; | ||
|
|
||
| /** | ||
| * The environment the sub-agents run under: the sandbox env minus the | ||
| * orchestrator's effort dial. The engine sets CLAUDE_CODE_EFFORT_LEVEL for | ||
| * the ORCHESTRATOR (Sonnet at low effort), and env vars reach every process | ||
| * in the sandbox — without this filter the SDK-spawned reviewers would | ||
| * silently inherit low effort too, degrading the reasoning roles the | ||
| * orchestrator dial was never meant to touch. Sub-agents keep the harness | ||
| * default (their definitions' effort annotations remain the human-facing | ||
| * table in the README until gh-aw grows a per-agent effort field). | ||
| */ | ||
| export const subAgentEnv = ( | ||
| env: Record<string, string | undefined>, | ||
| ): Record<string, string> => | ||
| Object.fromEntries( | ||
| Object.entries(env).filter( | ||
| (entry): entry is [string, string] => | ||
| entry[1] !== undefined && | ||
| entry[0] !== "CLAUDE_CODE_EFFORT_LEVEL", | ||
| ), | ||
| ); | ||
|
|
||
| /** | ||
| * Build the production runner. The SDK and zod are imported lazily here | ||
| * (both installed by the scripted-mode `npm ci` pre-agent step); zod is the | ||
|
|
@@ -57,6 +78,7 @@ export const createSdkRunner = async (): Promise<AgentRunner> => { | |
| allowedTools, | ||
| permissionMode: "bypassPermissions", | ||
| abortController: abort, | ||
| env: subAgentEnv(process.env), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question (non-blocking): The shield assumes the SDK treats
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. note (non-blocking): The unit test covers the pure |
||
| }; | ||
| // The structured-final channel (trial suggestion h): an in-process | ||
| // MCP tool whose handler runs the same contract parse the collection | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -7,6 +7,7 @@ import { | |||||
| type DispatchFs, | ||||||
| } from "./dispatch"; | ||||||
| import {computeDiffProvenance} from "./provenance"; | ||||||
| import {subAgentEnv} from "./dispatch-runner"; | ||||||
|
|
||||||
| /** | ||||||
| * Post-trial follow-up tests for the scripted dispatcher: the | ||||||
|
|
@@ -322,3 +323,15 @@ describe("open-thread suppression (trial suggestion g)", () => { | |||||
| expect(result.claims).toHaveLength(1); | ||||||
| }); | ||||||
| }); | ||||||
|
|
||||||
| describe("subAgentEnv (the orchestrator effort dial never reaches sub-agents)", () => { | ||||||
| it("strips CLAUDE_CODE_EFFORT_LEVEL and undefined values, keeps the rest", () => { | ||||||
| expect( | ||||||
| subAgentEnv({ | ||||||
| CLAUDE_CODE_EFFORT_LEVEL: "low", | ||||||
| REVIEW_REPO_ROOT: "/work", | ||||||
| EMPTY: undefined, | ||||||
| }), | ||||||
| ).toEqual({REVIEW_REPO_ROOT: "/work"}); | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. note (non-blocking):
Suggested change
Low-confidence (1)
|
||||||
| }); | ||||||
| }); | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -167,20 +167,34 @@ observability: | |
| headers: | ||
| x-sentry-auth: ${{ secrets.GH_AW_OTEL_SENTRY_AUTHORIZATION }} | ||
|
|
||
| # Pin the orchestrator to a specific model version rather than a floating tier alias, so | ||
| # the review doesn't silently change behavior when a new Opus ships. If we use Opus, we | ||
| # use Opus 4.8. Sub-agents pin their own versions in their frontmatter below. | ||
| # Pin the orchestrator to a specific model version rather than a floating tier | ||
| # alias, so the review doesn't silently change behavior when a new model ships. | ||
| # Sub-agents pin their own versions in their frontmatter below; the reasoning | ||
| # stays on their models (Fable/Opus). The orchestrator itself is Sonnet at LOW | ||
| # effort: with task mode removed its job is transcription plus a handful of | ||
| # small judgments (thread staging, the tier questions, Steps 7-8), the | ||
| # dispatch-conformance gate turns any transcription drift into a red run | ||
| # rather than a wrong review, and the trial measured the orchestrator line at | ||
| # $2-3/run on Opus — the single largest non-reviewer cost. claude-sonnet-5 is | ||
| # its own pin (Sonnet 5 has no dated alias). | ||
| # | ||
| # The `env:` overrides gh-aw's 60s Bash tool timeout defaults (compile-verified: | ||
| # these replace the generated values on the engine execution step). Needed by the | ||
| # scripted dispatch mode (ROUTING `dispatch scripted`): the orchestrator invokes | ||
| # the deterministic dispatcher (lib/dispatch.ts) as ONE blocking Bash call that | ||
| # waits for the whole sub-agent fan-out, which takes minutes, not seconds. The | ||
| # job-level timeout-minutes still bounds the run. | ||
| # CLAUDE_CODE_EFFORT_LEVEL sets the orchestrator's reasoning effort (it | ||
| # outranks the CLI flag and settings.json). It reaches the whole sandbox env, | ||
| # so the dispatcher's runner (lib/dispatch-runner.ts) strips it from the | ||
| # sub-agent environment — reviewer effort must never silently follow the | ||
| # orchestrator dial. | ||
| # | ||
| # The BASH_* env overrides replace gh-aw's 60s Bash tool timeout defaults | ||
| # (compile-verified: these replace the generated values on the engine | ||
| # execution step). Needed because the orchestrator invokes the deterministic | ||
| # dispatcher (lib/dispatch.ts) as ONE blocking Bash call that waits for the | ||
| # whole sub-agent fan-out, which takes minutes, not seconds. The job-level | ||
| # timeout-minutes still bounds the run. | ||
| engine: | ||
| id: claude | ||
| model: claude-opus-4-8 | ||
| model: claude-sonnet-5 | ||
| env: | ||
| CLAUDE_CODE_EFFORT_LEVEL: "low" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thought (non-blocking): The Opus→Sonnet swap delivers the dominant cost cut on its own; the |
||
| BASH_DEFAULT_TIMEOUT_MS: "60000" | ||
| BASH_MAX_TIMEOUT_MS: "1200000" | ||
| timeout-minutes: 20 | ||
|
|
@@ -217,6 +231,15 @@ models: | |
| output: 5.0e-05 | ||
| cache_read: 1.0e-06 | ||
| cache_write: 1.25e-05 | ||
| # The orchestrator's model (same caveat: keeps gh-aw's cost display | ||
| # correct; the firewall api-proxy's own table, v0.27.27, already | ||
| # prices it). | ||
| claude-sonnet-5: | ||
| cost: | ||
| input: 3.0e-06 | ||
| output: 1.5e-05 | ||
| cache_read: 3.0e-07 | ||
| cache_write: 3.75e-06 | ||
|
|
||
| # The shared review workflow is more than this markdown file: its deterministic | ||
| # pieces (the finding schema and validator today; the router, computed verdict, and | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion (non-blocking): This strips exactly one key, so any future orchestrator-only engine-env dial (or a model bump with a different default effort) would pass through to sub-agents by default. Since scripted dispatch owns sub-agent spawns, consider inverting to a pass-through allow-list plus an explicit per-role effort, so future engine-env additions are opt-in rather than leak-by-default.