fix(#1116): use deployment-agnostic model aliases in review sub-agents - #1181
Conversation
Replace pinned claude-sonnet-4-6@default model strings with the sonnet alias in 5 review sub-agent personas (cross-repo-contracts, docs-currency, intent-coherence, risk-assessment, style-conventions). The pinned version resolved to claude-sonnet-4-5@20250929, which is unavailable on the Vertex deployment, causing dispatch failures and fallback to the parent opus model. Add ANTHROPIC_DEFAULT_SONNET_MODEL=claude-sonnet-4-6 to env/gcp-vertex.env so the Claude Code runtime resolves the sonnet alias to the model version served by the fleet Vertex projects. Add explicit tools: Read, Grep, Glob to security-triage.md so the persona carries its own read-only tool declaration, preparing for the dispatch-by-persona-name change in fullsend#7031. Note: pre-commit hooks were not fully run. pre-commit could not complete (network restriction in sandbox). Hooks were run directly where possible (trailing whitespace, EOF newline, merge conflict, secret scan). skillsaw lint passed. Closes #1116
|
🤖 Finished Review · ✅ Success · Started 11:08 PM UTC · Completed 11:21 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.60 |
ReviewFindingsMedium
Previous runReviewFindingsMedium
Low
Previous run (2)ReviewFindingsMedium
Previous run (3)ReviewFindingsMedium
Next steps:
Previous run (4)ReviewFindingsMedium
Low
Labels: PR modifies review agent sub-agent configurations exclusively Next steps:
Previous run (5)ReviewFindingsMedium
Next steps:
|
|
🤖 Finished Fix · ✅ Success · Started 11:23 PM UTC · Completed 11:28 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.20 |
Add `permissionMode: dontAsk` to security-triage.md frontmatter, consistent with all other tool-bearing sub-agents. Without it, the newly-added tools could trigger interactive permission prompts that stall the automated review pipeline. Note: pre-commit hooks were not run. `pre-commit` could not complete (infrastructure failure — sandbox network blocks git fetch to remote hook repos). YAML validation and trailing-whitespace checks were run directly. skillsaw lint passed (Grade A, 0 errors). Addresses #1181
🔧 Fix agent — iteration 1 (bot-triggered)Added permissionMode: dontAsk to security-triage.md frontmatter to prevent interactive permission prompts in the automated review pipeline. The protected-path finding is a governance note requiring human approval, not a code fix. Fixed (1):
Disagreed (1):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 11:30 PM UTC · Completed 11:44 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.69 |
|
🤖 Finished Fix · ✅ Success · Started 11:46 PM UTC · Completed 11:53 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.05 |
Add `background: true` to security-triage.md frontmatter for consistency with all other sub-agent definitions in the directory. Note: pre-commit hooks were not run. `pre-commit` could not complete (infrastructure failure — sandbox cannot fetch remote hook repos), and the direct-execution fallback was used for basic checks only. Lint (`make lint`) could not run due to missing `uvx` in sandbox. Addresses #1181
🔧 Fix agent — iteration 2 (bot-triggered)Added background: true to security-triage.md frontmatter for consistency with all other sub-agent definitions. The protected-path finding is a governance note requiring human approval, not a code fix. Fixed (1):
Disagreed (1):
Tests: passed Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 11:55 PM UTC · Completed 12:11 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.96 |
|
🤖 Finished Fix · ✅ Success · Started 12:12 AM UTC · Completed 12:20 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.44 |
Change background to false in security-triage.md frontmatter to match the pr-review SKILL.md requirement (step 3c-1) that this agent runs synchronously. Its classification output feeds into step 3d context package assembly. Matches risk-assessment.md pattern for pre-pass agents. Addresses #1181
🔧 Fix agent — iteration 3 (bot-triggered)Fixed frontmatter-dispatch-contradiction by changing background: true to background: false in security-triage.md, matching the SKILL.md requirement for synchronous pre-pass agents. Protected-path finding acknowledged as requiring human approval (not a code fix). Fixed (1):
Disagreed (1):
Tests: passed
Decision points
Next steps:
|
|
🤖 Finished Review · ✅ Success · Started 12:21 AM UTC · Completed 12:33 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.54 |
Superseded by updated review
|
🤖 Finished Review · ✅ Success · Started 2:22 AM UTC · Completed 2:39 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.07 |
|
🤖 Review · Commit: |
|
/fs-fix-stop |
|
Fix agent disabled for this PR. Remove the |
The env file is mounted by every harness here, so the sonnet pin applies to all agents and, being a pin, has no CLI fallback. Say so next to the export in eleven lines: what it fixes, how a consumer whose Vertex project lacks the id overrides it (env.sandbox from a composing harness, exported after .env.d), that it is bumped together with fullsend's pi alias table, and why opus and haiku stay unpinned. Add the variable to the review Variables table so the override is documented where the other env.sandbox overrides are. Assisted-by: Claude (fix), Claude (review), Codex gpt-5.6-sol (review), Grok 4.6 (review) Signed-off-by: Wayne Sun <gsun@redhat.com>
3102398 to
e3cdf14
Compare
|
🤖 Finished Review · ✅ Success · Started 3:00 AM UTC · Completed 3:15 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.73 |
waynesun09
left a comment
There was a problem hiding this comment.
Approved after takeover: A/B-verified sonnet pin on the fleet image, three squad review rounds (Claude, Codex, Grok), all threads resolved, CI green on e3cdf14.
|
🤖 Finished Retro · ✅ Success · Started 1:12 PM UTC · Completed 1:27 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.90 |
Retro: PR #1181 — deployment-agnostic model aliases in review sub-agentsTimelineIssue #1116 was filed by the retro agent after observing model dispatch failures in a review run on fullsend-ai/fullsend. Four of seven review sub-agents failed because Human issued This triggered a 3-iteration fix loop across 6 review cycles:
Human then took over ( Total cost: ~$21.87 (code $1.59, fix $3.69, review $16.59 across 6 completed reviews). If the code agent had added all three frontmatter fields correctly, estimated cost would have been ~$4.19. Review QualityExcellent. All three review findings were legitimate, actionable, and confirmed by the human:
The review agent demonstrated strong cross-file reasoning by checking SKILL.md dispatch logic against the persona frontmatter. Existing Issue Evidence
Proposals filed |
Summary
model: claude-sonnet-4-6@defaultwithmodel: sonnetin the 5 review sub-agent personas that carried it (cross-repo-contracts,docs-currency,intent-coherence,risk-assessment,style-conventions). Personas name an alias, never a generation; the runtime resolves it (fullsendpiModelAliaseson pi, the Claude Code CLI on Claude). Unpinned, the CLI's Vertex default forsonnetisclaude-sonnet-4-5@20250929, which the fleet projects do not serve — that is the dispatch failure and opus fallback seen in Review sub-agent model config should use deployment-agnostic aliases to prevent dispatch failures #1116, Review orchestrator passes roster-table model alias instead of frontmatter model to sub-agents #256 and fullsend#6666.ANTHROPIC_DEFAULT_SONNET_MODEL=claude-sonnet-4-6toenv/gcp-vertex.envas the fleet-wide base pin for thesonnetalias on the Claude Code runtime. Every harness in this repo mounts the file, so this applies to all agents, not review only (a BYO consumer that copied the file rather than referencing it does not receive the pin). A pinned alias has no CLI startup fallback: a consumer whose Vertex project does not serveclaude-sonnet-4-6overrides it from a harness that composes this one (base:or anoverlays:entry) and setsenv.sandbox.ANTHROPIC_DEFAULT_SONNET_MODEL—env.sandboxis exported after.env.dis sourced (ADR 0055); fullsend's env.sandbox/host_files lint warning on that key is the intended collision. Withopusunpinned, a run that reaches the CLI with no--modeldefaults to this sonnet id; every harness here setsmodel:, so only a consumer harness that omits it is affected. The comment block in the file records this, the cross-reference to fullsend's pi alias table (bumped together, fullsend#7028 / fullsend#7031), and whyopusstays unpinned.tools: Read, Grep, Glob,permissionMode: dontAsk,background: falsetosecurity-triage.md. Inert today (it is dispatched assubagent_type: Explore); prepares the persona for dispatch by name in fullsend#7031.Context
Observed in fullsend PR #6010 review run — 4 of 7 review sub-agents failed initial dispatch because the pinned
claude-sonnet-4-6@defaultstring fell through to the CLI's Vertex default (claude-sonnet-4-5@20250929), which the fleet project does not serve; the baresonnetalias lands on the same id without the pin added here. The review agent recovered by re-dispatching without model overrides, so all 7 sub-agents ran on opus instead of the intended 3 opus + 4 sonnet split.Testing
make lint);env/gcp-vertex.envis outside skillsaw's scope and was verified by sourcing in the A/B runs below.eval-results-reviewartifact of run 33931136204 (commit bf7c925, which differs from head only by the inertbackground: true → falseand the comment/docs commits that carry no runtime change): the orchestrator dispatchedmodel: sonnetfor risk-assessment, style-conventions, intent-coherence and docs-currency andmodel: opusfor correctness, security and challenger; every sonnet child ran onclaude-sonnet-4-6(claude-sonnet-4-6:streamRawPredictin the fleet region) with zero dispatch errors; all judges at pass_rate 1.0. That sandbox ran Claude Code 2.1.258 (the current release image; fullsend main pins 2.1.260).reviewharness at this branch, imagefullsend-code@sha256:ea2a31f3…= Claude Code 2.1.258, Vertex dev project,--no-post-script, nothing posted): with the env line everysonnetchild ranclaude-sonnet-4-6(risk-assessment, intent-coherence, style-conventions, docs-currency) and everyopuschildclaude-opus-4-6; without it the samesonnetchildren ranclaude-sonnet-4-5-20250929. The env line is load-bearing for sub-agents, not only the main model. In both runs the orchestrator passed an explicitmodelargument on 13 of 13 dispatches, transcribed from the persona frontmatter (SKILL.md steps 4 and 6d do not instruct it to), so the rename is honoured by the LLM rather than enforced by the harness — deterministic enforcement is fullsend#7031.ANTHROPIC_DEFAULT_SONNET_MODELgoverns thesonnetalias for sub-agents as well as the main model (Claude Code model-config docs; single resolver in the CLI), and the env file reaches the CLI process viahost_files→/sandbox/workspace/.env.d/gcp-vertex.env→ sourced by.envbeforeclaudelaunches. Note: fullsend'sdocs/contributing/runtime-implementation.mdcurrently states the opposite for Vertex ("ANTHROPIC_DEFAULT_*_MODELdoes not steer the request there"); the A/B above contradicts it; follow-up filed as docs: runtime-implementation.md says ANTHROPIC_DEFAULT_*_MODEL does not steer aliases on Vertex — it does (verified A/B on 2.1.258) fullsend#7039.functional-tests-completeon this head was a label-event self-cancel of the in-flight review job (the review bot'srequires-manual-reviewlabel started a second run in the same concurrency group); the rerun of the same run is green.Closes #1116
Post-script verification
agent/1116-depin-subagent-model-aliases)