Skip to content

chore(config): revert per-agent runtime/model routing (#7116) - #7168

Closed
waynesun09 wants to merge 1 commit into
mainfrom
revert-diverse-routing
Closed

chore(config): revert per-agent runtime/model routing (#7116)#7168
waynesun09 wants to merge 1 commit into
mainfrom
revert-diverse-routing

Conversation

@waynesun09

Copy link
Copy Markdown
Member

What

Reverts the per-agent runtime/model routing merged today (#7116), restoring the previous config: every agent on Claude Code / opus, no models.aliases.

Why — the fleet is failing at the first model call

First fleet review after the merge (#7167, run 34365183694): Bootstrap resolved all nine personas, then the orchestrator's first request failed:

404 Publisher model `projects/…/locations/global/publishers/anthropic/models/claude-sonnet-5` was not found or your project does not have access
→ Result: ERROR · Validation failed: output/agent-result.json not found · Agent exit code: 1

The Vertex project the fleet runs on does not serve claude-sonnet-5. The alias sonnet: claude-sonnet-5 is on every sonnet parent (review, fix, triage, retro), and a pinned alias has no fallback, so every one of those agents fails on every trigger until this is reverted. code (Grok) and prioritize (Gemini) may or may not be served there — nothing has proven it on the fleet project, so the whole routing goes back until it is.

The local validation runs were done against a project that serves all three ids; the fleet's project is a different one. That is the mistake — the served-model check must be run against the project the fleet actually uses, which is the secret FULLSEND_GCP_PROJECT_ID, not a probe from a laptop.

Re-apply plan

  1. Confirm which ids the fleet project serves — a one-line fullsend run from CI against it, or an admin rawPredict probe from the project itself.
  2. Re-apply only the routing that project serves; keep sonnet on the default alias unless claude-sonnet-5 is enabled there.

The first fleet review after #7116 merged failed at the orchestrator's
first model call: Vertex returned 404 for claude-sonnet-5 in the project
the fleet runs on (#7167, run 34365183694). The
sonnet alias sits on every sonnet parent and has no fallback. Restore
the previous config until the served models are confirmed.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
@waynesun09 waynesun09 added the fullsend-no-fix Skip bot-triggered fix agent runs label Sep 9, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 Review · ❌ Terminated · Started 2:57 PM UTC · Ended 2:59 PM UTC

Commit: 2b72ad4 · View workflow run →

@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure (validation failed after 1 iteration(s)) · Started 2:57 PM UTC · Completed 2:59 PM UTC

Commit: 2b72ad4 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high

@waynesun09

Copy link
Copy Markdown
Member Author

Not needed: the cause was the fleet project (prod) not serving claude-sonnet-5 / xai/grok-4.6, and those were enabled in its Model Garden on 2026-09-09. A natural fleet review then succeeded on the merged config (fullsend run 34367379698: nine personas resolved, seven persona-tagged dispatches across claude-sonnet-5, gemini-3.8-flash and xai/grok-4.6, cost $2.90). Closing without merge.

@waynesun09 waynesun09 closed this Sep 9, 2026
@waynesun09
waynesun09 deleted the revert-diverse-routing branch September 9, 2026 15:21
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 9, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 3:23 PM UTC · Completed 3:28 PM UTC

Commit: 2b72ad4 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.00

@fullsend-ai-retro

Copy link
Copy Markdown

What happened: PR #7116 pinned sonnet → claude-sonnet-5 (plus other per-agent model routing) in .fullsend/config.yaml, validated against a dev/laptop GCP project that served all referenced model ids. The fleet's production Vertex project does not. The first fleet review after merge (PR #7167, run 34365183694) failed immediately with 404 Publisher model .../claude-sonnet-5 was not found; since the sonnet alias has no fallback, every sonnet-routed agent (review, fix, triage, retro) went down fleet-wide. waynesun09 opened PR #7168 to revert the routing. Its own review agent (run 34366865202) hit the identical 404 and crashed the same way — the reviewer's dispatch still resolved through the pre-revert sonnet alias on main, so the fix-in-flight could not be reviewed by the system it was fixing (confirmed from logs: clean pi exited 0, no retry or fallback model attempted; agent definitions resolve from fullsend-ai/agents@main). Independently, a GCP admin enabled claude-sonnet-5/xai/grok-4.6 in the fleet project's Model Garden; a subsequent natural fleet review then succeeded on the original (unreverted) config, and #7168 was closed without merging since the routing itself wasn't the defect.

Assessment: This is a genuine fleet-wide-outage incident, but not a new failure class — it's already covered by a substantial, overlapping open-issue backlog in both fullsend-ai/fullsend and fullsend-ai/agents. Per the before-proposing check, no new proposal is warranted here; instead this run supplies corroborating evidence for existing issues:

  • fullsend#7026 (no automatic fallback for an aliased model): this incident is a direct fleet-wide instance — one pinned sonnet alias took down review, fix, triage and retro simultaneously.
  • fullsend#7154 (review agent should fall back to a default model, filed today): the review-agent crash on chore(config): revert per-agent runtime/model routing (#7116) #7168 is a clean second data point — the reviewer couldn't even review the PR meant to fix the very outage it was suffering from.
  • agents#1186 (sonnet alias resolves to an unavailable model despite fleet pin): matches this incident's root cause almost exactly.
  • fullsend#6964 / fullsend#6883 (pre-dispatch / pre-sandbox model-availability preflight): would have caught this before merge; the PR author's own write-up independently reached the same "validate against the real fleet project, not a laptop" conclusion.
  • fullsend#423 (ensure GCP projects have required models enabled): root-cause remediation — this incident is further evidence the gap is still open.
  • agents#1100 (retro harness previously blocked fleet-wide by one unavailable model, 13 consecutive failures): shows this is a recurring pattern rather than a one-off.

Given how many overlapping open issues already exist for this exact failure mode (at least 6 across the two repos, two more closed in the last week), the backlog may benefit from human consolidation — but that's a triage call, not a new issue this retro should file. No proposals filed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fullsend-no-fix Skip bot-triggered fix agent runs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant