review: move the whole reviewer roster to Opus 5 #294
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| "review": minor | ||
| --- | ||
|
|
||
| Move the reviewer roster to Opus 5 (`claude-opus-5`). `pattern-triage` stays on Sonnet 4.6 as the cheap first pass. | ||
|
|
||
| The specialist lenses were kept off Fable 5 because cyber safety classifiers can refuse benign security-focused analysis, and a refused lens is a silent coverage hole: it surfaces as a missing agent result, not an error. Opus 5 can also return `stop_reason: "refusal"` on cyber-adjacent input, so this move re-opens that hole rather than closing it. The detector is the weekly drift corpus, where a refusing lens craters must-catch recall on security-adjacent cases while every other metric looks normal. On that signature, put the `security-auth` lens back on `claude-opus-4-8` first. Moving `security-auth` with the roster rather than carving it out pre-emptively is deliberate: the refusal hazard is intermittent and unproven on Opus 5, the one-hop runtime fallback (a refused agent re-dispatches once on `claude-opus-4-8`, recorded as `fellBackTo`) bounds each occurrence, and a uniform roster leaves one signature to watch and one revert to make instead of a permanent carve-out justified by a hazard that may not materialize. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| /** | ||
| * Mechanical pricing gates for the model pins in review.md (#294 review | ||
| * feedback: the merge-ordering constraint "do not ship an un-priced pin" | ||
| * rested entirely on human memory of a draft PR). | ||
| * | ||
| * Two hazards, both prose-only until this file: | ||
| * | ||
| * - The `models.providers` overlay matches per model and an unlisted model | ||
| * silently bills at FULL list price (the overlay's own MAINTENANCE note); | ||
| * the overlay omitting `claude-sonnet-4-6` shipped exactly that way in an | ||
| * earlier draft of #314. | ||
| * - On the stable toolchain (gh-aw v0.83.x -> firewall v0.27.42) the | ||
| * `providers` block is dropped silently and the api-proxy's credit guard | ||
| * rejects a model its curated table does not price with a 400 before the | ||
| * request reaches the model (#266). `claude-opus-5` is not in that table, | ||
| * so the `default-ai-credits-pricing` fallback is load-bearing for every | ||
| * dispatch until a gh-aw release defaults the firewall to v0.27.43+. | ||
| * | ||
| * DELETE the fallback test (only it) together with the fallback block when | ||
| * the toolchain moves; the coverage test is permanent. | ||
| */ | ||
| import {readFileSync} from "node:fs"; | ||
| import {join} from "node:path"; | ||
|
|
||
| import {describe, it, expect} from "vitest"; | ||
|
|
||
| const reviewMd = readFileSync(join(__dirname, "..", "review.md"), "utf8"); | ||
|
|
||
| /** The workflow frontmatter (between the first pair of --- fences). */ | ||
| const frontmatter = reviewMd.split(/^---$/m)[1] ?? ""; | ||
|
|
||
| /** | ||
| * Every model pin in the file: the engine's indented `model:` line in the | ||
| * frontmatter plus each sub-agent's `model:` line in its block frontmatter. | ||
| */ | ||
| const pins = [ | ||
| ...new Set( | ||
| [...reviewMd.matchAll(/^\s*model:\s*(claude-[a-z0-9.-]+)\s*$/gm)].map( | ||
|
Contributor
There was a problem hiding this comment. nitpick (non-blocking): The pin-extraction regex requires the model id to end the line, so a pin with a trailing inline comment silently escapes the coverage gate. Checked all 23 current pin lines in review.md — none carry inline comments today, so this is a latent gap introduced with the new test, not a live failure. Tolerating an optional trailing comment (or asserting the pin count against the number of agent blocks) would keep the gate from resting on that formatting convention. A sketch, not a committable replacement: |
||
| (match) => match[1], | ||
| ), | ||
| ), | ||
| ]; | ||
|
|
||
| /** | ||
| * The models the `providers` overlay prices: bare `claude-*:` mapping keys in | ||
| * the frontmatter (only the providers block declares them). | ||
| */ | ||
| const priced = new Set( | ||
| [...frontmatter.matchAll(/^\s+(claude-[a-z0-9.-]+):\s*$/gm)].map( | ||
| (match) => match[1], | ||
| ), | ||
| ); | ||
|
|
||
| describe("model pricing coverage (review.md frontmatter)", () => { | ||
| it("finds the pins and the overlay (guards the extraction itself)", () => { | ||
| // 22 agents plus the engine; a collapse to zero means the regexes | ||
| // rotted, not that the roster emptied. | ||
| expect(pins.length).toBeGreaterThanOrEqual(2); | ||
| expect(priced.size).toBeGreaterThanOrEqual(2); | ||
| }); | ||
|
|
||
| it("prices every pinned model in the providers overlay", () => { | ||
| const unpriced = pins.filter((pin) => !priced.has(pin)); | ||
| // An unlisted model silently bills at full list price; add an entry | ||
| // at 50% of its Anthropic list rate (see the MAINTENANCE note). | ||
| expect(unpriced).toEqual([]); | ||
| }); | ||
|
|
||
| it("keeps the stable-toolchain credit-guard fallback while claude-opus-5 is pinned", () => { | ||
| // Firewall v0.27.42's curated table does not price claude-opus-5; | ||
| // without `default-ai-credits-pricing` every dispatch 400s on the | ||
| // stable toolchain. Delete this test with the fallback block once a | ||
| // gh-aw release defaults the firewall to v0.27.43+. | ||
| if (pins.includes("claude-opus-5")) { | ||
| expect(frontmatter).toContain("default-ai-credits-pricing:"); | ||
| } | ||
| }); | ||
| }); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.