Skip to content
7 changes: 7 additions & 0 deletions .changeset/review-anthropic-pricing-overlay.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"review": patch
---

Meter AI credits at Khan's real Anthropic rate rather than list price. Khan bills list minus 50%, but the firewall api-proxy prices credits from a list-price catalog baked into its image, so `max-ai-credits` stops a run at half the spend it implies and the router's `maxUsd` targets are denominated in a currency twice as expensive as the real one. Adds gh-aw's `models.providers` overlay at 50% of list for every model the reviewer can run. Since a credit now means $0.01 of real spend, the existing caps and targets are correct as written, so `budgets.ts`, `credit-cap.ts` and the counters are untouched.

Inert until gh-aw v0.84.x is stable: `apiProxy.providers` needs firewall v0.27.43, and gh-aw v0.83.4 defaults to v0.27.42 and drops the key silently. Nothing else is required to activate it; recompiling on v0.84.x is enough.
92 changes: 91 additions & 1 deletion workflows/review/review.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,97 @@ post-steps:
echo "::warning title=dispatch-conformance gate::gate could not run (infra failure; review not blocked)"
exit 0

# Cost guardrails (AI credits; 1 credit = $0.01). gh-aw >= v0.79 bakes in
# Anthropic pricing overlay, so an AI credit means $0.01 of what Khan actually
# pays.
#
# INERT UNTIL gh-aw v0.84.x GOES STABLE. `apiProxy.providers` was added to
# awf-config-schema.json in AWF v0.27.43, and gh-aw gates emitting it on that
# floor (`AWFAPIProxyProvidersMinVersion`) because older AWF strict config
# validation rejects unknown apiProxy properties. gh-aw v0.83.4 (the version
# this lock was compiled with, and still `releases/latest`) defaults to AWF
# v0.27.42, one patch below, so it SILENTLY DROPS this block: it compiles
# clean, the rates land only in the informational `GH_AW_INFO_MODEL_COSTS` env
# var, and metering stays at list price. gh-aw raises `DefaultFirewallVersion`
# to v0.27.43 as of v0.84.0, but v0.83.5/v0.84.0/v0.84.1 are all prereleases,
# so recompiling is blocked on that line going stable rather than on any edit
# here. Do NOT pin the extension to a prerelease to force it: nothing in CI
# runs `gh aw compile`, so the next compile on stable would silently drop

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (non-blocking): Whole change hinges on a manual, un-CI-enforced source→lock recompile that can silently revert. The diff edits the distributable source (review.md) but deliberately leaves review.lock.yml — the artifact that actually runs — out of sync, and the comment itself notes nothing in CI recompiles, so activation and its persistence rely entirely on humans remembering to recompile and verify. A CI guard that fails when review.md and its compiled lock diverge (or asserts providers appears in both awf-config payloads once the AWF floor is met) would convert this silent-revert footgun into an enforced invariant.

# `providers` again with no test failure.
#
# VERIFY AFTER ANY TOOLCHAIN BUMP: `providers` must appear inside the
# `apiProxy` object of both awf-config payloads in review.lock.yml. Its
# presence in `GH_AW_INFO_MODEL_COSTS` alone means the overlay is NOT live.
#
# Deliberately NOT solved by pinning `sandbox.agent.version: v0.27.43`: the
# sandbox block above documents that a version is re-pinned here only to hold a
# firewall release BACK, never to move one forward, and a forward pin would
# recreate the stale-floor failure that broke run 30290472047.
#
# WHY THE OVERLAY: Khan bills Anthropic list minus 50%, but the firewall api-proxy meters
# credits against a list-price catalog baked into its image, so every dollar
# figure downstream (the caps below, the router's `maxUsd` soft targets, the
# cost counters) reads 2x high and a run is cut off at half the real spend its
# cap implies.
#
# `models.providers` is gh-aw's operator overlay: it compiles to the firewall's
# `apiProxy.providers` and is the highest-precedence pricing source, ahead of
# runtime provider discovery, the curated table, and the bundled models.dev
# catalog (awf-config-spec 10.7.1). It applies to the threat-detection proxy as
# well as the agent's. Rates are per-token USD at 50% of Anthropic list.
#
# MAINTENANCE: entries are matched per model, and an unlisted model silently
# falls through to full list price rather than erroring. Add an entry when the
# engine model changes, and re-halve these when Anthropic list prices move.
# Do NOT collapse these to a bare `claude-opus-4` prefix: prefix matching would
# also capture opus-4-0/4-1, which list at 3x the 4-5+ rate.
models:
Comment thread
khan-actions-bot marked this conversation as resolved.
providers:
anthropic:
models:
# Current engine model.
claude-opus-4-8:
cost:
input: "2.5e-06"
output: "1.25e-05"
cache_read: "2.5e-07"
cache_write: "3.125e-06"
# Engine models a consumer may select via an `engine:` override.
claude-opus-5:
cost:
input: "2.5e-06"
output: "1.25e-05"
cache_read: "2.5e-07"
cache_write: "3.125e-06"
claude-sonnet-5:
Comment thread
khan-actions-bot marked this conversation as resolved.
cost:
input: "1e-06"
output: "5e-06"
cache_read: "1e-07"
cache_write: "1.25e-06"
claude-haiku-4-5:
cost:
input: "5e-07"
output: "2.5e-06"
cache_read: "5e-08"
cache_write: "6.25e-07"
claude-fable-5:
cost:
input: "5e-06"
output: "2.5e-05"
cache_read: "5e-07"
cache_write: "6.25e-06"
# The pattern-triage sub-agent's pin (the cheap first pass; see its
# `model:` line below). Not an engine model, but dispatched inside
# the sandboxed agent step and metered by the same api-proxy.
claude-sonnet-4-6:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note (non-blocking): Sixth overlay model (claude-sonnet-4-6) is not in the description's five-row table. The description's What section says 'the five pinned models' and its table lists five, but the diff adds claude-sonnet-4-6 as a sixth entry. I confirmed it is genuinely dispatched (review.md:1502, the pattern-triage pin) and is covered by the changeset's broader 'every model the reviewer can run' wording, so this is a description/implementation mismatch rather than a defect — worth aligning the table with the six entries actually shipped.

cost:
input: "1.5e-06"
output: "7.5e-06"
cache_read: "1.5e-07"
cache_write: "1.875e-06"

# Cost guardrails (AI credits; 1 credit = $0.01 of real spend, given the
# pricing overlay above). gh-aw >= v0.79 bakes in
# defaults of 1000/run ($10) and 5000/day ($50). Disable the daily ceiling
# (-1) so reviews are never skipped on a busy PR day; the per-run cap below
# still bounds the cost of any single review.
Expand Down
Loading