-
Notifications
You must be signed in to change notification settings - Fork 1
review: meter AI credits at Khan's real Anthropic rate, not list price #314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
b1187fe
d71a4ed
6435550
d0c7b72
90e373a
83ee887
879902b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| "review": patch | ||
| --- | ||
|
|
||
| Meter AI credits at Khan's real Anthropic rate rather than list price. Khan bills list minus 50%, but the firewall api-proxy prices credits from a list-price catalog baked into its image, so `max-ai-credits` stops a run at half the spend it implies and the router's `maxUsd` targets are denominated in a currency twice as expensive as the real one. Adds gh-aw's `models.providers` overlay at 50% of list for every model the reviewer can run. Since a credit now means $0.01 of real spend, the existing caps and targets are correct as written, so `budgets.ts`, `credit-cap.ts` and the counters are untouched. | ||
|
|
||
| Inert until gh-aw v0.84.x is stable: `apiProxy.providers` needs firewall v0.27.43, and gh-aw v0.83.4 defaults to v0.27.42 and drops the key silently. Nothing else is required to activate it; recompiling on v0.84.x is enough. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -299,7 +299,97 @@ post-steps: | |
| echo "::warning title=dispatch-conformance gate::gate could not run (infra failure; review not blocked)" | ||
| exit 0 | ||
|
|
||
| # Cost guardrails (AI credits; 1 credit = $0.01). gh-aw >= v0.79 bakes in | ||
| # Anthropic pricing overlay, so an AI credit means $0.01 of what Khan actually | ||
| # pays. | ||
| # | ||
| # INERT UNTIL gh-aw v0.84.x GOES STABLE. `apiProxy.providers` was added to | ||
| # awf-config-schema.json in AWF v0.27.43, and gh-aw gates emitting it on that | ||
| # floor (`AWFAPIProxyProvidersMinVersion`) because older AWF strict config | ||
| # validation rejects unknown apiProxy properties. gh-aw v0.83.4 (the version | ||
| # this lock was compiled with, and still `releases/latest`) defaults to AWF | ||
| # v0.27.42, one patch below, so it SILENTLY DROPS this block: it compiles | ||
| # clean, the rates land only in the informational `GH_AW_INFO_MODEL_COSTS` env | ||
| # var, and metering stays at list price. gh-aw raises `DefaultFirewallVersion` | ||
| # to v0.27.43 as of v0.84.0, but v0.83.5/v0.84.0/v0.84.1 are all prereleases, | ||
| # so recompiling is blocked on that line going stable rather than on any edit | ||
| # here. Do NOT pin the extension to a prerelease to force it: nothing in CI | ||
| # runs `gh aw compile`, so the next compile on stable would silently drop | ||
| # `providers` again with no test failure. | ||
| # | ||
| # VERIFY AFTER ANY TOOLCHAIN BUMP: `providers` must appear inside the | ||
| # `apiProxy` object of both awf-config payloads in review.lock.yml. Its | ||
| # presence in `GH_AW_INFO_MODEL_COSTS` alone means the overlay is NOT live. | ||
| # | ||
| # Deliberately NOT solved by pinning `sandbox.agent.version: v0.27.43`: the | ||
| # sandbox block above documents that a version is re-pinned here only to hold a | ||
| # firewall release BACK, never to move one forward, and a forward pin would | ||
| # recreate the stale-floor failure that broke run 30290472047. | ||
| # | ||
| # WHY THE OVERLAY: Khan bills Anthropic list minus 50%, but the firewall api-proxy meters | ||
| # credits against a list-price catalog baked into its image, so every dollar | ||
| # figure downstream (the caps below, the router's `maxUsd` soft targets, the | ||
| # cost counters) reads 2x high and a run is cut off at half the real spend its | ||
| # cap implies. | ||
| # | ||
| # `models.providers` is gh-aw's operator overlay: it compiles to the firewall's | ||
| # `apiProxy.providers` and is the highest-precedence pricing source, ahead of | ||
| # runtime provider discovery, the curated table, and the bundled models.dev | ||
| # catalog (awf-config-spec 10.7.1). It applies to the threat-detection proxy as | ||
| # well as the agent's. Rates are per-token USD at 50% of Anthropic list. | ||
| # | ||
| # MAINTENANCE: entries are matched per model, and an unlisted model silently | ||
| # falls through to full list price rather than erroring. Add an entry when the | ||
| # engine model changes, and re-halve these when Anthropic list prices move. | ||
| # Do NOT collapse these to a bare `claude-opus-4` prefix: prefix matching would | ||
| # also capture opus-4-0/4-1, which list at 3x the 4-5+ rate. | ||
| models: | ||
|
khan-actions-bot marked this conversation as resolved.
|
||
| providers: | ||
| anthropic: | ||
| models: | ||
| # Current engine model. | ||
| claude-opus-4-8: | ||
| cost: | ||
| input: "2.5e-06" | ||
| output: "1.25e-05" | ||
| cache_read: "2.5e-07" | ||
| cache_write: "3.125e-06" | ||
| # Engine models a consumer may select via an `engine:` override. | ||
| claude-opus-5: | ||
| cost: | ||
| input: "2.5e-06" | ||
| output: "1.25e-05" | ||
| cache_read: "2.5e-07" | ||
| cache_write: "3.125e-06" | ||
| claude-sonnet-5: | ||
|
khan-actions-bot marked this conversation as resolved.
|
||
| cost: | ||
| input: "1e-06" | ||
| output: "5e-06" | ||
| cache_read: "1e-07" | ||
| cache_write: "1.25e-06" | ||
| claude-haiku-4-5: | ||
| cost: | ||
| input: "5e-07" | ||
| output: "2.5e-06" | ||
| cache_read: "5e-08" | ||
| cache_write: "6.25e-07" | ||
| claude-fable-5: | ||
| cost: | ||
| input: "5e-06" | ||
| output: "2.5e-05" | ||
| cache_read: "5e-07" | ||
| cache_write: "6.25e-06" | ||
| # The pattern-triage sub-agent's pin (the cheap first pass; see its | ||
| # `model:` line below). Not an engine model, but dispatched inside | ||
| # the sandboxed agent step and metered by the same api-proxy. | ||
| claude-sonnet-4-6: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. note (non-blocking): Sixth overlay model (claude-sonnet-4-6) is not in the description's five-row table. The description's What section says 'the five pinned models' and its table lists five, but the diff adds claude-sonnet-4-6 as a sixth entry. I confirmed it is genuinely dispatched (review.md:1502, the pattern-triage pin) and is covered by the changeset's broader 'every model the reviewer can run' wording, so this is a description/implementation mismatch rather than a defect — worth aligning the table with the six entries actually shipped. |
||
| cost: | ||
| input: "1.5e-06" | ||
| output: "7.5e-06" | ||
| cache_read: "1.5e-07" | ||
| cache_write: "1.875e-06" | ||
|
|
||
| # Cost guardrails (AI credits; 1 credit = $0.01 of real spend, given the | ||
| # pricing overlay above). gh-aw >= v0.79 bakes in | ||
| # defaults of 1000/run ($10) and 5000/day ($50). Disable the daily ceiling | ||
| # (-1) so reviews are never skipped on a busy PR day; the per-run cap below | ||
| # still bounds the cost of any single review. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
suggestion (non-blocking): Whole change hinges on a manual, un-CI-enforced source→lock recompile that can silently revert. The diff edits the distributable source (review.md) but deliberately leaves review.lock.yml — the artifact that actually runs — out of sync, and the comment itself notes nothing in CI recompiles, so activation and its persistence rely entirely on humans remembering to recompile and verify. A CI guard that fails when review.md and its compiled lock diverge (or asserts
providersappears in both awf-config payloads once the AWF floor is met) would convert this silent-revert footgun into an enforced invariant.