feat(shadow_eval)!: gate the per-key budget on dollar spend instead of turns

[tin-berri]

TLDR

Problem this solves:

• Shadow eval budgets are turn counts, but the thing operators care about is dollars
• A turn's cost varies wildly by model and prompt, so max_turns bounds spend only loosely
• The job's own judge_spend read $0.00 whenever name-based pricing missed

How it solves it:

• start_shadow_eval takes max_budget (USD per key) instead of max_turns
• Each attempt records the shadow arm's and judge's billed cost on its row
• Sampler, sweep, stop guard, and derived status all gate on recorded spend
• Costs come from the billed response_cost figure, so deployment pricing is respected
• Billed cost is recorded on every exit, even empty replies and pipeline errors
• Admission re-checks the proxy's cross-pod spend counter, the same owner key budgets use
• A request still sending max_turns gets a 422 naming the replacement
• A 10,000-sample internal valve terminates zero-cost error loops
• Pre-migration jobs keep their configured turn budget (max_budget stays NULL)

User Flow

Before: an operator wants to cap a shadow eval's own overhead at a dollar amount, and there is no way to say that

1. They send POST http://localhost:4000/auto_router/shadow_eval/start with {"router_name": "my-router", "api_key_ids": ["<hash>"], "judge_model": "judge", "shadow_percentage": 100, "max_budget": 0.01}
2. The job is created anyway: the max_budget field is silently ignored and the response shows only "max_turns": 200
3. Their key's traffic gets sampled until 200 turns are judged, whatever that costs; the detail response shows "judge_spend": 0.0 even though the spend logs bill real dollars for every judge call

After: the same request caps the eval at one cent per key

1. They send the same POST http://localhost:4000/auto_router/shadow_eval/start with "max_budget": 0.01
2. The response shows "max_budget": 0.01 on their key, and sampling stops as soon as the key's recorded shadow plus judge spend reaches it, checked against a cross-pod counter
3. The detail response shows real dollar figures per key, "spend": 0.012327, "status": "completed", matching what was billed

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Rig: local proxy from this branch on :4321 with real Postgres (all migrations plus this PR's). Every deployment (cheap/strong/judge, plus auto-router my-router) points at claude-haiku-4-5-20251001 through the litellm sandbox gateway, so every shadow and judge call is a real provider call priced by the real cost map and costing real money. No stubs and no mock responses anywhere in this run

Before (3f2e0ba)

A dollar budget cannot be expressed and judge_spend lies

1. curl -s http://127.0.0.1:4321/auto_router/shadow_eval/start -H "Authorization: Bearer sk-1234" -d '{"router_name": "my-router", "api_key_ids": ["<hash>"], "judge_model": "judge", "shadow_percentage": 100, "max_turns": 2, "max_budget": 0.001}' creates the job; max_budget is silently ignored, the response carries only "max_turns": 2
2. 3 real chat turns through the shadowed key ({"model": "cheap", "max_tokens": 30, "messages": [{"role": "user", "content": "real before turn N: name one prime number"}]})
3. Detail reads "attempt_count": 2, "judge_spend": 0.0, "status": "completed": the turn count gated it, and judge_spend reads zero because name-based pricing missed for the openai/-prefixed deployment
4. Meanwhile LiteLLM_SpendLogs shows the judge legs billing real money: 2 rows tagged shadow_eval_judge totaling $0.001160, uncappable by any knob

After (f5d959a; behavior captured at c033429, since unchanged on these paths)

The dollar budget bounds the job with real per-attempt costs

1. curl -s http://127.0.0.1:4321/auto_router/shadow_eval/start -H "Authorization: Bearer sk-1234" -d '{"router_name": "my-router", "api_key_ids": ["<hash>"], "judge_model": "judge", "shadow_percentage": 100, "max_budget": 0.01}' returns "max_budget": 0.01, "max_turns": 10000, "status": "running"
2. 8 real chat turns through the shadowed key ({"model": "cheap", "max_tokens": 500, "messages": [{"role": "user", "content": "real after turn N: write a 250 word overview of a different chemical element"}]})
3. Exactly 4 are sampled and the detail reads "attempt_count": 4, "spend": 0.012327, "judge_spend": 0.005002, "status": "completed": sampling stopped the moment recorded spend crossed the cap, with the 10,000 turn valve untouched
4. SELECT outcome, shadow_cost, judge_cost FROM "LiteLLM_ShadowEvalAttempt" ... shows every row carrying real billed costs: shadow $0.001785 to $0.001870, judge $0.001232 to $0.001266
5. curl -s -X POST .../auto_router/shadow_eval/<job>/stop answers 400 "Job <job> is already completed"

A caller still sending the retired max_turns is told, not silently defaulted

1. curl -s http://127.0.0.1:4321/auto_router/shadow_eval/start ... -d '{"router_name": "my-router", "api_key_ids": ["<hash>"], "judge_model": "judge", "shadow_percentage": 100, "max_turns": 200}'
2. Answers 422 with "max_turns was replaced by max_budget, the per-key USD cap on the eval's own spend"

A pre-migration job keeps its turn budget

1. Create a job, then reshape its row into the legacy form with psql: UPDATE "LiteLLM_ShadowEvalJob" SET max_budget = NULL, max_turns = 2 WHERE group_id = '<job>'
2. 3 real chat turns through the shadowed key
3. Detail reads "attempt_count": 2, "max_budget": null, "spend": 0.00114, "status": "completed": the turn budget gated it, the recorded spend is display only

Type

🆕 New Feature

Caveats (if any)

• Breaking: start_shadow_eval replaces max_turns with max_budget
• Overshoot is bounded by samples already in flight when the cap is crossed
• During a rolling deploy old pods gate new jobs by the valve only
• Attempt rows written by old pods mid-deploy price as judge-only
• Rebased over merged feat(ui): multi-key shadow eval picker and per-key breakdown #37389; its per-key Budget used column now shows dollars

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Breaking API change plus new spend-budget gating that writes Redis spend counters and attempt costs. Mispricing or counter failure can overshoot or skip samples, but it reuses existing budget primitives and fail-closed reads.

Overview
Breaking: starting a shadow eval now takes per-key max_budget (USD, default $10) instead of max_turns. Callers still sending max_turns get a 422. New jobs write a 10k-turn valve only to stop zero-cost error loops; pre-migration jobs keep max_budget null and stay turn-gated.

Each attempt now stores shadow_cost plus judge_cost, priced from the billed response_cost stamp (price-map fallback). Sampling, sweep, stop, and derived status all complete a key when recorded spend hits the cap. Admission re-checks the shared spend:shadow_eval:{job_id} counter so pods stop before the next provider call; unverifiable spend skips the sample.

The dashboard start form and per-key “budget used” column show dollars ($spend / $max_budget), with a turns fallback for legacy jobs.

^{Reviewed by Cursor Bugbot for commit f5d959a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's why:

What's done well:

• Fixes the root cause correctly. The public price map returns $0 for deployment-priced models. Reading _hidden_params.response_cost (what the spend pipeline actually bills) is the right fix — the test test_call_cost_prefers_the_billed_figure_over_the_public_price_map pins that contract cleanly.

• Backward compatibility is correct. The max_budget IS NULL branch throughout the sampler, stop guard SQL, and _is_running check means pre-migration jobs gate on max_turns alone, never on spend. That's the right invariant and it's tested (test_legacy_jobs_without_a_spend_budget_sample_on_turns_alone).

• Error-loop valve is well-reasoned. Zero-cost failure loops (shadow arm bails before billing) can't consume max_budget, so the 10,000-turn ceiling terminates them. The constant is named and documented.

• Tests cover the generation-boundary edge case. test_completed_pipelines_hold_spend_budget_within_a_cache_generation is the hardest correctness property here — that a recorded cost closes the pod-local budget even before the cache refills — and it passes.

• Migration is clean and additive (DOUBLE PRECISION nullable columns, no destructive changes).

What holds it to 4/5:

• Silent breaking API change. Callers sending max_turns in their POST body will have it silently ignored by Pydantic (unknown fields) and get max_budget=10.0 instead. There's no deprecation shim, no warning, and no error — the old parameter just vanishes. A 400 with a clear message or at least model_config = ConfigDict(extra='forbid') on StartShadowEvalRequest would make this observable rather than silent.

• The _IS_COMPLETED_SQL stop guard fires two correlated subqueries scanning LiteLLM_ShadowEvalAttempt independently (one COUNT, one SUM) per leg per check. At scale a single CTE or one aggregated subquery would be cleaner, though this is a low-throughput path so it's unlikely to be a practical issue.

• Default max_budget=10.0 means a 100-key job can spend up to ~$1,000 before operators realize it. The old default (200 turns) was more predictable. A lower default or a per-job ceiling in the validator would help, but this is a judgment call.

[greptile-apps]

Greptile Summary

This PR replaces shadow-evaluation turn budgets with per-key dollar budgets and records shadow plus judge costs on each attempt

• Adds nullable job budgets and persisted shadow costs across all Prisma schemas and the migration
• Gates sampling, lifecycle status, sweeping, and stopping on recorded spend while preserving legacy turn-gated jobs
• Updates the management API, generated types, dashboard form, spend display, and focused tests

Confidence Score: 3/5

The PR should not merge until billed costs remain accounted for across failed persistence and unjudgeable shadow responses

The new budget can permanently undercount real evaluation spend, allowing jobs to continue billing after their configured dollar limit

Files Needing Attention: litellm/integrations/shadow_eval_logger.py

Important Files Changed

Filename	Overview
litellm/integrations/shadow_eval_logger.py	Adds spend-based admission and billed-cost recording, but loses costs after persistence failures and on successful empty shadow responses
litellm/proxy/management_endpoints/auto_router_endpoints.py	Updates lifecycle SQL and response aggregation to use persisted shadow plus judge spend while retaining legacy turn behavior
litellm/types/management_endpoints/auto_router_endpoints.py	Replaces the start request's turn limit with a validated dollar budget and exposes per-key spend
litellm-proxy-extras/litellm_proxy_extras/migrations/20260819000000_shadow_eval_max_budget/migration.sql	Additively introduces nullable max_budget and non-null shadow_cost while preserving legacy rows
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Replaces the turn-budget input and display with per-key dollar-budget and total-spend UI

Comments Outside Diff (1)

1. litellm/integrations/shadow_eval_logger.py, line 831-832 (link)

   Empty Responses Lose Billed Cost

   When a billed shadow response has no judgeable text, this branch records zero cost, allowing subsequent sampling beyond max_budget.

   Knowledge Base Used: Auto-router Shadow Evaluations

_{Reviews (1): Last reviewed commit: "feat(shadow_eval)!: gate the per-key bud..." | Re-trigger Greptile}

[greptile-apps]

Cache Refill Drops Failed Writes

When an attempt write fails, refill clears its local cost and rebuilds spend from persisted rows, allowing billing beyond max_budget.

Knowledge Base Used: Auto-router Shadow Evaluations

[tin-berri]

Intentional: spend derives from persisted rows, the file's core invariant. A failed write loses its sample and spend alike, exactly as the turn budget did; the valve bounds the residue

[greptile-apps]

Acknowledged — deriving spend only from persisted rows is a clean invariant, and failed-write residue being bounded by the valve is consistent with how the turn budget handled the same case. Makes sense.

[tin-berri]

Revisited in c033429: attempt costs now advance the proxy's cross-pod spend counter, so a failed row write can no longer reopen the gate

[veria-ai]

Medium: Spend budget is not reserved before dispatch

This check only includes completed attempts. A key holder can send a burst of sampled requests while every pod still sees the same spend; each pod can launch up to 16 shadow-and-judge pipelines concurrently and independently consume toward the full cap before _job_costs or the database is updated. Since forwarded generation parameters include user-controlled output limits and new jobs have a 10,000-attempt valve, the user can drive eval charges well beyond max_budget. Use a shared atomic per-job reservation before dispatch and reconcile it with actual cost afterward, or otherwise enforce a cross-pod reservation that bounds the maximum in-flight spend.

[veria-ai]

PR overview

This pull request changes shadow evaluation so each key’s evaluation budget is enforced using dollar spend rather than turn count. It updates the shadow evaluation logger’s budget tracking and dispatch behavior.

One issue has been addressed, but spend enforcement remains vulnerable to concurrent oversubscription because costs are recorded only after evaluation attempts complete. A key holder can burst sampled requests across workers and trigger shadow-and-judge charges substantially beyond the configured maximum budget. Shared atomic reservation before dispatch is still needed to make the dollar cap reliable.

Open issues (1)

• Medium: Spend budget is not reserved before dispatch — litellm/integrations/shadow_eval_logger.py:654

Fixed/addressed: 1 · PR risk: 6/10

[codecov]

Codecov Report

❌ Patch coverage is 82.60870% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/shadow_eval_logger.py	77.55%	11 Missing ⚠️
litellm/proxy/proxy_server.py	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[tin-berri]

Addressed the silent rename in 82a488b: a request still sending max_turns now gets a 422 naming max_budget as the replacement, with a pinning test, and the live proof in the description shows the response. A blanket extra forbid would also reject harmless unknown fields the rest of the API tolerates, so the rejection is targeted at the one retired knob

On the stop guard's two correlated subqueries: each subquery is bounded by the 10,000-row valve per leg through the job_id index and the stop endpoint is a rare admin action, so the CTE consolidation buys nothing measurable here

On the default: the dollar exposure is now explicit per key in the request and the form, unlike a turn count whose dollar value was unbounded by construction, and a 100-key job requires naming 100 keys deliberately. Happy to lower it if you disagree

Same sha also fixes Bugbot's two findings: billed shadow cost is now recorded on every exit path, empty replies and pipeline errors included

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

4/5 — the core concerns from the original review are addressed and the implementation is clean. Here's the breakdown:

---

What's solid

Cost extraction is correct. _call_cost now correctly prefers _hidden_params.response_cost (the deployment-priced figure the router stamps) over the public price map fallback, which reads $0.00 for deployment-priced models. This is the root fix for the lying judge_spend figure.

Every exit path records shadow_cost. The two-try-block structure in _run_shadow_eval (pre-billing vs post-billing) is clean and correct: pipeline errors before the shadow call record shadow_cost=0, the empty-reply path records the actual billed cost, and all post-shadow exits (judge failure, pipeline error in the judge leg, success) carry both costs. The tests for empty replies and mid-pipeline errors confirm this.

_record_attempt updates _job_costs before the DB write. This is the right ordering — the pod-local accumulator advances even if the row write fails, so a transient DB blip can never reopen the spend gate.

_job_costs is symmetric with _job_starts. Both are cleared on cache refill, both hold per-generation pod-local deltas, and both gate admission the same way. The legacy-job test (max_budget=None with spend=999.0) correctly confirms that a NULL budget is never a gate.

422 on max_turns is targeted and correct. The model_validator(mode="before") fires before Pydantic drops unknown fields, so the retired knob is caught rather than silently ignored. The pinning test confirms this.

Migration is safe. max_budget DOUBLE PRECISION (nullable), shadow_cost DOUBLE PRECISION NOT NULL DEFAULT 0 — both additive, no UPDATE or DROP, and the test asserting exactly this structure is a good safeguard.

Test coverage is thorough. The new tests cover: spend-gated skip paths, the pod-local cost accumulator closing the budget within a cache generation, legacy turn-gated jobs, empty-reply cost billing, post-shadow pipeline error cost billing, the 422 rejection, and the SQL shape assertions for both the sweep and stop guard.

---

What prevents a 5

Multi-pod overshoot can be significant at small budgets. _job_costs only reflects this pod's spend since the last cache fill (10 s TTL). With N active pods each potentially having one in-flight sample when the budget is nearly exhausted, actual spend can exceed max_budget by up to N × (shadow_cost + judge_cost). At $0.08/attempt with 4 pods against a $0.20 cap (the motivating example in the PR description), the worst-case overshoot is $0.32 — 160% of the cap. This is documented and accepted as a known caveat, but operators setting tight budgets should know their per-pod overshoot exposure.

Stub-backed proof only. Real-provider re-run is acknowledged as owed. The local stub uses a fixed deterministic cost, which is a good test rig, but deployment-pricing behavior at a real provider (the reason _hidden_params.response_cost was needed) isn't verified against a live endpoint yet.

The stop guard's correlated subqueries. Two SELECT ... FROM "LiteLLM_ShadowEvalAttempt" correlated subqueries per leg in the stop SQL (count < max_turns, sum < max_budget). The author's defense (rare admin action, 10K-row valve, job_id index) is reasonable, but on a job with hundreds of legs and accumulated attempt history, these two subqueries run once per leg row. A CTE or lateral join would be cleaner at scale, even if the current form is acceptable.

---

The implementation is correct, the migration is backward-compatible, and the test suite covers the right cases including the edge paths that existed as bugs before. The remaining items are documented caveats or polish rather than correctness issues.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 82a488b. Configure here.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadoweval_spend_budget (f5d959a) with litellm_internal_staging (d542c82)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (4873567) during the generation of this report, so d542c82 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

c033429 closes the multi-pod overshoot with the codebase's established owner: attempt costs advance the proxy's cross-pod spend counter (spend:shadow_eval:{leg}) via the same increment path key budgets use, and every pipeline re-checks get_current_spend before its shadow call, with the fill-time DB sum as the counter's authoritative floor. The pod-local accumulator is deleted. Overshoot is now bounded by samples already in flight when the cap crosses, and a two-pod test pins the shared-counter behavior

The proof in the description is also fully real now: every leg is claude-haiku-4-5 through the litellm sandbox gateway, real provider calls priced by the real cost map, before and after legs both. No stubs remain

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4.5/5

Here is what lands the score where it is.

---

What works well

• Root-cause fix is correct. The original judge_spend: 0.0 bug came from reading the public price map for deployment-priced models. _call_cost now prefers _hidden_params.response_cost, which the router client stamps from the deployment's own pricing. This is the right place to read it.

• Cross-pod counter reuses established infrastructure. _job_spend_from_counter / _add_job_spend_to_counter wrap get_current_spend / _increment_spend_counter_cache, the same path key and team budget gates use. No new coordination mechanism invented.

• Double-gate design is sound. The soft gate in async_log_success_event uses the cache-fill job.spend (stale by up to the TTL) as an early admission filter. The hard gate in _run_shadow_eval reads the live counter before any provider call. Overshoot is bounded by samples already in flight when the cap crosses, which is correctly documented and acceptable.

• Every exit path records its cost. Empty shadow replies, pipeline errors between shadow and judge, and judge failures all record shadow_cost before returning, so the counter never undercounts.

• Backward compatibility is clean. max_budget IS NULL → turn-only gating, no migration required for existing rows. The SQL status derivation, the sampler eligibility check, and budget_spent all condition on max_budget is not None.

• The 422 on max_turns is correct. A model_validator that rejects the field with a named error is better than silently ignoring it; the old behavior was the bug.

• Tests cover the two-pod scenario explicitly. test_a_sibling_pod_sees_spend_through_the_shared_counter pins the exact behavior the design claims, using a shared dict as the counter store. test_legacy_jobs_without_a_spend_budget_sample_on_turns_alone is also present.

---

What leaves 0.5 on the table

• Counter write is fire-and-forget. _add_job_spend_to_counter swallows all exceptions at debug level. When Redis is degraded, the counter never advances. The hard gate in _run_shadow_eval then falls back to the fallback_spend floor, which is the cache-fill DB sum — lagging by up to the TTL. Under a prolonged Redis outage this allows the overshoot to grow to (samples in flight) + (requests sampled in one TTL window). That is survivable but worth naming as a known failure mode, not just a debug log.

• The SQL stop guard and the sampler eligibility check read spend from different sources. _COMPLETED_JOBS_SQL sums judge_cost + shadow_cost from the DB; the sampler reads the Redis counter (with the DB sum as the floor). During a rolling deploy where old pods write shadow_cost = 0 (they don't have the column yet), the DB sum undercounts, the counter is the truth, and the SQL guard may not fire when it should. This is the mid-deploy window the description acknowledges, but the interaction between the two spend sources under that window is subtle enough to warrant a comment in the SQL block.

Neither of these is a blocker; both are bounded failure modes with documented limits. The implementation is correct, the tests are substantive, and the real-call proof in the description closes the loop.

[tin-berri]

Rebased onto current staging: the lint job needs .github/actions/detect-changes, which my base predated. Zero diff change. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c033429. Configure here.}

[tin-berri]

8c19cbf aligns the counter wrappers with the budget owner's failure semantics: a raising read (fail-closed enforcement) now skips the sample, increment failures log at warning with the outage mode named, and the SQL block documents the two spend sources. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

The three claims about commit 8c19cbf9db:

1. Raising read → fail-closed (skip the sample)
2. Increment failures log at warning with outage mode named
3. SQL block documents both spend sources

Looking at the diff, all three are present:

Raising read skips the sample (_run_shadow_eval):

try:
    spend = await self._read_job_spend(...)
except Exception as e:  # unverifiable budget: skip the sample rather than spend on it
    verbose_logger.warning("shadow_eval: budget unverifiable for %s, sample skipped: %s", job.id, e)
    return

Correct fail-closed semantics.

Increment failure logs at warning (_add_job_spend_to_counter):

except Exception as e:
    verbose_logger.warning("shadow_eval: spend counter increment failed for %s: %s", counter_key, e)

The docstring names the degradation mode ("Redis outage freezes the counter, gate degrades to fill floor, overshoot grows to in-flight plus one TTL"), but the warning message itself doesn't inline that description — an operator reading logs won't immediately know what the system's behavior is under that failure.

SQL block documents two spend sources — sum={"judge_cost": True, "shadow_cost": True} is clear.

The two-try restructuring in _run_shadow_eval is the most substantive improvement: pre-billing exceptions are caught by the outer block (cost=0, row written if prisma available), and post-shadow exceptions are caught by the inner block carrying shadow_cost. The guard if judge_cost + shadow_cost > 0 prevents zero-cost failures from advancing the counter, which is how the 10,000-turn valve does its job without noise. That's clean.

One real gap: the warning for counter increment failures doesn't name the degradation mode inline. "spend counter increment failed" tells you something broke; it doesn't tell the operator that the gate is now falling back to the DB fill floor and overshoot will grow to in-flight plus one TTL. For an on-call engineer, that context matters. It belongs in the log line, not just the docstring.

Score: 4/5. The semantics are correct — fail-closed read, fire-and-forget write, cost recorded on every billed exit — and the injectable reader/writer pair makes the wrappers properly testable. The only gap is the warning message not carrying the degradation mode inline.

[tin-berri]

On the two 4/5 notes: LiteLLM_ShadowEvalAttempt has carried an index on (job_id) since its first migration, so the SUM subqueries are index-served and valve-bounded. The zero-cost interaction is documented where SHADOW_EVAL_TURN_VALVE is defined; that valve exists precisely because zero-cost attempts cannot consume spend budget

[tin-berri]

b8bdcb5 fixes Bugbot's HiddenParams finding and the lint budget breach (reportPrivateUsage suppression with reason). The image-scan red is base-image busybox/openssl CVEs with upstream fixes, unrelated to this diff and not a required check. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's what's working well and the one thing holding it back from 5:

Why it's strong:

The dual-gate admission design is correct — the eligible filter uses the cached DB floor as a fast-path precheck, and _run_shadow_eval re-checks the live cross-pod counter before any provider spend happens. Overshoot is bounded to samples already in flight, which the comments explicitly acknowledge.

_call_cost prefers _hidden_params.response_cost over the public price map — exactly the right call, since the price map reads 0 for deployment-priced models (the original bug). The test test_call_cost_prefers_the_billed_figure_over_the_public_price_map pins this behavior.

Cost is recorded on every exit path — empty shadow reply, pipeline error between shadow and judge, judge failure — so the counter never undercounts and the gate can always close. The tests test_an_empty_shadow_reply_still_bills_its_cost and test_a_pipeline_error_after_the_shadow_call_keeps_its_billed_cost lock in these two previously untracked cases.

The migration is correctly additive (the test_max_budget_migration_is_additive_and_leaves_legacy_rows_null test checks the SQL directly for the absence of UPDATE/DROP), legacy jobs stay turn-gated via max_budget IS NULL, and the _reject_the_retired_turn_budget validator is the right choice over silent defaulting.

The cross-pod spend counter is the same one key budgets use, so counter keys and invalidation semantics can't drift from auth's.

The one structural concern:

_increment_spend_counter_cache is a private function imported with # pyright: ignore[reportPrivateUsage]. The comment gives a correct reason ("the counter owner exposes no public raw-key increment; forking one would split the semantics"), and the DI seam via job_spend_writer in the constructor means tests never touch it. But it's still a fragility — a rename or refactor of proxy_server._increment_spend_counter_cache silently breaks the gate without a type error. A thin public wrapper in the counter owner would close this. Minor, but it's the only place the design relies on something the type system can't enforce.

Everything else — sweep SQL, stop guard SQL, budget_spent property backward compat, the 10,000-turn valve — looks correct. The b8bdcb5 fixes to HiddenParams access and the lint suppression reason are in order.

[tin-berri]

On the private-import note: agreed it is the one unenforced seam. A public wrapper adds surface to the owner for no behavior change, so leaving it suppressed with the reason; a rename would surface as the warning-logged degradation plus a grep hit in the renamer's sweep

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b8bdcb5. Configure here.}

[tin-berri]

46e3987 adds the public increment_spend_counter wrapper in the counter owner (suppression gone) and rebases over merged #37389, converting its per-key Budget used column to dollars with a turn fallback for legacy rows. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 5/5

Here's the full picture on commit 46e3987:

What the commit does:

1. increment_spend_counter public wrapper (proxy_server.py): A 7-line thin wrapper over _increment_spend_counter_cache. This is the right call — the previous version in the shadow_eval logger was importing the private function (or suppressing the import); this gives shadow eval a stable, documented public interface to the same primitive, so invalidation and read semantics can never silently drift.

2. UI rebase over feat(ui): multi-key shadow eval picker and per-key breakdown #37389 (ShadowEvalSection.tsx): The per-key Budget column now shows $spend / $max_budget for new jobs, and N / M turns with the "turns" suffix for legacy rows. totalBudget correctly returns null when any key has max_budget == null, which suppresses the budget denominator in the job summary line rather than showing a misleading mixed-currency sum. keySpent checks both gates in the right priority order.

Why it holds up:

• The wrapper's docstring accurately describes its purpose (non-entity budget domains sharing the entity counter primitive) — a future reader knows exactly why this exists and why it shouldn't be collapsed back to a private call.
• The UI null-propagation in totalBudget is correct: the reduce starts at 0 and shorts to null the moment any key is pre-migration, so the "of $X.XX" denominator is only shown when every key has a dollar budget.
• The rest of the PR's correctness (cost stamping, gate ordering, _call_cost preferring _hidden_params.response_cost, cost recorded on every exit path including empty shadow replies) was established in prior commits and holds through the rebase.
• Test coverage for the new behaviors is thorough: sibling-pod isolation, unverifiable-budget skip, empty-reply billing, pipeline-error cost capture, and legacy-job turn-only path.

The one acknowledged operational caveat — old pods mid-deploy price attempts as judge-only — is documented and bounded by deploy duration, and doesn't affect gate correctness post-deploy.

[tin-berri]

785a1a2 is test-only: it pins the owner contract refuting Bugbot's cold-counter finding (a reset counter degrades to the fill floor, never zero). No production change since your 5/5 sha. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

5/5 — no production change, and the new tests are doing real work.

The key test that refutes the cold-counter concern exercises the actual proxy_server.get_current_spend function directly (no mock of that function), covering both the case where the counter exists but is below the fallback and the case where the key is absent entirely:

# counter at 0.05, fallback 0.42 → max wins → 0.42 ✓
assert await proxy_server.get_current_spend(counter_key=counter_key, fallback_spend=0.42, max_budget=1.0) == 0.42
# key deleted (cold) → counter reads 0, fallback 0.42 → still 0.42 ✓
proxy_server.spend_counter_cache.in_memory_cache.delete_cache(key=counter_key)
assert await proxy_server.get_current_spend(counter_key=counter_key, fallback_spend=0.42, max_budget=1.0) == 0.42

That directly pins the owner contract: a Redis expiry degrades to the ≤10s-stale DB sum (the fallback_spend argument), never to zero, so an expired key cannot re-open a spent budget by a full cap. The concern is resolved.

The remaining new tests also fill real gaps: sibling-pod counter visibility, legacy max_budget=None turn-gating, empty-reply billing, and pipeline-error cost recording. All of them test observable behavior on the real production surface rather than mock internals.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 785a1a2. Configure here.}

[devin-ai-integration]

Live proxy QA with real Anthropic calls: $0.01 budget completed at $0.010313, 15 attempts, costs match SpendLogs.

[devin-ai-integration]

Ran a live E2E QA of the per-key spend budget on this branch at 785a1a2 (proxy :4000, real Anthropic claude-haiku-4-5 deployments behind auto-router my-router, real chat traffic through a virtual key).

Budget gating works. Job started via UI with max_budget: $0.01, 100% sampling:

• Spend grew 0.0 → 0.007064 (12 turns) → 0.010586 ≥ 0.01 at 18 attempts, status → completed; overshoot only $0.0006 (~6%).
• All 18 LiteLLM_ShadowEvalAttempt rows have nonzero costs (shadow_cost min 5.2e-05, judge_cost min 5.34e-04).
• 10 further turns after completion produced 0 new attempts — spend and attempt_count frozen at 18 / $0.010586.

422 on max_turns + legacy turn gating

• POST /auto_router/shadow_eval/start with max_turns: 200 → HTTP 422: "max_turns was replaced by max_budget, the per-key USD cap on the eval's own spend".
• Legacy path: job UPDATEd to max_budget=NULL, max_turns=2, 8 turns driven → stopped at exactly 2 attempts, completed, spend $0.0012.

Setup note

If prisma db push --skip-generate is used, the proxy 500s on start (FieldNotFoundError ... max_budget) until prisma generate is re-run — the Python client must be regenerated after this migration.

Screen recording of the full run: /home/ubuntu/screencasts/rec-774d3a10-3be6-440c-9404-94910e6c185f/rec-774d3a10-3be6-440c-9404-94910e6c185f-edited.mp4

[tin-berri]

f5d959a clears the two frontend-lint warnings: keySpent's boolean split into named conditions and the legacy fixture object named in the test. No behavior change. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

The two fixes are both correct and clean.

keySpent decomposition — the original one-liner mixed a != null guard on attempt_count against a max_turns comparison, while the new logic adds a second, orthogonal condition (spendBudgetReached). Extracting each into a named boolean makes the OR contract explicit and eliminates the lint rule (likely no-mixed-operators or a complexity threshold). The logic itself is correct: spend budget reached OR turn valve reached.

Named fixture — legacyTurnBudgetLeg as a const gives the reader a one-phrase summary of what the object is testing (a key that pre-dates max_budget, falling back to max_turns), which also silences the "object literal not assigned" or inline-object lint warning.

Both are no-behavior-change as stated.

For the commit specifically: 5/5. It does exactly what it says — two targeted lint fixes, no logic altered, and the resulting code is clearer than before.

For the PR overall, the feature is well-structured: the migration is additive and backward-compatible, the NULL guard for legacy jobs is consistent throughout (DB schema, API types, UI display, and tests all handle the max_budget: null path), and the 10 000-turn error-loop valve is a sensible safeguard against zero-cost runaway. The test coverage for the legacy turn-budget fallback path (3 / 500 turns display) is a nice touch. No material concerns.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f5d959a. Configure here.}