fix(shadow-eval): bill the creator and enforce their budgets by tin-berri · Pull Request #44356 · BerriAI/litellm

tin-berri · 2026-10-03T07:34:18Z

TLDR

Problem this solves:

Admin evaluations charged sampled users and agents
Concurrent evaluations could exceed the creator's budgets

How it solves it:

Bill evaluation receipts exclusively to the initiating admin
Reserve creator budgets before paid provider calls
Preserve incurred costs and release unused reservations

Intentional product change: The evaluation form identifies the initiating admin as payer

User Flow

Before: the sampled user pays for an admin's evaluation, and the admin's limits do not stop it

As an admin, send POST https://litellm-domain/auto_router/shadow_eval/start selecting a user and 100% sampling
That user sends POST https://litellm-domain/v1/chat/completions
Usage includes evaluation charges on the sampled user, even when the initiating admin has exhausted their budget

After: the admin pays for evaluation work within their configured budgets

As an admin, start the same evaluation for the sampled user
That user sends the same Chat Completions request
Usage charges production to the sampled user and evaluation to the admin; exhausted admin budgets stop evaluation work

Changes

Evaluation work captures the creator separately from the sampled request's routing and access context. Database receipts, collector events and billing exports charge the creator, clearing sampled user, key, team, organization, project, agent and tag attribution

Each SDK attempt owns its reservation. Nested adapters share that ownership, while fallback attempts and cache hits get separate receipt state. Delayed callbacks retain the reservation and cost from their original attempt

Admission prices the selected deployment through the router and enforces its logical model group with the existing total-user budget engine. Per-model reservations use unique expiring holds, separate from recorded spend. Admission reads spend and acquires a hold atomically. Settlement records actual cost and removes that hold in the same operation; cancellation finishes and refunds any interrupted acquisition. Ordinary user-model admission includes outstanding evaluation holds

Missing or unreadable named creators withhold evaluation work. Invalid optional end-user attribution cannot bypass authenticated model budgets. There are no workflow, dependency, migration or lint-budget changes

Linear ticket

Resolves LIT-9199

Pre-Submission checklist

Meaningful billing, budget, concurrency, cancellation and retry regressions
Focused tests covering changed behavior pass locally
Final repository checks pass
All required CI/CD checks pass on the current commit
Scope remains evaluation billing and its budget enforcement
Fresh Greptile confidence is at least 4/5

Validation

make check passes on the current commit. The combined SDK, budget, auth and cache run passes 949 tests. One unchanged auth subprocess stalls during Python shutdown; its exact probe and original assertions pass separately. The model-limiter suite passes 65 tests, and four integration cases exercise real Redis

Targeted mutations killed 13 budget regressions and seven SDK regressions. These cover routing prices, ownership, cache receipt isolation, fallback admission, concurrent settlement, lease expiry and cancellation

Real Redis verification used four separate processes and 50 competing admissions under a budget of 2.5 estimates. Two calls were admitted and 48 rejected: $0.056 remained in temporary holds, recorded spend stayed zero during execution, and settlement left $0.014 actual spend

Delayed Redis responses exercised repeated cancellation after total and model writes had committed. Both scopes returned to zero. Expired or released holds could neither remove another request's reservation nor be renewed. A Redis error reading recorded spend rejected admission and refunded its hold

Admission and settlement passed a forced concurrent-settlement race, 150 reads during 100 settlements, 56 namespace/hash-slot combinations and persistent-window expiry checks. Rejected oversized requests leave no hold

Four SDK transport cases verify paid classifier and embedding calls enforce both creator budgets. Origin metadata does not enable the separate flag used for costs intentionally aggregated into a parent request

Bugbot's repeated-failure settlement finding was reproduced against real Redis and found incorrect. Shallow reservation copies share entries with applied_adjustment, so reconciliation applies only the remaining difference. Wrapper release, two failure callbacks and eight concurrent duplicates preserved earlier and unrelated spend. A later higher cost added only its increase, and a later zero cost changed nothing

Screenshots / Proof of Fix

The same six cases ran against isolated PostgreSQL databases and a local proxy, with real paid Together DeepSeek-V4.1-Flash source/shadow calls and Moonshot Kimi-K3 judge calls. Keys below are placeholders. <source-agent> in the receipt output represents the generated agent UUID

Shared proxy configuration:

model_list:
- model_name: qa-source
  litellm_params:
    model: together_ai/deepseek-ai/DeepSeek-V4.1-Flash
    api_key: os.environ/TOGETHERAI_API_KEY
    max_tokens: 50
    reasoning_effort: none
- model_name: qa-judge
  litellm_params:
    model: moonshot/kimi-k3
    api_key: os.environ/MOONSHOT_API_KEY
    extra_body:
      thinking:
        type: disabled
- model_name: qa-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      classifier_type: heuristic
      tiers:
        SIMPLE: qa-source
        MEDIUM: qa-source
        COMPLEX: qa-source
        REASONING: qa-source
- model_name: qa-priced
  litellm_params:
    model: openai/deepseek-ai/DeepSeek-V4.1-Flash
    api_key: os.environ/TOGETHERAI_API_KEY
    api_base: https://api.together.xyz/v1
    max_tokens: 50
    extra_body:
      reasoning_effort: none
    input_cost_per_token: 1.0e-06
    output_cost_per_token: 2.0e-06
  model_info:
    mode: chat
    max_input_tokens: 1048576
    max_output_tokens: 50
- model_name: qa-priced-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      classifier_type: heuristic
      tiers:
        SIMPLE: qa-priced
        MEDIUM: qa-priced
        COMPLEX: qa-priced
        REASONING: qa-priced
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 1
  allow_requests_on_db_unavailable: false
litellm_settings:
  drop_params: true
  telemetry: false
router_settings:
  num_retries: 0
  timeout: 60

Start the proxy on port 55183 with this configuration, provider credentials, and a fresh QA database for each commit. Create a separate evaluation owner and an agent-bound source key, then enable 100% shadow sampling:

PROXY=http://127.0.0.1:55183
MASTER_KEY='<local QA master key>'
QA_DATABASE=shadow_eval_base  # shadow_eval_after for the final commit

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-source","qa-router","qa-judge"]}' > admin.json
CREATOR_KEY=$(jq -r .key admin.json)
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-source-user","user_role":"internal_user","auto_create_key":false}'
AGENT_ID=$(curl -fsS "$PROXY/v1/agents" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"agent_name":"qa-source-agent"}' | jq -r .agent_id)
jq -n --arg agent "$AGENT_ID" \
  '{user_id:"qa-source-user",agent_id:$agent,models:["qa-source","qa-router","qa-judge"],max_budget:1,key_alias:"qa-source-key"}' > source-key-request.json
SOURCE_KEY=$(curl -fsS "$PROXY/key/generate" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  --data-binary @source-key-request.json | jq -r .key)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-source-user"],"router_name":"qa-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'

The setup returned HTTP 200 for users/agent/key creation and HTTP 201 for evaluation creation. Run cases in the listed order. New evaluation jobs propagate through the existing active-job cache within about ten seconds. For replay, wait eleven seconds after creating a job before sending its first sample. After each source response, wait for the evaluation receipts, then inspect the new rows with:

docker exec litellm-44356-pg psql -U qa -d "$QA_DATABASE" -c \
  'SELECT "user", agent_id, spend, metadata::jsonb->>'"'"'internal_call_origin'"'"' AS origin FROM "LiteLLM_SpendLogs" ORDER BY "startTime";'
docker exec litellm-44356-pg psql -U qa -d "$QA_DATABASE" -c \
  'SELECT outcome, error, shadow_cost, judge_cost FROM "LiteLLM_ShadowEvalAttempt" ORDER BY created_at;'

Before (`ad8baba`)

Chat billing

Run:

curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","prompt_tokens":34,"completion_tokens":20}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000342
shadow_eval_router  | qa-source-user | <none>         | 0.0000342
shadow_eval_judge   | qa-source-user | <none>         | 0.001578

evaluation outcome: tie
shadow_cost: 3.42e-05; judge_cost: 0.001578

Creator total budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1e-06,"spend":1}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly READY."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"READY.","prompt_tokens":35,"completion_tokens":15}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000285
shadow_eval_router  | qa-source-user | <none>         | 0.0000333
shadow_eval_judge   | qa-source-user | <none>         | 0.001578

evaluation outcome: shadow
shadow_cost: 3.33e-05; judge_cost: 0.001578

Creator per-model budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{"qa-source":{"budget_limit":1e-06,"time_period":"1h"},"qa-router":{"budget_limit":1e-06,"time_period":"1h"},"qa-judge":{"budget_limit":1e-06,"time_period":"1h"}}}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly DONE."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"DONE","prompt_tokens":35,"completion_tokens":32}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000489
shadow_eval_router  | qa-source-user | <none>         | 0.0000537
shadow_eval_judge   | qa-source-user | <none>         | 0.001518

evaluation outcome: tie
shadow_cost: 5.37e-05; judge_cost: 0.001518

Responses billing

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{}}'
curl -fsS "$PROXY/v1/responses" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","input":"Reply exactly OK.","max_output_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":27}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000426

evaluation attempts: 0 after 75 seconds
job funnel: not_sampled=0, unjudgeable=1, shed=0, withheld=0

The matching active job classified this bridged Responses sample as unjudgeable. This case verifies the paid source request only; evaluation billing is unverified for this bridge

Messages billing

Run:

curl -fsS "$PROXY/v1/messages" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":16}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000294
shadow_eval_router  | qa-source-user | <none>         | 0.000033
shadow_eval_judge   | qa-source-user | <none>         | 0.0008268

evaluation outcome: tie
shadow_cost: 3.3e-05; judge_cost: 0.0008267999999999999

Configured deployment pricing

Run:

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-priced","qa-priced-router","qa-judge"]}' > priced-admin.json
PRICED_CREATOR_KEY=$(jq -r .key priced-admin.json)
curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","model_max_budget":{"qa-priced":{"budget_limit":0.01,"time_period":"1h"}}}'
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-source-user","user_role":"internal_user","auto_create_key":false}'
jq -n --arg agent "$AGENT_ID" '{user_id:"qa-priced-source-user",agent_id:$agent,models:["qa-source","qa-priced","qa-priced-router","qa-judge"],max_budget:1}' > priced-key-request.json
curl -fsS "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' --data-binary @priced-key-request.json > priced-key.json
PRICED_SOURCE_KEY=$(jq -r .key priced-key.json)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $PRICED_CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-priced-source-user"],"router_name":"qa-priced-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'
sleep 11  # allow the existing active-job cache to refresh
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $PRICED_SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly PRICE."}],"max_tokens":50}'

Observed response and new receipts:
```
HTTP 200
{"content":"PRICE","prompt_tokens":35,"completion_tokens":37}

origin              | billed user    | agent          | USD
source              | qa-priced-source-user | <source-agent> | 0.0000549
shadow_eval_router  | qa-priced-source-user | <none>         | 0.000015
shadow_eval_judge   | qa-priced-source-user | <none>         | 0.001524

evaluation outcome: tie
shadow_cost: 1.5e-05; judge_cost: 0.001524
```
The shadow deployment uses the real Together endpoint through its OpenAI-compatible API. Its provider-qualified model has no built-in price entry; the proxy config supplies input/output prices of $1/$2 per million tokens. The evaluation succeeds under the creator’s $1 total limit. The configured $0.01 hourly qa-priced cap is not exercised because this routed request uses the qa-priced-router budget identity; focused tests separately cover configured per-model pricing

After (`0e460ab`)

Chat billing

Run:

curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","prompt_tokens":34,"completion_tokens":19}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.000033
shadow_eval_router  | qa-eval-admin  | <none>         | 0.0000246
shadow_eval_judge   | qa-eval-admin  | <none>         | 0.0008868

evaluation outcome: tie
shadow_cost: 2.46e-05; judge_cost: 0.0008868

Creator total budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1e-06,"spend":1}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly READY."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"READY","prompt_tokens":35,"completion_tokens":14}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000273

evaluation outcome: error
error: shadow router call failed: BudgetExceededError at evaluation_budget.py:355: Budget has been exceeded! Current cost: 1.0, Max budget: 1e-06
shadow_cost: 0; judge_cost: 0

Creator per-model budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{"qa-source":{"budget_limit":1e-06,"time_period":"1h"},"qa-router":{"budget_limit":1e-06,"time_period":"1h"},"qa-judge":{"budget_limit":1e-06,"time_period":"1h"}}}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly DONE."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"DONE","prompt_tokens":35,"completion_tokens":30}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000465

evaluation outcome: error
error: shadow router call failed: BudgetExceededError at evaluation_budget.py:405: Budget has been exceeded! Current cost: 0.0, Max budget: 1e-06
shadow_cost: 0; judge_cost: 0

Responses billing

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{}}'
curl -fsS "$PROXY/v1/responses" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","input":"Reply exactly OK.","max_output_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":20}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000342

evaluation attempts: 0 after 75 seconds
job funnel: not_sampled=0, unjudgeable=1, shed=0, withheld=0

The matching active job classified this bridged Responses sample as unjudgeable. This case verifies the paid source request only; evaluation billing is unverified for this bridge

Messages billing

Run:

curl -fsS "$PROXY/v1/messages" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":12}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000246
shadow_eval_router  | qa-eval-admin  | <none>         | 0.0000282
shadow_eval_judge   | qa-eval-admin  | <none>         | 0.0007818

evaluation outcome: tie
shadow_cost: 2.82e-05; judge_cost: 0.0007818

Configured deployment pricing

Run:

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-priced","qa-priced-router","qa-judge"]}' > priced-admin.json
PRICED_CREATOR_KEY=$(jq -r .key priced-admin.json)
curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","model_max_budget":{"qa-priced":{"budget_limit":0.01,"time_period":"1h"}}}'
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-source-user","user_role":"internal_user","auto_create_key":false}'
jq -n --arg agent "$AGENT_ID" '{user_id:"qa-priced-source-user",agent_id:$agent,models:["qa-source","qa-priced","qa-priced-router","qa-judge"],max_budget:1}' > priced-key-request.json
curl -fsS "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' --data-binary @priced-key-request.json > priced-key.json
PRICED_SOURCE_KEY=$(jq -r .key priced-key.json)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $PRICED_CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-priced-source-user"],"router_name":"qa-priced-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'
sleep 11  # allow the existing active-job cache to refresh
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $PRICED_SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly PRICE."}],"max_tokens":50}'

Observed response and new receipts:
```
HTTP 200
{"content":"PRICE","prompt_tokens":35,"completion_tokens":29}

origin              | billed user    | agent          | USD
source              | qa-priced-source-user | <source-agent> | 0.0000453
shadow_eval_router  | qa-priced-admin | <none>         | 0.000015
shadow_eval_judge   | qa-priced-admin | <none>         | 0.001584

evaluation outcome: tie
shadow_cost: 1.5e-05; judge_cost: 0.001584
```
The shadow deployment uses the real Together endpoint through its OpenAI-compatible API. Its provider-qualified model has no built-in price entry; the proxy config supplies input/output prices of $1/$2 per million tokens. The evaluation succeeds under the creator’s $1 total limit. The configured $0.01 hourly qa-priced cap is not exercised because this routed request uses the qa-priced-router budget identity; focused tests separately cover configured per-model pricing

Paid nested evaluation-owned Messages/compaction remains unverified: native compaction requires Anthropic, whose account rejected the earlier probe for exhausted credits. The Messages case above tests a public Messages source request; focused provider-transport regressions separately exercise evaluation-owned nested calls

Type

Bug Fix

Caveats

Medium

Responses bridge remains unjudgeable on base and head
Ordinary concurrent requests retain existing model-budget enforcement behavior

Low

Paid nested Messages compaction unverified; Anthropic credits unavailable
Nested Messages success, failure and cancellation have transport regressions
UI billing copy lacks screenshots; isolated browser unavailable

Final Attestation

Regressions cover ownership, pricing, cancellation, retries and budget resets

Verification status

Current commit: 0e460ab47af3c2a0c6599277b0ab8e66138ac92b. All ten required CI checks pass. Greptile reports 5/5 and Veria passed. Bugbot's fresh recheck passed with no issues after reviewing the repeated-settlement evidence above

Codecov's final commit report and an independent merge of CI artifacts agree on 95.30% changed-line coverage: 426 of 447 executable lines. SDK wrappers, internal billing metadata and cache changes are fully covered; the model limiter covers 51/52 changed lines and evaluation budgets cover 214/226. The GitHub coverage check still shows an earlier partial-upload result

Broader CI is still completing. The push-triggered Rust job failed an unchanged ClickHouse row-limit test, while the parallel PR Rust job passed. A rerun request was rejected while its workflow was still active

The CodeQL aggregate flags existing API-key hashing in alert 9134. The merge-base callback already passed the same API-key getter to the same spend writer and SHA-256 token helper. This PR removes the token for evaluation receipts and preserves the existing path for ordinary requests; it adds no password-hashing path

greptile-apps · 2026-10-03T07:38:26Z

Confidence Score: 5/5

[Critical risk] Redirects shadow evaluation billing from requesters to job creators.

The PR appears safe to merge based on this review; no new actionable issue was established.

Findings

Deployment IDs lose configured pricing ▶

Summary

The PR attributes shadow-evaluation spend to its initiating admin and adds creator-budget reservations around paid evaluation calls. The latest changes make model-budget admission and settlement atomic, use selected-deployment pricing, and refine nested SDK attempt ownership.

_{Reviews (5) · Last reviewed commit: "fix(shadow-eval): bill the creator and e..."}

codspeed · 2026-10-03T07:44:29Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_shadow_eval_admin_billing (0e460ab) with main (af36e5c)¹}

No successful run was found on main (9e9c29f) during the generation of this report, so af36e5c was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

veria-ai · 2026-10-03T07:47:12Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

codecov · 2026-10-03T07:48:39Z

Codecov Report

❌ Patch coverage is 95.74944% with 19 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/spend_tracking/evaluation_budget.py	94.69%	12 Missing ⚠️
litellm/litellm_core_utils/litellm_logging.py	78.94%	4 Missing ⚠️
litellm/integrations/shadow_eval_logger.py	96.00%	1 Missing ⚠️
litellm/proxy/hooks/model_max_budget_limiter.py	98.07%	1 Missing ⚠️
litellm/proxy/hooks/proxy_track_cost_callback.py	85.71%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

tin-berri · 2026-10-03T18:10:13Z

@greptileai Please review the current commit, including creator billing isolation, concurrent budget admission, and the fixes to your previous findings

tin-berri · 2026-10-03T18:10:14Z

bugbot run

Please review commit 731213e for billing isolation, creator budget enforcement, cancellation handling, and regressions after rebasing

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T18:45:17Z

@greptileai Please review f974ac8: budget ownership, configured pricing, cancellation, and expiring model holds were rebuilt with focused regressions

tin-berri · 2026-10-03T18:45:18Z

bugbot run Please review f974ac8 for nested-call ownership, cancellation cleanup, configured pricing, and per-request budget hold settlement

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T19:06:14Z

@greptileai Please review this revision for atomic creator budgets, empty fallback handling, cancellation safety, and configured deployment pricing

tin-berri · 2026-10-03T19:06:16Z

bugbot run Please review the latest revision, especially atomic budget settlement, budget-window expiry, empty fallback lists, and nested SDK calls

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T19:13:09Z

@greptileai Please review the latest commit, including selected-deployment pricing through auto-routing, atomic creator budgets, and nested SDK ownership

tin-berri · 2026-10-03T19:13:10Z

bugbot run Please review the latest commit, especially selected-deployment pricing through auto-routing, atomic settlement, and empty fallback handling

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T19:23:31Z

bugbot run Please recheck failure settlement against the Redis evidence in the description. Shared entries retain applied_adjustment, so callbacks apply only deltas

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0e460ab. Configure here.}

tin-berri requested a review from a team October 3, 2026 07:34

tin-berri had a problem deploying to e2e-changed October 3, 2026 07:34 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/internal_call_metadata.py Outdated

Comment thread litellm/proxy/auth/user_api_key_auth.py

This was referenced Oct 3, 2026

fix(usage): preserve daily spend by public model #44357

Open

fix(usage): preserve daily spend by public model #42587

Closed

veria-ai Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/hooks/model_max_budget_limiter.py

tin-berri had a problem deploying to e2e-changed October 3, 2026 08:13 — with GitHub Actions Error

tin-berri changed the title ~~fix(shadow-eval): bill evaluation usage to the initiating admin~~ fix(shadow-eval): bill the creator and enforce their budgets Oct 3, 2026

github-advanced-security Bot found potential problems Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py Fixed

tin-berri had a problem deploying to e2e-changed October 3, 2026 08:22 — with GitHub Actions Error

tin-berri had a problem deploying to e2e-changed October 3, 2026 08:24 — with GitHub Actions Error

tin-berri force-pushed the litellm_shadow_eval_admin_billing branch from 4f435d7 to 731213e Compare October 3, 2026 17:43

tin-berri had a problem deploying to e2e-changed October 3, 2026 17:44 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py Outdated

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py Outdated

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py Outdated

tin-berri force-pushed the litellm_shadow_eval_admin_billing branch from 731213e to f974ac8 Compare October 3, 2026 18:44

tin-berri had a problem deploying to e2e-changed October 3, 2026 18:44 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py Outdated

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py

Comment thread litellm/utils.py

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py

tin-berri force-pushed the litellm_shadow_eval_admin_billing branch from f974ac8 to a5922fa Compare October 3, 2026 19:05

tin-berri had a problem deploying to e2e-changed October 3, 2026 19:05 — with GitHub Actions Error

cursor Bot reviewed Oct 3, 2026

View reviewed changes

fix(shadow-eval): bill the creator and enforce their budgets

0e460ab

tin-berri force-pushed the litellm_shadow_eval_admin_billing branch from a5922fa to 0e460ab Compare October 3, 2026 19:12

tin-berri requested a deployment to e2e-changed October 3, 2026 19:13 — with GitHub Actions Waiting

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py Outdated

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py

cursor Bot reviewed Oct 3, 2026

View reviewed changes

This was referenced Oct 3, 2026

fix(shadow-eval): isolate creator billing and budget settlement #44427

Open

fix(shadow-eval): run evaluation calls as the job creator #44464

Open

moyai-devin-berriai Bot mentioned this pull request Oct 5, 2026

fix(shadow-eval): validate dependencies and stop repeated failures #44652

Open

3 of 5 tasks

Conversation

tin-berri commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Changes

Linear ticket

Pre-Submission checklist

Validation

Screenshots / Proof of Fix

Before (ad8baba)

Chat billing

Creator total budget

Creator per-model budget

Responses billing

Messages billing

Configured deployment pricing

After (0e460ab)

Chat billing

Creator total budget

Creator per-model budget

Responses billing

Messages billing

Configured deployment pricing

Type

Caveats

Medium

Low

Final Attestation

Verification status

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Findings

Summary

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

veria-ai Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

codecov Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tin-berri commented Oct 3, 2026

tin-berri commented Oct 3, 2026 •

edited

Loading

Before (`ad8baba`)

After (`0e460ab`)

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

codspeed Bot commented Oct 3, 2026 •

edited

Loading

veria-ai Bot commented Oct 3, 2026 •

edited

Loading

codecov Bot commented Oct 3, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading