fix(shadow-eval): isolate creator billing and budget settlement by tin-berri · Pull Request #44427 · BerriAI/litellm

tin-berri · 2026-10-03T20:01:58Z

TLDR

Problem this solves:

Evaluation costs and limits incorrectly follow sampled application traffic
Duplicate receipts can undo or double-count settled spend

How it solves it:

Bill evaluation calls to the evaluation creator
Reserve creator budgets before each paid provider attempt
Give each attempt sole ownership of budget settlement

Intentional product change: evaluations consume their creator’s total and per-model budgets. Exhausted or unverifiable budgets withhold evaluation work while source requests continue. Missing creator accounts, including legacy admin jobs, also withhold work. Evaluation calls require complete responses and override deployment streaming defaults

This is a fresh implementation from main, replacing the approach in #44356. The original PR remains open for reference

User Flow

Before: an administrator evaluates a router, and sampled application users pay for the extra calls

The administrator starts an evaluation with POST http://localhost:55183/auto_router/shadow_eval/start
An application user sends POST http://localhost:55183/v1/chat/completions and receives HTTP 200
Usage receipts assign shadow and judge costs to that application user
The administrator exhausts their budget, but evaluation calls continue

After: the administrator pays for evaluations, and their budgets limit the additional calls

The administrator starts an evaluation with POST http://localhost:55183/auto_router/shadow_eval/start
An application user sends POST http://localhost:55183/v1/chat/completions and receives HTTP 200
Source usage stays with that user; shadow and judge costs belong to the administrator
The administrator exhausts their budget, and further evaluations are withheld

Design

A captured creator context leaves source routing and authorization metadata intact. Logging creates detached billing receipts for downstream consumers. Queued synchronous success and failure handlers capture their evaluation receipt and detach nested routing metadata before logger reuse. One evaluation attempt reserves and settles budgets, including nested calls, cached responses, cancellation and late receipts

Total budgets reuse the existing reservation engine. Per-model budgets retain their existing fixed windows and numeric spend keys, with atomic expiring holds for concurrent evaluation calls. Ordinary spend writers recognize centrally settled reservations and skip those counters. Settlement preserves the full model-budget window. Explicit empty fallback lists disable inherited fallbacks so each paid attempt receives its own admission and receipt. Messages uses the dispatcher’s provider selection to admit native requests directly and reserve bridged provider calls independently. Judge calls disable fallbacks in both dispatch paths

Pre-Submission checklist

I have added meaningful tests
The test files covering the change pass locally
My PR passes all required CI/CD checks
My PR solves one specific problem
Greptile confidence is at least 4/5

Validation

Commit 5c76148f717c8037c614b605deef6a679c168f4f adds 894 production lines and 933 test lines, with 106 deletions across 29 files in one commit. The production diff is 995 changed lines (+894/-101). No workflow, migration or lint-budget files change

make check passes for the final changes; the unchanged areas passed the full PR checks previously. All 1,161 affected tests pass: 422 Logging tests and 739 SDK, cache, shadow-evaluation and Messages tests. Logging passed again after the final formatting change. The wider 2,208-test corpus passed before the callback amendments. Changed executable Python coverage is 355/390 lines (91.0%), using fresh coverage for the changed modules and verified unchanged-source coverage elsewhere

Targeted mutations caught 36 of 38 faults (94.7%), with no setup or import errors. Both new regressions fail before their fixes and pass afterward. The dictionary-copy suggestion from review is caught by the new Messages regression: two provider calls produce four synchronous exports instead of two. The retained implementation produces exactly two asynchronous and two synchronous receipts, with the original creator, routing metadata and cost

The two surviving probes remove the later streaming guard, masked by early rejection in tested scenarios, and the logger-parameter rebind, masked by deployment IDs in chat responses. Both retained protections have narrower paths outside those tests. The parameter-rebind probe was rerun against both Chat and Messages and still survives. The final formatting change has an identical syntax tree, and the formatting-sensitive probe was rerun. Recorded source hashes match the final code

Delayed callback regressions use real Router and SDK calls, real asynchronous dispatch and synchronous S3 serialization, with only external transports doubled. They verify each receipt retains its original model group, deployment, creator and cost after logger reuse. Streaming regressions exercise the real code-interpreter hook and preserve the existing internal-call boundary

Real Redis verification covers concurrent admission across four processes, repeated cancellation after writes, atomic settlement, fixed windows, namespaces and Redis Cluster key placement. The real Logging and spend-writer counter path preserves unrelated spend and applies late costs once

Logging tests run separately because two unchanged test pairs leak callbacks into later tests and fail identically on the merge base. No selected tests were skipped

Production is byte-identical to b3029cf; the final amendment adds the Messages case to the existing regression. The prior production checks and 739-test result remain applicable, and all 422 Logging tests were rerun. The exact-tip paid replay passed: six successful source requests and twelve paid receipts totaling $0.0041673. Six evaluation receipts bill the correct creator, and both exhausted creator budgets withhold evaluation calls at zero shadow/judge cost

All ten required CI checks pass on this commit. Greptile reviewed it at 5/5, Bugbot reported no issues, Veria found no security issues, and all review threads are resolved

CodeQL is red for existing alert #9134, created August 17, 2026, at hash_token in litellm/proxy/utils.py:7141. The annotated module is byte-identical to the merge base. The earlier inherited Lens failure has cleared in the current CI run. Some additional CI jobs are still running; they are not counted as passes

Screenshots / Proof of Fix

The same six cases ran against isolated PostgreSQL databases and a local proxy, with real paid Together DeepSeek-V4.1-Flash source/shadow calls and Moonshot Kimi-K3 judge calls. Keys below are placeholders. <source-agent> in the receipt output represents the generated agent UUID

Shared proxy configuration:

model_list:
- model_name: qa-source
  litellm_params:
    model: together_ai/deepseek-ai/DeepSeek-V4.1-Flash
    api_key: os.environ/TOGETHERAI_API_KEY
    max_tokens: 50
    reasoning_effort: none
- model_name: qa-judge
  litellm_params:
    model: moonshot/kimi-k3
    api_key: os.environ/MOONSHOT_API_KEY
    extra_body:
      thinking:
        type: disabled
- model_name: qa-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      classifier_type: heuristic
      tiers:
        SIMPLE: qa-source
        MEDIUM: qa-source
        COMPLEX: qa-source
        REASONING: qa-source
- model_name: qa-priced
  litellm_params:
    model: openai/deepseek-ai/DeepSeek-V4.1-Flash
    api_key: os.environ/TOGETHERAI_API_KEY
    api_base: https://api.together.xyz/v1
    max_tokens: 50
    extra_body:
      reasoning_effort: none
    input_cost_per_token: 1.0e-06
    output_cost_per_token: 2.0e-06
  model_info:
    mode: chat
    max_input_tokens: 1048576
    max_output_tokens: 50
- model_name: qa-priced-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      classifier_type: heuristic
      tiers:
        SIMPLE: qa-priced
        MEDIUM: qa-priced
        COMPLEX: qa-priced
        REASONING: qa-priced
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 1
  allow_requests_on_db_unavailable: false
litellm_settings:
  drop_params: true
  telemetry: false
router_settings:
  num_retries: 0
  timeout: 60

Start the proxy on port 55183 with this configuration, provider credentials, and a fresh QA database for each commit. Create a separate evaluation owner and an agent-bound source key, then enable 100% shadow sampling:

PROXY=http://127.0.0.1:55183
MASTER_KEY='<local QA master key>'
QA_DATABASE=shadow_eval_rebuild_base  # shadow_eval_rebuild_after_5c76148 for the final commit

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-source","qa-router","qa-judge"]}' > admin.json
CREATOR_KEY=$(jq -r .key admin.json)
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-source-user","user_role":"internal_user","auto_create_key":false}'
AGENT_ID=$(curl -fsS "$PROXY/v1/agents" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"agent_name":"qa-source-agent"}' | jq -r .agent_id)
jq -n --arg agent "$AGENT_ID" \
  '{user_id:"qa-source-user",agent_id:$agent,models:["qa-source","qa-router","qa-judge"],max_budget:1,key_alias:"qa-source-key"}' > source-key-request.json
SOURCE_KEY=$(curl -fsS "$PROXY/key/generate" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  --data-binary @source-key-request.json | jq -r .key)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-source-user"],"router_name":"qa-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'

The setup returned HTTP 200 for users/agent/key creation and HTTP 201 for evaluation creation. Run cases in the listed order. New evaluation jobs propagate through the existing active-job cache within about ten seconds. For replay, wait eleven seconds after creating a job before sending its first sample. After each source response, wait for the evaluation receipts, then inspect the new rows with:

docker exec litellm-44356-pg psql -U qa -d "$QA_DATABASE" -c \
  'SELECT "user", agent_id, spend, metadata::jsonb->>'"'"'internal_call_origin'"'"' AS origin FROM "LiteLLM_SpendLogs" ORDER BY "startTime";'
docker exec litellm-44356-pg psql -U qa -d "$QA_DATABASE" -c \
  'SELECT outcome, error, shadow_cost, judge_cost FROM "LiteLLM_ShadowEvalAttempt" ORDER BY created_at;'

Before (`d306d6d`)

Chat billing

Run:

curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","prompt_tokens":34,"completion_tokens":33}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000498
shadow_eval_router  | qa-source-user | <none>         | 0.0000318
shadow_eval_judge   | qa-source-user | <none>         | 0.001518

evaluation outcome: real
shadow_cost: 3.179999999999999e-05; judge_cost: 0.001518

Creator total budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1e-06,"spend":1}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly READY."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"READY","prompt_tokens":35,"completion_tokens":14}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000273
shadow_eval_router  | qa-source-user | <none>         | 0.0000333
shadow_eval_judge   | qa-source-user | <none>         | 0.001473

evaluation outcome: tie
shadow_cost: 3.33e-05; judge_cost: 0.001473

Creator per-model budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{"qa-source":{"budget_limit":1e-06,"time_period":"1h"},"qa-router":{"budget_limit":1e-06,"time_period":"1h"},"qa-judge":{"budget_limit":1e-06,"time_period":"1h"}}}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly DONE."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"DONE","prompt_tokens":35,"completion_tokens":29}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000453
shadow_eval_router  | qa-source-user | <none>         | 0.0000501
shadow_eval_judge   | qa-source-user | <none>         | 0.001473

evaluation outcome: tie
shadow_cost: 5.01e-05; judge_cost: 0.001473

Responses billing

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{}}'
curl -fsS "$PROXY/v1/responses" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","input":"Reply exactly OK.","max_output_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":12}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000246

evaluation attempts: 0 after 75 seconds
job funnel: not_sampled=0, unjudgeable=1, shed=0, withheld=0

The matching active job classified this bridged Responses sample as unjudgeable. This case verifies the paid source request only; evaluation billing is unverified for this bridge

Messages billing

Run:

curl -fsS "$PROXY/v1/messages" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":41}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000594
shadow_eval_router  | qa-source-user | <none>         | 0.0000378
shadow_eval_judge   | qa-source-user | <none>         | 0.001578

evaluation outcome: tie
shadow_cost: 3.78e-05; judge_cost: 0.001578

Configured deployment pricing

Run:

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-priced","qa-priced-router","qa-judge"]}' > priced-admin.json
PRICED_CREATOR_KEY=$(jq -r .key priced-admin.json)
curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","model_max_budget":{"qa-priced":{"budget_limit":0.01,"time_period":"1h"}}}'
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-source-user","user_role":"internal_user","auto_create_key":false}'
jq -n --arg agent "$AGENT_ID" '{user_id:"qa-priced-source-user",agent_id:$agent,models:["qa-source","qa-priced","qa-priced-router","qa-judge"],max_budget:1}' > priced-key-request.json
curl -fsS "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' --data-binary @priced-key-request.json > priced-key.json
PRICED_SOURCE_KEY=$(jq -r .key priced-key.json)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $PRICED_CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-priced-source-user"],"router_name":"qa-priced-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'
sleep 11  # allow the existing active-job cache to refresh
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $PRICED_SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly PRICE."}],"max_tokens":50}'

Observed response and new receipts:
```
HTTP 200
{"content":"PRICE","prompt_tokens":35,"completion_tokens":37}

origin              | billed user    | agent          | USD
source              | qa-priced-source-user | <source-agent> | 0.0000549
shadow_eval_router  | qa-priced-source-user | <none>         | 0.000015
shadow_eval_judge   | qa-priced-source-user | <none>         | 0.001584

evaluation outcome: tie
shadow_cost: 1.5e-05; judge_cost: 0.001584
```
The shadow deployment uses the real Together endpoint through its OpenAI-compatible API. Its provider-qualified model has no built-in price entry; the proxy config supplies input/output prices of $1/$2 per million tokens. The evaluation succeeds under the creator’s $1 total limit. The configured $0.01 hourly qa-priced cap is not exercised because this routed request uses the qa-priced-router budget identity; focused tests separately cover configured per-model pricing

After (`5c76148`)

Chat billing

Run:

curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","prompt_tokens":34,"completion_tokens":17}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000306
shadow_eval_router  | qa-eval-admin  | <none>         | 0.0000294
shadow_eval_judge   | qa-eval-admin  | <none>         | 0.001473

evaluation outcome: tie
shadow_cost: 2.94e-05; judge_cost: 0.001473

Creator total budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1e-06,"spend":1}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly READY."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"READY","prompt_tokens":35,"completion_tokens":14}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000273

evaluation outcome: error
error: shadow router call failed: BudgetExceededError at evaluation_budget.py:314: Budget has been exceeded! Current cost: 1.0, Max budget: 1e-06
shadow_cost: 0; judge_cost: 0

Creator per-model budget

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{"qa-source":{"budget_limit":1e-06,"time_period":"1h"},"qa-router":{"budget_limit":1e-06,"time_period":"1h"},"qa-judge":{"budget_limit":1e-06,"time_period":"1h"}}}'
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly DONE."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"DONE.","prompt_tokens":35,"completion_tokens":32}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000489

evaluation outcome: error
error: shadow router call failed: BudgetExceededError at evaluation_budget.py:371: Budget has been exceeded! Current cost: 0.0, Max budget: 1e-06
shadow_cost: 0; judge_cost: 0

Responses billing

Run:

curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-eval-admin","max_budget":1,"spend":0,"model_max_budget":{}}'
curl -fsS "$PROXY/v1/responses" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","input":"Reply exactly OK.","max_output_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK.","input_tokens":34,"output_tokens":20}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000342

evaluation attempts: 0 after 75 seconds
job funnel: not_sampled=0, unjudgeable=1, shed=0, withheld=0

The matching active job classified this bridged Responses sample as unjudgeable. This case verifies the paid source request only; evaluation billing is unverified for this bridge

Messages billing

Run:

curl -fsS "$PROXY/v1/messages" \
  -H "Authorization: Bearer $SOURCE_KEY" -H 'Content-Type: application/json' -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly OK."}],"max_tokens":50}'

Observed response and new receipts:

HTTP 200
{"content":"OK","input_tokens":34,"output_tokens":12}

origin              | billed user    | agent          | USD
source              | qa-source-user | <source-agent> | 0.0000246
shadow_eval_router  | qa-eval-admin  | <none>         | 0.0000426
shadow_eval_judge   | qa-eval-admin  | <none>         | 0.0008268

evaluation outcome: tie
shadow_cost: 4.26e-05; judge_cost: 0.0008267999999999999

Configured deployment pricing

Run:

curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","user_role":"proxy_admin","max_budget":1,"models":["qa-priced","qa-priced-router","qa-judge"]}' > priced-admin.json
PRICED_CREATOR_KEY=$(jq -r .key priced-admin.json)
curl -fsS "$PROXY/user/update" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-admin","model_max_budget":{"qa-priced":{"budget_limit":0.01,"time_period":"1h"}}}'
curl -fsS "$PROXY/user/new" \
  -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' \
  -d '{"user_id":"qa-priced-source-user","user_role":"internal_user","auto_create_key":false}'
jq -n --arg agent "$AGENT_ID" '{user_id:"qa-priced-source-user",agent_id:$agent,models:["qa-source","qa-priced","qa-priced-router","qa-judge"],max_budget:1}' > priced-key-request.json
curl -fsS "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" -H 'Content-Type: application/json' --data-binary @priced-key-request.json > priced-key.json
PRICED_SOURCE_KEY=$(jq -r .key priced-key.json)
curl -fsS "$PROXY/auto_router/shadow_eval/start" \
  -H "Authorization: Bearer $PRICED_CREATOR_KEY" -H 'Content-Type: application/json' \
  -d '{"user_ids":["qa-priced-source-user"],"router_name":"qa-priced-router","shadow_percentage":100,"judge_model":"qa-judge","duration_days":1,"max_budget":0.1}'
sleep 11  # allow the existing active-job cache to refresh
curl -fsS "$PROXY/v1/chat/completions" \
  -H "Authorization: Bearer $PRICED_SOURCE_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"qa-source","messages":[{"role":"user","content":"Reply exactly PRICE."}],"max_tokens":50}'

Observed response and new receipts:
```
HTTP 200
{"content":"PRICE","prompt_tokens":35,"completion_tokens":17}

origin              | billed user    | agent          | USD
source              | qa-priced-source-user | <source-agent> | 0.0000309
shadow_eval_router  | qa-priced-admin | <none>         | 0.000015
shadow_eval_judge   | qa-priced-admin | <none>         | 0.001584

evaluation outcome: tie
shadow_cost: 1.5e-05; judge_cost: 0.001584
```
The shadow deployment uses the real Together endpoint through its OpenAI-compatible API. Its provider-qualified model has no built-in price entry; the proxy config supplies input/output prices of $1/$2 per million tokens. The evaluation succeeds under the creator’s $1 total limit. The configured $0.01 hourly qa-priced cap is not exercised because this routed request uses the qa-priced-router budget identity; focused tests separately cover configured per-model pricing

Paid nested evaluation-owned Messages/compaction remains unverified: native compaction requires Anthropic, whose account rejected the earlier probe for exhausted credits. The Messages case above tests a public Messages source request; focused provider-transport regressions separately exercise evaluation-owned nested calls

Type

Bug Fix

Caveats

Severe

Missing creator accounts withhold evaluations until an account exists

Medium

Bridged Responses samples remain unjudgeable on both tested revisions
Paid native compaction lacks funded Anthropic credentials

Low

Screenshot and recording tools are unavailable in this session
CodeQL flags unchanged API-token hashing

tin-berri · 2026-10-03T20:02:05Z

bugbot run Please review this fresh implementation for creator billing, budget admission, cancellation, concurrent settlement, and receipt attribution regressions.

greptile-apps · 2026-10-03T20:05:56Z

Confidence Score: 5/5

[High risk] Adds evaluation billing isolation and budget tracking.

The PR appears safe to merge based on the reviewed changes and the resolved prior findings.

Summary

The PR bills shadow-evaluation calls to the evaluation creator, admits paid attempts against the creator’s budgets, and settles their spend independently of sampled source traffic. The latest change extends the deferred-receipt regression to Messages and verifies that logger reuse preserves distinct Chat and Messages receipts without duplicate synchronous exports.

_{Reviews (7) · Last reviewed commit: "fix(shadow-eval): isolate creator billin..."}

cursor

Stale Bugbot comment from a previous run.

codspeed · 2026-10-03T20:12:08Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_shadow_eval_creator_billing (5c76148) with main (98337c9)}

codecov · 2026-10-03T20:16:36Z

Codecov Report

❌ Patch coverage is 92.80206% with 28 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/spend_tracking/evaluation_budget.py	88.77%	21 Missing ⚠️
litellm/litellm_core_utils/litellm_logging.py	89.47%	4 Missing ⚠️
litellm/integrations/shadow_eval_logger.py	96.00%	1 Missing ⚠️
litellm/proxy/hooks/model_max_budget_limiter.py	83.33%	1 Missing ⚠️
litellm/utils.py	97.22%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

tin-berri · 2026-10-03T20:26:45Z

@greptileai Please re-review the fixes: complete evaluation responses, verified creator accounts, window-aligned counter expiry, and independent admission for each fallback attempt

tin-berri · 2026-10-03T20:26:46Z

bugbot run Please review the updated creator billing implementation, including global fallback admission, complete evaluation responses, missing creators, and budget expiry

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T20:47:16Z

@greptileai Please review the updated Messages admission and SDK judge fallback fixes, including native calls, translated attempts, cached receipts, and creator budgets

tin-berri · 2026-10-03T20:47:17Z

bugbot run Please review the latest complete implementation, especially Messages fallback admission, native budgets, receipt isolation, and fallback-free judge calls

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T21:10:52Z

@greptileai Please review the final callback isolation fix and overall creator billing design, including cached receipts, delayed failures, and budget settlement

tin-berri · 2026-10-03T21:10:53Z

bugbot run Please review the final creator billing implementation, especially queued synchronous callbacks, receipt ownership after reuse, and budget settlement

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T21:20:07Z

@greptileai Please review the final provider helper placement and confirm the creator billing and receipt isolation design has no remaining issues

tin-berri · 2026-10-03T21:20:08Z

bugbot run Please review the final creator billing implementation after relocating Messages provider selection. Verify callback isolation, admission, and settlement remain correct

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T21:37:28Z

@greptileai Please review the latest commit, including detached routing metadata in deferred receipts and evaluation streaming rejection before deployment hooks

tin-berri · 2026-10-03T21:37:29Z

bugbot run Please review the latest commit, focusing on receipt isolation, pre-hook streaming rejection, creator billing, and budget settlement

cursor

Stale Bugbot comment from a previous run.

tin-berri · 2026-10-03T21:54:23Z

@greptileai Please review the Chat/Messages receipt regression and reassess shared dictionary concerns; copying produces duplicate synchronous exports, while current behavior passes

tin-berri · 2026-10-03T21:54:24Z

bugbot run Please review the final test amendment verifying receipt ownership, routing metadata, and exactly one synchronous export per bridged call

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5c76148. Configure here.}

tin-berri requested a review from a team October 3, 2026 20:01

tin-berri had a problem deploying to e2e-changed October 3, 2026 20:02 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/evaluation_budget.py Outdated

Comment thread litellm/integrations/shadow_eval_logger.py Outdated

This comment has been minimized.

Sign in to view

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from c8d26d1 to d4b2c44 Compare October 3, 2026 20:26

tin-berri had a problem deploying to e2e-changed October 3, 2026 20:26 — with GitHub Actions Error

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py

Comment thread litellm/integrations/shadow_eval_logger.py

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from d4b2c44 to ddcfa3c Compare October 3, 2026 20:46

tin-berri had a problem deploying to e2e-changed October 3, 2026 20:47 — with GitHub Actions Error

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/caching/caching_handler.py

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from ddcfa3c to d30808f Compare October 3, 2026 21:10

tin-berri had a problem deploying to e2e-changed October 3, 2026 21:10 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/utils.py Outdated

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from d30808f to 5613f70 Compare October 3, 2026 21:19

tin-berri had a problem deploying to e2e-changed October 3, 2026 21:19 — with GitHub Actions Error

cursor Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/litellm_logging.py

Comment thread litellm/utils.py

cursor Bot reviewed Oct 3, 2026

View reviewed changes

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from 5613f70 to b3029cf Compare October 3, 2026 21:37

tin-berri had a problem deploying to e2e-changed October 3, 2026 21:37 — with GitHub Actions Error

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/litellm_core_utils/litellm_logging.py

cursor Bot reviewed Oct 3, 2026

View reviewed changes

fix(shadow-eval): isolate creator billing and budget settlement

5c76148

tin-berri force-pushed the litellm_shadow_eval_creator_billing branch from b3029cf to 5c76148 Compare October 3, 2026 21:53

tin-berri requested a deployment to e2e-changed October 3, 2026 21:53 — with GitHub Actions Waiting

cursor Bot reviewed Oct 3, 2026

View reviewed changes

tin-berri mentioned this pull request Oct 3, 2026

fix(shadow-eval): run evaluation calls as the job creator #44464

Open

4 of 6 tasks

moyai-devin-berriai Bot mentioned this pull request Oct 5, 2026

fix(shadow-eval): validate dependencies and stop repeated failures #44652

Open

3 of 5 tasks

Conversation

tin-berri commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Design

Pre-Submission checklist

Validation

Screenshots / Proof of Fix

Before (d306d6d)

Chat billing

Creator total budget

Creator per-model budget

Responses billing

Messages billing

Configured deployment pricing

After (5c76148)

Chat billing

Creator total budget

Creator per-model budget

Responses billing

Messages billing

Configured deployment pricing

Type

Caveats

Severe

Medium

Low

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

tin-berri commented Oct 3, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tin-berri commented Oct 3, 2026 •

edited

Loading

Before (`d306d6d`)

After (`5c76148`)

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codspeed Bot commented Oct 3, 2026 •

edited

Loading

codecov Bot commented Oct 3, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading