feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state)

[tin-berri]

TLDR

Problem this solves:

• Admins cannot answer "is it safe to turn the auto-router on?" before routing production traffic through it
• Prior attempts (feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates) #36250, feat: pre-adoption shadow eval for the auto-router (backend) #36571) accumulated counters, a status column, and lifecycle loops whose cross-pod consistency drew review round after review round

How it solves it:

• A shadow eval job samples a slice of one key's successful chat traffic, duplicates each sampled request through the auto-router in a detached task (zero added latency, shadow responses never served), and has an LLM judge compare both responses blind with A/B labels randomized
• One append-only LiteLLM_ShadowEvalAttempt row per sampled pipeline (a verdict or an error) is the feature's only hot-path write; counts, status, judge spend, and the latest error are all derived at read time, so nothing can disagree across pods, stop races, or partial failures
• The job row is immutable config plus stopped_at; status is derived (stopped, completed by expiry, running). A max_turns sample budget bounds the job instead of dollar-estimate machinery, and the hook reads active jobs through a 10s TTL in-memory cache filled lazily on the logging path, so registration is one callback add with nothing to start

User Flow

1. Admin calls POST /auto_router/shadow_eval/start with a key, a configured auto-router, a sampling percentage, a judge model, a duration, and a turn budget
2. Traffic flows normally; pods pick the job up within one 10s cache TTL and each sampled request produces exactly one attempt row
3. GET /auto_router/shadow_eval/{job_id} returns derived counts, judge spend, latest error, and win rates by tier and by incumbent model; POST .../stop sets one timestamp; the job also ends on its own at max_turns or its window's end

Relevant issues

• Ground-up rebuild of the shadow eval feature; supersedes feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates) #36250, feat: pre-adoption shadow eval for the auto-router (backend) #36571, and feat(ui): shadow eval section on the auto-router usage tab #36572, incorporating the full review ledger from all three (the design doc's disposition table covers every finding)
• Carries forward the review-validated shared owners from feat: pre-adoption shadow eval for the auto-router (backend) #36571 unchanged: litellm_core_utils/llm_judge.py (fence-tolerant verdict parse, router-or-SDK judge dispatch, shared resolvability predicate, now also used by the llm_as_a_judge guardrail instead of its private copies) and litellm_core_utils/internal_call_metadata.py (identity forwarding, budget-reservation stripping, also adopted by the complexity router)

Changes

• New LiteLLM_ShadowEvalJob (10 columns, one mutable) and append-only LiteLLM_ShadowEvalAttempt tables; one active job per key via a partial unique index on stopped_at IS NULL, with the start endpoint stamping expired jobs so the index never blocks a new eval
• New ShadowEvalLogger: skip chain (internal-origin, redaction via should_redact_message_logging itself, sampling hash, fail-closed call_type allowlist, self-shadow, turn budget, in-flight cap of 16) then one detached pipeline ending in one row; budget gate delegates to the auth path's own _virtual_key_max_budget_check / _team_max_budget_check
• Four endpoints (start, list, get, stop); start validates router, judge resolvability through the same predicate the dispatch uses, and key existence; responses validate straight off prisma records with status as a pydantic computed field
• Shadow/judge sub-calls carry the caller's identity, bill to the shadowed key, run with retries and fallbacks disabled, and are excluded from api_requests, auto-router savings, session rollups, and the v3 rate limiter's TPM counters, so a running eval cannot throttle the key's real traffic

What a reviewer will ask about

• Read cost: detail aggregates are two GROUP BYs plus one totals scan over one job's attempts, bounded by max_turns (cap 2000) through the job_id index; list reads job rows only. The hook's cache fill is one indexed point-read per pod per 10s, on the logging task, never the response path
• Stop/expiry propagate within one cache TTL. The turn budget pairs the cached attempt count with a pod-local count of starts since that cache fill, which never decrements within a generation, so a pod admits at most the job's remaining budget between refills; a trailing attempt row after stop is just data, and every aggregate derives from the same rows, so nothing desynchronizes
• Redacted requests are skipped outright: redaction rewrites logged content before callbacks run, so the hook only ever sees placeholders for them
• The budget gate is advisory by design (auth already rejects real traffic on over-budget keys; this closes the crossed-mid-flight race), and a judge reached via the SDK honors a deployer's global model_fallbacks, an explicit deployment-wide policy
• The owed live-provider pass is done: the second proof block below is the same flow against real Anthropic models on paid credentials, at the same commit

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Live proxy on this branch (port 4216, fresh Postgres with the migration history deployed, stub upstream minting unique response ids). On the merge base the same start and list curls return 404

$ curl -s -X POST localhost:4216/auto_router/shadow_eval/start -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"api_key_id":"df3131c0...","router_name":"auto_router1","shadow_percentage":100,"judge_model":"gpt-5-mini","duration_days":1,"max_turns":50}'
{"job_id":"cmspd95ef0000tpfbcfi5oacw","status":"running","max_turns":50,"judged_count":null, ...}

duplicate start -> 409    unknown key -> 400    judge "not a model!!" -> 400

10 chat requests on the shadowed key, then the detail endpoint derives everything:

{'status': 'running', 'judged_count': 10, 'error_count': 0, 'judge_spend': 0.000962, 'last_error': None, 'max_turns': 50}
by_tier: [('REASONING', 4, 50.0), ('SIMPLE', 3, 100.0), ('MEDIUM', 2, 50.0), ('COMPLEX', 1, 0.0)]
by_model: [('openai/gpt-5', 10)]
overall shadow/tie: 60.0 10.0

Spend attribution and metric exclusion, straight from the rig's Postgres:

        origin         | count |  spend
-----------------------+-------+----------
 autorouter_classifier |    10 | 0.000182
 shadow_eval_judge     |    10 | 0.000962
 shadow_eval_router    |    10 | 0.001653
 user_request          |    10 | 0.001151
-- all 40 rows billed to the shadowed key

 openai/gpt-5      api_requests=10   <- user traffic
 openai/gpt-5-mini api_requests=0    <- judge + shadow: spend yes, requests no
 (three more internal models, all api_requests=0)

 LiteLLM_AutoRouterSession rows = 0  <- shadow duplicates never reach adoption metrics

Stop and restart, both through the derived state and the partial index:

stop -> {"status": "stopped"}    stop again -> 400    new start on the same key -> 201
list -> [running, stopped]  (statuses derived, never stored)

Live provider e2e, real Anthropic calls, real $

Same commit d98b0fe, second rig: proxy on port 4000 against Postgres, a qa-router complexity auto-router over anthropic/claude-haiku-4-5 and anthropic/claude-sonnet-4-5, judge anthropic/claude-sonnet-5, incumbent traffic on anthropic-haiku-4-5. No stubs anywhere, every call billed to a live Anthropic account

Lifecycle and validation, admin key:

$ curl -s -w " %{http_code}\n" -X POST localhost:4000/auto_router/shadow_eval/start -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"api_key_id":"62f000e9...","router_name":"qa-router","shadow_percentage":100,"judge_model":"anthropic-sonnet-5","duration_days":1,"max_turns":6}'
{"job_id":"cmsqwz5yb0000qhlmf3v5ml9t","router_name":"qa-router","judge_model":"anthropic-sonnet-5","shadow_percentage":100.0,"max_turns":6,"ends_at":"2026-08-14T02:43:44.482000Z","stopped_at":null,"status":"running"} 201

$ # same key again
{"detail":"Key already has an active shadow eval job (cmsqwz5yb0000qhlmf3v5ml9t). Stop it first."} 409

unknown key -> 400    unknown router -> 400    unresolvable judge -> 400    judge that is itself an auto-router -> 400
unknown job id -> 404    non-admin key on start and list -> 401 at the auth layer

Four real chat requests on the shadowed key (a factual one, a coding one, a definition, a system-design one), then the detail endpoint after the detached tasks land:

$ curl -s localhost:4000/auto_router/shadow_eval/cmsqwz5yb0000qhlmf3v5ml9t -H "Authorization: Bearer sk-1234"
{
  "judged_count": 4, "error_count": 0, "judge_spend": 0.007274, "last_error": null, "status": "running",
  "results": {
    "by_tier": [
      {"group": "SIMPLE",  "turn_count": 2, "real_win_rate_pct": 0.0, "shadow_win_rate_pct": 50.0, "tie_rate_pct": 50.0,  "avg_judge_confidence": 0.825},
      {"group": "COMPLEX", "turn_count": 1, "real_win_rate_pct": 0.0, "shadow_win_rate_pct": 0.0,  "tie_rate_pct": 100.0, "avg_judge_confidence": 0.55},
      {"group": "MEDIUM",  "turn_count": 1, "real_win_rate_pct": 0.0, "shadow_win_rate_pct": 0.0,  "tie_rate_pct": 100.0, "avg_judge_confidence": 0.55}
    ],
    "by_current_model": [
      {"group": "anthropic/claude-haiku-4-5", "turn_count": 4, "shadow_win_rate_pct": 25.0, "tie_rate_pct": 75.0, "avg_judge_confidence": 0.688}
    ],
    "overall_shadow_win_rate_pct": 25.0, "overall_tie_rate_pct": 75.0
  }
}

The four attempt rows behind those aggregates, with the tier the router picked and what the judge paid:

$ psql -d litellm -c 'select outcome, tier, real_model, shadow_model, confidence, judge_cost, error from "LiteLLM_ShadowEvalAttempt";'
 outcome |  tier   |         real_model         |        shadow_model        | confidence | judge_cost | error
---------+---------+----------------------------+----------------------------+------------+------------+-------
 tie     | SIMPLE  | anthropic/claude-haiku-4-5 | claude-haiku-4-5-20251001  |          1 |   0.000946 |
 tie     | COMPLEX | anthropic/claude-haiku-4-5 | claude-sonnet-4-5-20250929 |       0.55 |   0.002064 |
 shadow  | SIMPLE  | anthropic/claude-haiku-4-5 | claude-haiku-4-5-20251001  |       0.65 |   0.002218 |
 tie     | MEDIUM  | anthropic/claude-haiku-4-5 | claude-haiku-4-5-20251001  |       0.55 |   0.002046 |

Spend attribution and metric exclusion on live spend, straight from Postgres:

$ psql -d litellm -c 'select api_key, api_requests, successful_requests, round(spend::numeric,8) spend, autorouter_savings_spend, model from "LiteLLM_DailyUserSpend" order by api_requests desc;'
     api_key      | api_requests | successful_requests |   spend    | autorouter_savings_spend |            model
------------------+--------------+---------------------+------------+--------------------------+-----------------------------
 62f000e9...83896 |            4 |                   4 | 0.00398200 |                        0 | anthropic/claude-haiku-4-5
 62f000e9...83896 |            0 |                   0 | 0.00232800 |                        0 | anthropic/claude-sonnet-4-5
 62f000e9...83896 |            0 |                   0 | 0.00727400 |                        0 | anthropic/claude-sonnet-5

$ psql -d litellm -c 'select count(*) from "LiteLLM_AutoRouterSession";'
 0

Four user requests counted once, shadow and judge rows billed to the same key at zero requests and zero savings, no adoption sessions from the duplicates

Turn budget, on a fresh job with max_turns: 1 and three more real requests:

$ psql -d litellm -c 'select job_id, count(*) from "LiteLLM_ShadowEvalAttempt" group by job_id;'
 cmsqwz5yb0000qhlmf3v5ml9t |     4
 cmsqx1oe80009qhlmvfk1f9ka |     1

$ curl -s localhost:4000/auto_router/shadow_eval/cmsqx1oe80009qhlmvfk1f9ka -H "Authorization: Bearer sk-1234"
judged 1  errors 0  spend 0.001868  last_error None

Stop, restart, and the list, all derived:

stop            -> {"stopped_at":"2026-08-13T02:45:11.406000Z","status":"stopped"} 200
stop again      -> {"detail":"Job cmsqwz5yb0000qhlmf3v5ml9t is already stopped"} 400
new start, same key -> {"job_id":"cmsqx111z0008qhlmsdhftfzf","status":"running"} 201
list            -> [running, stopped] 200

Type

🆕 New Feature

Caveats (if any)

• Turn-level only; the shadow never influences the next real turn
• Only /v1/chat/completions traffic is sampled, by the declared call_type allowlist
• Requests with message-logging redaction are never sampled; their content is unavailable to callbacks by design
• shadow_model records the deployment's model name while real_model records the public alias
• Tier slices where the router picks the incumbent model compare one model against itself
• Local test note: tests/test_litellm/proxy/db/test_prisma_planned_engine_restart.py fails identically at the merge base in this environment (prisma engine instrumentation, unrelated to this diff)

QA runbook

1. Configure an auto-router plus tier models, start the proxy with a database
2. Start a job as admin with a real key hash; expect 201 with derived status running, 409 on repeat, 400 for unknown router, unresolvable judge, or unknown key
3. Send chat requests with the shadowed key, wait ~15s, GET the job; expect judged_count, judge_spend, and both stratifications to grow, and LiteLLM_DailyUserSpend.api_requests to count only user requests while shadow and judge spend bills to the key
4. Stop the job; expect status stopped and a new start on the same key to succeed

Note for anyone rebuilding a local rig: the partial unique index ships in the migration only, so a prisma db push schema sync leaves the one-active-job-per-key guard off

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Touches spend attribution, rate limiting, and budget checks, and bills background shadow/judge calls to user keys. Mitigated by admin-only APIs, never serving shadow responses, and excluding internal calls from request/TPM/adoption metrics.

Overview
Admins can now pre-evaluate an auto-router against one key's live traffic before adoption. A sampled slice of successful chat requests is duplicated through the router in a detached task; an LLM judge compares real vs shadow responses blind (A/B labels randomized). Shadow responses are never served to users.

Job state is append-only and derived. New LiteLLM_ShadowEvalJob / LiteLLM_ShadowEvalAttempt tables store immutable config plus one attempt row per sample. Counts, status, judge spend, and stratified win rates (by tier and incumbent model) are computed at read time. One active job per key is enforced with a partial unique index.

Admin APIs: POST /auto_router/shadow_eval/start, list/get, and stop. Sampling runs via a new ShadowEvalLogger (10s job cache, turn budget, concurrency cap). Shadow/judge spend bills to the shadowed key but is excluded from request counts, auto-router adoption metrics, and TPM rate limits.

Also extracts shared llm_judge and internal_call_metadata helpers (adopted by the llm-as-a-judge guardrail and complexity router).

^{Reviewed by Cursor Bugbot for commit d98b0fe. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/69a057010d944f3f97d4ac59d9edb451

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's the breakdown:

---

What earns the high marks:

• Derived-state architecture is the right call. No counters to flush, no status fields to race on — everything aggregates from append-only attempt rows, so nothing can desync across pods. This directly solves what killed the predecessor PRs.
• Zero-latency hot path. asyncio.create_task with no await; the success hook is a dict lookup plus task scheduling. The 10s TTL cache means one point-read per pod per TTL on the logging task, never the response path.
• Skip chain is airtight and tested. Internal-origin → redaction → key hash → job → expiry → turn budget → call_type → self-shadow → in-flight cap. Every skip path has a test asserting zero attempt rows.
• Identity forwarding is correct. sanitized_forwardable_call_metadata strips user_api_key_budget_reservation and per-request state (routing decision, logging payload) while carrying key/team/org/user identity. The budget-double-counting vector is closed.
• Spend exclusion in db_spend_update_writer.py is correct. is_internal_call zeros api_requests, successful_requests, failed_requests, and autorouter_savings_spend for shadow and judge sub-calls. The proof confirms this in live Postgres output.
• Concurrent-start race is handled properly. Partial unique index + _is_unique_violation fallback turns a concurrent 500 into a reliable 409.

---

What holds it back from 5/5:

1. Missing FK constraint on LiteLLM_ShadowEvalAttempt.job_id. Neither schema.prisma nor the migration adds a foreign key to LiteLLM_ShadowEvalJob. Prisma supports this via @relation. Without it, attempt rows can reference non-existent jobs, cascade deletes are impossible, and orphaned rows accumulate silently.

2. N+1 count queries in _active_jobs. On every cache miss, it runs one litellm_shadowevalattempt.count() per active job to populate the attempts field. With the 1-per-key constraint this is bounded in practice, but a single GROUP BY job_id query would be cleaner and cheaper.

3. _pct_of is a dead wrapper that just delegates to _pct with the same arguments. Minor noise, but it adds nothing.

4. Live provider staging is still pending. The PR explicitly acknowledges this: "a staging pass with live providers is owed before merge." The proof-of-fix used stub upstreams with no real provider credentials. That's a process gap for something that bills real spend to keys.

5. ShadowEvalLogger is registered unconditionally in cost_tracking() whenever prisma_client is not None. There's no feature flag or config opt-out. For deployments that never start a shadow eval job, this adds a hook on every success callback (fast, but non-zero).

---

The design is genuinely solid and well-engineered. Items 1 and 4 are the ones worth resolving before merge; the rest are minor.

[greptile-apps]

Greptile Summary

The PR adds pre-adoption shadow evaluation for auto-router configurations, with append-only attempt records and derived status and aggregates

• Adds shadow-evaluation job and attempt schemas, migration, management endpoints, and response models
• Registers a logging callback that samples successful chat traffic and runs detached router and judge calls
• Centralizes internal-call metadata and judge utilities, then excludes internal calls from request and auto-router adoption metrics

Confidence Score: 4/5

The PR appears safe to merge after considering two non-blocking maintainability and database-access concerns

The accepted issues concern direct database reads during success-callback processing and excessive explanatory comments; no blocking correctness or security failure remains established

Files Needing Attention: litellm/integrations/shadow_eval_logger.py, litellm/litellm_core_utils/internal_call_metadata.py, litellm/proxy/management_endpoints/auto_router_endpoints.py

Important Files Changed

Filename	Overview
litellm/integrations/shadow_eval_logger.py	Adds the detached shadow-evaluation pipeline, but directly queries Prisma while handling successful-call callbacks and contains excessive narrative documentation
litellm/proxy/management_endpoints/auto_router_endpoints.py	Adds admin-only start, list, detail, and stop endpoints with derived aggregate results and concurrent-start conflict handling
litellm/litellm_core_utils/internal_call_metadata.py	Adds shared identity forwarding and budget-reservation stripping for internal LLM calls, with excessive module-level explanation
litellm/litellm_core_utils/llm_judge.py	Extracts shared judge parsing, model resolution, text extraction, and dispatch helpers
litellm/proxy/db/db_spend_update_writer.py	Preserves internal-call spend and token attribution while excluding those calls from request counts and auto-router savings
litellm/proxy/db/autorouter_session_rollup.py	Excludes internal shadow traffic from auto-router session and adoption rollups
litellm-proxy-extras/litellm_proxy_extras/migrations/20260811172448_add_shadow_eval/migration.sql	Creates shadow-evaluation tables, indexes attempts by job, and enforces one unstopped job per key with a partial unique index
schema.prisma	Adds matching shadow-evaluation job and attempt models to the root schema

_{Reviews (1): Last reviewed commit: "feat: pre-adoption shadow eval for the a..." | Re-trigger Greptile}

[greptile-apps]

Direct database reads in callback

On each cache miss, the success callback directly queries jobs and counts every job's attempts, adding database work proportional to active evaluations to request logging.

Rule Used: What: In critical path of request, there should be... (source)

Knowledge Base Used:

• Logging & Observability Integrations
• Database Schema and Proxy DB Access Layer

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[tin-berri]

Reduced to two queries per cache TTL via group_by; the cache-aside read on the logging task mirrors ownership.py's established pattern

[tin-berri]

Follow-up in 4213533: the count is now scoped to active job ids and skipped when none are active, both pinned by tests

[greptile-apps]

Excessive narrative source comments

The new production modules extensively restate feature design and implementation details, increasing maintenance cost and creating prose that can drift from the code.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[tin-berri]

Trimmed: the module docstring is five lines now and the narrative block comments are removed

[veria-ai]

Low: Shadow-eval turn limit can be bypassed

job.attempts remains unchanged for the cache TTL and no turn is claimed before scheduling provider calls. A user of the shadowed key can submit concurrent sampled completions while this value is below max_turns; even a one-turn job can start up to 16 pipelines per pod, with further waves possible before the cache refreshes, exceeding the configured spend bound. Atomically reserve an attempt in the database before dispatch and skip the calls when the reservation would exceed max_turns.

[tin-berri]

Hardened: in-flight pipelines now count against the turn budget, and the residual overshoot bound (one TTL, 16 per pod) is documented; a DB reservation would reintroduce the state class this design removed

[veria-ai]

PR overview

This pull request adds pre-adoption shadow evaluation for the auto-router, using blind pairwise judging and derived evaluation state. It also integrates the associated classifier, shadow, and judge model calls with proxy request handling.

Two security issues remain open, while one has already been addressed. Authenticated users can cause internal classifier calls to consume provider tokens without being counted against TPM limits, and concurrent shadow-evaluation requests can exceed the configured turn and spend bound. These gaps could permit bounded but repeated excess provider usage and cost until accounting and atomic turn reservation are enforced.

Open issues (2)

• Low: Shadow-eval turn limit can be bypassed — litellm/integrations/shadow_eval_logger.py:323
• Medium: Internal model calls bypass TPM limits — litellm/proxy/hooks/parallel_request_limiter_v3.py:3003

Fixed/addressed: 1 · PR risk: 6/10

[codecov]

Codecov Report

❌ Patch coverage is 91.35560% with 44 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/shadow_eval_logger.py	86.56%	34 Missing ⚠️
...roxy/management_endpoints/auto_router_endpoints.py	91.15%	10 Missing ⚠️

📢 Thoughts on this report? Let us know!

[tin-berri]

All four findings addressed in one round on 669823b. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

4/5 — here's the breakdown.

What earns the 4:

• Derived-state design is airtight. Job rows hold immutable config + stopped_at; every count, status, and spend aggregates from append-only attempt rows. Zero cross-pod races by construction.
• Skip chain is comprehensive and fail-closed. Internal-origin → redaction → sampling hash → call_type allowlist → self-shadow → turn budget → inflight cap. Unknown call types fall through to the disallow branch, not the allow branch.
• Budget gate delegates to auth's own checks. _virtual_key_max_budget_check / _team_max_budget_check directly — counter keys and thresholds are literally the same as what auth uses, so drift is impossible.
• Metadata forwarding is correct. Budget reservation stripped, per-request state (routing_decision, logging payload) excluded from detached calls via sanitized_forwardable_call_metadata, only identity keys forwarded.
• DB enforcement is correct. Partial unique index for one-active-per-key, race handled at the DB layer and surfaced as a 409 via _is_unique_violation.
• _validate_judge_model uses the same predicate as dispatch (router_resolves_model), so start-time validation cannot accept a name the call path then fails on.
• Tests are thorough. All 7 skip paths including redaction, cache miss/empty/fault, shadow fail, unparseable verdict (with billed cost preserved), metadata forwarding + budget reservation exclusion — all exercised explicitly.

What keeps it from 5:

1. Staging pass still owed. Live proof ran against a stub upstream. Spend attribution to the key, api_requests exclusion, and session rollup exclusion are the features the PR's own proof-of-work screenshot demonstrates — and none of those ran through a real provider. That's the highest-risk path in this diff.

2. _active_jobs cache fill does an unbounded GROUP BY. The attempt count query has no WHERE clause:

   grouped = await prisma.db.litellm_shadowevalattempt.group_by(by=["job_id"], count=True)

   This scans every attempt row for every job ever — active, stopped, and completed. For a long-running deployment the table grows unboundedly and the 10s cache miss pays a full table scan. The fix is cheap: filter to the active job IDs fetched in the same call:

   where={"job_id": {"in": [str(r.id) for r in records]}} if records else {}

Merge after confirming the staging pass; fix the GROUP BY scope before this sees a high-throughput deployment.

[tin-berri]

Round two batch on 4213533: scoped attempt counts, window-passed status wins over the sweep stamp, in-flight pipelines count against the turn budget. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

All three round-two items are correctly implemented:

Scoped attempt counts — _active_jobs scopes the group_by to {"job_id": {"in": [str(record.id) for record in records]}}, so the cached attempts field only reflects attempts for currently active jobs, not the full history of a key's completed evals.

Window-passed status wins over sweep stamp — the status computed field checks ends_at <= now before stopped_at is not None. A job whose window expired after a sweep stamped stopped_at still returns completed, not stopped. The stop endpoint correctly rejects the call in that case too.

In-flight pipelines count against the turn budget — job.attempts + self._inflight_shadow_tasks >= job.max_turns gates before launching each pipeline, and the budget-consumed-by-inflight test case (attempts=199, max_turns=200, _inflight=1) verifies the increment is counted before the cap fires. The known per-pod overshoot bound (_inflight_shadow_tasks max 16) is acknowledged and acceptable.

The spend exclusion in db_spend_update_writer.py is clean — is_internal_call suppresses api_requests, successful_requests, failed_requests, and autorouter_savings_spend in one place with a clear comment.

The one thing keeping this from 5/5 is what the PR itself flags: the staging pass with live providers is still owed before merge. The live proof ran against a stub upstream, so real routing-decision write-back into shadow_metadata, frontier-model judge JSON structure, and end-to-end spend attribution through a real Prisma instance haven't been exercised with actual credentials. That's not a design problem — the design is sound — but it's a gap that the PR itself treats as pre-merge work.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadoweval3_backend (d98b0fe) with litellm_internal_staging (8bfb777)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (89217c4) during the generation of this report, so 8bfb777 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

All open findings addressed in one round: turn budget via a per-generation starts counter, internal sub-calls excluded from TPM accounting, types sync regenerated. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4 / 5

Here's where I land after reading all three addressed findings:

Turn budget via per-generation starts counter — correct.
_job_starts[job.id] is incremented before create_task (so a task that errors still consumes budget within the generation), and self._job_starts = {} is the rebind on cache refill (which absorbs the written attempt rows and resets the counter). The test pair test_completed_pipelines_hold_turn_budget_within_a_cache_generation and the budget-consumed-by-started-tasks parametrized case together close both sides of the race. The "never decremented within a generation" invariant is exactly right.

TPM exclusion for internal sub-calls — correct.
_build_success_event_pipeline_operations now checks INTERNAL_CALL_ORIGIN_METADATA_KEY and returns [] before touching any rate-limit counters. Combined with the db_spend_update_writer.py change (api_requests=0 if is_internal_call else 1), shadow and judge spend bills to the key's budget without inflating request-volume or autorouter savings figures. Clean.

Types sync — present.
ui/litellm-dashboard/src/lib/http/schema.d.ts regenerated.

What keeps this from 5/5:
The PR explicitly calls out that the live-provider staging test is owed before merge. The proof-of-concept ran against a stub upstream (every local credential out of credit), which means the spend attribution, api_requests=0 isolation, and autorouter session exclusion haven't been verified end-to-end against real provider responses. That's not a design flaw — the design is correct — but it's a gap in the pre-merge evidence.

Everything else is solid: the append-only schema, the partial unique index in the migration (correctly in raw SQL since Prisma can't express it), _sweep_finished_jobs_sql freeing expired jobs before the unique check, sanitized_forwardable_call_metadata correctly restricting what a detached sub-call inherits (no budget reservation, no routing decision, no parent request state), the _is_unique_violation fallback for the concurrent-start race, and the DB-fault-doesn't-cache path in _active_jobs. A staging run with live providers is the remaining bar before this is merge-ready.

[tin-berri]

Judge sampling-params finding fixed at the shared owner, covers the llm_as_a_judge guardrail too. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

The sampling-params fix is clearly visible in llm_judge.py: drop_params=True is now passed in both dispatch branches (router path and SDK path), and the docstring explicitly documents the intent. That's the right fix — it's clean, applies symmetrically, and propagates to the llm_as_a_judge guardrail for free since they share the module.

Score: 4/5

Why not 5:

• The PR description itself flags that the live proof used a stub upstream and a staging pass with live providers is owed before merge. That's a meaningful gap for a feature that touches spend attribution, the rate limiter's TPM counters, and logging hooks — the interaction surface where things tend to behave differently with real provider credentials, real latency variance, and real budget checks.
• The budget gate being "advisory by design" is defensible (auth already rejects over-budget real traffic), but it means a burst of sampled shadow calls can still bill to an over-budget key mid-flight. The PR acknowledges this; it's not a blocker, but it's a documented rough edge.

Why 4 and not lower:

• The derived-state design is solid — one mutable column, append-only attempt rows, all aggregates computed at read time eliminates the cross-pod consistency and stop-race problems that blocked prior attempts.
• The skip chain in ShadowEvalLogger (internal-origin, redaction, sampling hash, call_type allowlist, self-shadow, turn budget, in-flight cap) is comprehensive and fail-closed in the right direction.
• The partial unique index (stopped_at IS NULL) enforcing one active job per key is the correct primitive — no application-level locking needed.
• The drop_params=True fix at the shared owner is correct and covers both dispatch paths.
• The exclusion of shadow/judge calls from api_requests, auto-router adoption metrics, and v3 rate limiter TPM is implemented at the right layers.

Ready to merge once the staging pass with live providers completes.

[veria-ai]

Medium: Internal model calls bypass TPM limits

An authenticated user can repeatedly invoke an LLM-based auto-router while its classifier calls consume provider tokens without incrementing any TPM counters because those calls carry the autorouter_classifier origin. Keep internal calls out of request-volume analytics, but reconcile their billed tokens against the classifier, shadow, or judge model's TPM counter instead of returning no operations here.

[tin-berri]

Intentional boundary: v3 scopes are caller quotas, and charging internal calls back recreates the eval-throttles-user bug this PR fixes. Deployment protection stays with the router's own rate limits

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d98b0fe. Configure here.}

[mateo-berri]

LGTM. Thanks Tin!

Noticing that the PR description mentions stubbing

[mateo-berri]

I see that you say "Live proof used a stubli upstream; a staging pass with live providers is owed before merge". Can you do a full live e2e QA?

[mateo-berri]

Thanks for adding e2e proof!

LGTM