feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates)

[tin-berri]

What

Pre-adoption shadow evaluation for the auto-router: answer "is it safe to turn the router on?" with evidence from the customer's own live traffic, before any production traffic is routed.

• A sampled slice (configurable %) of a key's successful requests is duplicated through the auto-router in a detached background task, so there is zero added latency and shadow responses are never served to users
• An LLM judge compares the real response vs. the router's pick blind, with A/B labels randomized to cancel position bias
• Verdicts are stratified by the router's own tier classification (SIMPLE / COMPLEX / REASONING), so the result reads: "router's pick judged good-or-better 84% overall, weakest on REASONING at 58%"

How

Backend

• LiteLLM_ShadowEvalJob + LiteLLM_ShadowEvalVerdict tables, plus migration 20260807000000_add_shadow_eval_job; verdict rows record ids and completion-token counts for BOTH arms, so verdicts can later be checked for judge verbosity bias and joined to the shadow call's spend log
• ShadowEvalLogger (litellm/integrations/shadow_eval_logger.py): hooks async_log_success_event; the hook is one dict lookup against a job snapshot and never touches the DB; deterministic hash-based sampling, so the same request samples the same way across pods; caps in-flight shadow pipelines at 16 per pod by shedding rather than queueing, so a traffic spike can't build an unbounded task backlog; skips any request stamped internal_call_origin, so it never recurses on its own shadow/judge calls and never samples the auto-router's classifier sub-calls as if a user sent them
• Job lifecycle runs on a 10s loop owned and started by logger registration itself (no separate startup step to reorder), not on the request path: counter flushes land without waiting for another request, request_count freezes when a job stops (status-guarded writes), and a job whose window or judge-spend cap has passed is completed even if its key goes quiet
• The judge dispatches through the proxy's router when judge_model is a configured deployment (DB-stored credentials work) and through the SDK for provider-qualified public names; start rejects a judge name neither path can resolve (400) instead of accepting a job that fails every turn, and the most recent shadow/judge failure is persisted as last_error on the job and shown in the UI
• Internal sub-calls (shadow, judge, classifier) keep billing spend and tokens to the shadowed key, but no longer count as api_requests in the daily rollups, no longer accrue autorouter_savings_spend, and are excluded from the LiteLLM_AutoRouterSession benchmarks rollup, so a running eval cannot inflate the adoption metrics or its own next cost estimate
• Endpoints in auto_router_endpoints.py:
  • POST /auto_router/shadow_eval/start: validates the router is a configured pre-routing strategy and the judge model is resolvable (a deployment prices by its underlying provider model), rejects a second active job per key (409), returns an upfront judge-cost estimate priced from the key's trailing 7-day user-request volume
  • GET /auto_router/shadow_eval/{job_id}: counters and per-tier win rates (SQL aggregate over verdicts)
  • GET /auto_router/shadow_eval: list jobs
  • POST /auto_router/shadow_eval/{job_id}/stop: halt sampling (~10s propagation), keep verdicts
• Start and stop require PROXY_ADMIN; the read endpoints also allow PROXY_ADMIN_VIEW_ONLY
• Logger registered in cost_tracking(), which already owns the same prisma dependency

UI (Cost Optimization -> Auto-Router Usage)

• ShadowEvalSection: start form (key, router, %, judge model, upfront cost estimate warning), live-polling job card, per-tier results table with low-sample warnings, running judge spend
• useShadowEval.ts hooks; OpenAPI schema.d.ts regenerated

Scope / known limitations

• Turn-level only. The shadow never influences the next real turn, so multi-turn compounding effects are out of scope for v1
• Judge quality gate is a single verdict per turn, no multi-judge panel yet
• Counters are buffered up to one 10s lifecycle tick, so displayed counts can lag briefly; stopping a job freezes request_count (up to a tick of tail counts on other pods is dropped rather than written after stop)
• Judge cost estimate uses a flat 4k-prompt/200-output token assumption when the judge model is in the price map, with a $0.01/call fallback otherwise

Test plan

• 76 unit tests in tests/test_litellm/integrations/test_shadow_eval_logger.py covering deterministic sampling, A/B unmask mapping, fence-tolerant verdict JSON parsing, success-hook skip paths (including classifier sub-calls), request-parameter forwarding, the in-flight backlog cap, router-first judge dispatch, last_error persistence, and the lifecycle-loop flush/finalize/status-guard behaviors, each mutation-checked against the unfixed code
• Endpoint, rollup, and daily-writer suites cover judge-model validation at the start seam, internal-origin exclusion from LiteLLM_AutoRouterSession, and internal calls keeping spend while not counting as requests
• 20 UI tests in ShadowEvalSection.test.tsx (failure banner included), plus the existing AutoRouterBenchmarksTab suite
• Needs a staging pass with a live key and a configured auto-router before merge. Note for existing local DBs: migration 20260807000000_add_shadow_eval_job was amended in place (adds last_error, drops the never-read result_json), so a DB that already applied it needs a re-resolve or db push

---

Note

High Risk
Background duplicate LLM traffic bills the shadowed key and touches spend rollups, budget checks, and multi-pod job lifecycle; mis-attribution or metric inflation would affect billing and adoption dashboards.

Overview
Adds pre-adoption shadow evaluation so admins can sample a virtual key’s live traffic, replay it through an auto-router in the background, and compare answers with a blind LLM judge—without serving shadow responses to users.

Data & control plane: New LiteLLM_ShadowEvalJob / LiteLLM_ShadowEvalVerdict schema and migration, including a partial unique index so only one active job exists per key across pods. Admin endpoints under /auto_router/shadow_eval start jobs (with cost estimate from daily rollup volume), list/get aggregated win rates by tier and incumbent model, and stop jobs.

Runtime: ShadowEvalLogger registers with cost_tracking(), samples deterministically on successful completions, runs shadow + judge in detached tasks (concurrency cap, budget and per-job spend/time stops), and uses a 10s lifecycle loop for job snapshots, counter flushes, and auto-completion. Shared internal_call_metadata helpers stamp shadow/judge/classifier sub-calls; daily spend rollups and auto-router session benchmarks exclude those calls from request/savings metrics while still billing spend to the shadowed key.

UI: Cost Optimization’s auto-router tab gains ShadowEvalSection (start form, live results, jump link from benchmarks).

^{Reviewed by Cursor Bugbot for commit e51008f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR adds pre-adoption auto-router shadow evaluation across persistence, management APIs, detached shadow/judge execution, spend attribution, and the dashboard.

• Adds shadow-evaluation jobs, verdict aggregation, lifecycle endpoints, and Prisma schema support.
• Adds sampled background router comparisons with blind judging and per-tier results.
• Adds dashboard controls and live result polling.
• The success callback still performs an inline spend lookup that can reach the database.

Confidence Score: 4/5

The PR should not merge until the sampled success callback no longer performs a spend lookup that can reach the database inline.

The active-job lookup was successfully moved to a detached snapshot refresh, but sampled requests still await key and team spend checks whose fallback path can query the database from the success callback.

Files Needing Attention: litellm/integrations/shadow_eval_logger.py

Important Files Changed

Filename	Overview
litellm/integrations/shadow_eval_logger.py	Implements the detached evaluation pipeline and snapshot lookup, but retains an inline spend check that can reach the database from the success callback.
litellm/litellm_core_utils/internal_call_metadata.py	Adds sanitized attribution metadata for internal router and judge calls while removing parent budget reservations.
litellm/proxy/management_endpoints/auto_router_endpoints.py	Adds authorized shadow-evaluation lifecycle and result endpoints with cost estimation and aggregate reporting.
litellm-proxy-extras/litellm_proxy_extras/migrations/20260807000000_add_shadow_eval_job/migration.sql	Adds job and verdict persistence, including a partial unique index enforcing one active job per key.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksTab.tsx	Correctly renders the shadow-evaluation section outside benchmark loading, error, and empty-state returns.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Adds the start form, running-job cards, polling results, tier summaries, and spend presentation.

_{Reviews (4): Last reviewed commit: "feat(ui): lead shadow eval results with ..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 4 · PR risk: 0/10

[tin-berri]

@BugBot

[codecov]

Codecov Report

❌ Patch coverage is 93.00000% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/shadow_eval_logger.py	91.22%	28 Missing ⚠️
...roxy/management_endpoints/auto_router_endpoints.py	92.30%	14 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

Live-proxy test: backend flow works. UI unreachable (accessToken undefined; section hidden with zero sessions), judge default lacks provider prefix, cost_actual stays null, complexity routers rejected.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing shadow-eval-pre-adoption (e51008f) with litellm_internal_staging (b0fac57)}

[tin-berri]

All four are fixed now. The first three landed before your run; cost_actual was real, the column now defaults to 0 so increments accumulate.

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 3.5 / 5

---

What works well

Architecture is sound. The core design decisions hold up under scrutiny:

• The recursion guard (INTERNAL_CALL_ORIGIN_METADATA_KEY) reliably breaks the shadow→judge→shadow cycle
• Lifecycle management (counter flushes, snapshot refreshes, job finalization) runs entirely off the request path — the success hook is one dict lookup
• Deterministic hash-based sampling gives cross-pod consistency without coordination
• The spend cap (1.5× estimate, $1 floor) and in-flight concurrency cap (16) provide two independent runaway-spend mitigations
• Budget soft-check before shadow dispatch (_key_or_team_is_over_budget) closes the common case without needing a reservation
• sanitized_forwardable_call_metadata correctly strips user_api_key_budget_reservation to prevent the double-finalization bug on parent spend
• Internal call exclusion from api_requests / autorouter_savings_spend / LiteLLM_AutoRouterSession is applied consistently across all three write paths

---

Real concerns

1. Logger registration is not idempotent (proxy_server.py)

_register_shadow_eval_logger() unconditionally creates a new ShadowEvalLogger() and registers it every time cost_tracking() is called. If cost_tracking() is called more than once (config reload, test setUp, etc.), the second instance:

• handles async_log_success_event independently with a stale/empty _jobs_by_key
• has its own _pending_seen that no lifecycle loop ever drains (unbounded growth)
• has its own _inflight_shadow_tasks counter, making the pod-level cap per-instance rather than global

Fix: guard with an isinstance check before registering, similar to what _start_shadow_eval_lifecycle_loop already does:

def _register_shadow_eval_logger() -> None:
    from litellm.integrations.shadow_eval_logger import ShadowEvalLogger
    if any(isinstance(cb, ShadowEvalLogger) for cb in litellm.callbacks):
        return
    litellm.logging_callback_manager.add_litellm_callback(ShadowEvalLogger())

2. _start_shadow_eval_lifecycle_loop() and _register_shadow_eval_logger() have an implicit ordering dependency

_start_shadow_eval_lifecycle_loop is called at startup and searches litellm.callbacks for an already-registered ShadowEvalLogger. It only works if cost_tracking() (which calls _register_shadow_eval_logger) has already run at that point. Nothing in the code enforces this ordering — a future refactor that reorders startup steps silently breaks the lifecycle loop: no error, jobs just never finalize and counters never flush. The lifecycle loop start should live inside _register_shadow_eval_logger itself, called once after registration, rather than being a separate startup step that depends on registration having happened first.

3. Unused schema fields

LiteLLM_ShadowEvalVerdict.shadow_request_id and real_response_tokens are declared in the schema but never written in _run_shadow_eval's prisma.db.litellm_shadowevalverdict.create(data={...}). Dead columns in the migration from day one — either populate them or drop them before the schema is committed.

4. PostgreSQL-specific raw SQL without annotation

::bigint, ::int, ::float casts in _ESTIMATE_VOLUME_SQL and _VERDICT_AGG_SQL are PostgreSQL-only syntax. If litellm's DB support is PostgreSQL-only this is fine, but it should be noted since query_raw bypasses Prisma's portability layer.

---

Summary

The feature is thoughtfully designed and the critical-path safety properties (no latency added, no recursion, no unbounded backlog) are correct. The 3.5 reflects that the registration/lifecycle wiring has a real correctness bug (non-idempotent registration) and a fragile startup dependency that will silently break if startup order ever changes — both fixable in a few lines before merge.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ec32f70. Configure here.}

[tin-berri]

Addressed the review in 59ae51c. Finding 2: registration now owns the lifecycle loop start and the separate startup step is deleted, so no ordering exists to break. Finding 3: shadow_request_id and real_response_tokens are now written on every verdict. Finding 1 does not reproduce: the callback manager dedups CustomLogger instances by class (_add_custom_logger_to_list), so a second cost_tracking() discards the new instance; pinned by test_cost_tracking_twice_registers_one_shadow_eval_logger rather than duplicating that dedup at the call site. Finding 4: the proxy schema is PostgreSQL-only and AUTOROUTER_BENCHMARKS_SQL in the same file already uses the same casts, so the raw SQL matches repo practice

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 59ae51c. Configure here.}

[greptile-apps]

Based on a thorough review of the diff, here's my assessment:

---

Score: 4 / 5

This is a well-engineered, substantial feature. Here's what earns that score:

What's done well:

• Zero-latency design is correct — asyncio.create_task in the success hook, work done in a detached background task
• Deterministic, hash-based sampling across pods without coordination is the right approach
• _MAX_CONCURRENT_SHADOW_TASKS = 16 shed-not-queue is correct; it prevents a backlog that never clears under a traffic spike
• Recursion guard via internal_call_origin is solid and covers all sub-call variants (shadow, judge, classifier)
• Spend attribution (shadow/judge spend counts toward key budget but not api_requests / autorouter_savings_spend) is correct and consistent across db_spend_update_writer.py and autorouter_session_rollup.py
• Partial unique index in SQL (correctly calling out that schema.prisma can't express it) is the right place for the one-active-job-per-key guarantee
• Status-guarded update_many for all counter/finalize writes prevents two pods from resurrecting a stopped job
• Test coverage is strong — 76 unit tests including mutation-checked edge cases

Concerns that keep this from 5/5:

1. _run_shadow_eval incurs shadow + judge spend even when prisma is None (checked only after the pipeline runs, for verdict write). If Prisma is unavailable mid-request, both API calls happen but no verdict is stored — spend with no record.

prisma: Final = self._prisma_provider()
# ... budget check uses prisma via get_current_spend ...
shadow: Final = await self._call_router_shadow(...)  # spend incurred
verdict: Final = await self._call_judge(...)          # spend incurred
if prisma is None:  # ← checked too late
    return

The prisma is None guard should move above the shadow/judge dispatch.

2. list_shadow_eval_jobs has a hard take=50 with no pagination. Operators with many historical jobs will silently get a truncated list.

3. The spend cap can overshoot by a full lifecycle tick (up to 10s of concurrent judge calls), acknowledged in comments. For a high-volume key with an underestimated cost, this could be material. Worth flagging for awareness even if the tradeoff is intentional.

4. _register_shadow_eval_logger relies on litellm.callbacks containing the registered instance to call start_lifecycle_loop(). If the callback manager routes async callbacks to a different internal list and litellm.callbacks doesn't include the instance, the lifecycle loop never starts (jobs never finalize, counters never flush) with no error surfaced.

[tin-berri]

Addressed in e51008f. Concern 1: the prisma gate now sits above the shadow and judge dispatch, so no provider spend happens without a place to record the verdict (pinned by a mutation-checked test). Concern 2: the list endpoint takes a bounded limit query param (default 50, max 200). Concern 3 stands as designed: overshoot is bounded by one 10s tick times the 16-task cap, a few cents at judge prices, and the tick already sets stop propagation. Concern 4 does not reproduce: add_litellm_callback appends to litellm.callbacks via _safe_add_callback_to_list, and test_registration_starts_the_lifecycle_loop_on_the_registered_instance fails if that routing ever changes

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e51008f. Configure here.}

[greptile-apps]

Score: 3/5

The core architecture shows careful engineering — zero-latency detached tasks, hash-based deterministic sampling, recursion guard via internal_call_origin, DB-level one-active-job-per-key uniqueness + race-safe 409, spend-cap with floor, and the clean split between forwarded_internal_call_metadata (in-request) and sanitized_forwardable_call_metadata (detached) to prevent double-finalization of budget reservations. The exclusion of internal calls from daily api_requests / autorouter savings rollups is correct and the reasoning is well-documented. These are all genuinely good decisions.

What pulls it down:

1. Judge prompt can reach ~48k chars (correctness/cost bug)
_MAX_JUDGE_CHARS = 16_000 is applied separately to conversation, response_a, and response_b inside _call_judge. The resulting user_prompt is conversation[-16000] + "\n\nResponse A:\n" + response_a[:16000] + "\n\nResponse B:\n" + response_b[:16000], so the prompt alone can approach 50k characters before the system prompt is added. This will overflow the context window of many judge models and produce expensive, wasted calls that land in failed_count with no clear diagnosis.

2. Lifecycle loop start path is fragile
_register_shadow_eval_logger adds a ShadowEvalLogger() via add_litellm_callback, then scans litellm.callbacks to find the registered instance to call start_lifecycle_loop(). If the callback manager stores the canonical instance somewhere other than litellm.callbacks (e.g. it deduplicates before appending), the loop is never started, jobs never finalize, and counters never flush — silently. This warrants a direct reference or a post-registration check.

3. Migration amended in-place
The PR description says "a DB that already applied it needs a re-resolve or db push." In-place amendment of an already-applied migration corrupts Prisma's migration history checksum. This is a production hazard; a new additive migration is the correct path.

4. Staging pass explicitly not done
The PR description states this is required before merge, which means the review is premature on that axis.

5. Judge model validation is shallow
litellm.get_llm_provider(model=judge_model) only validates the provider prefix is known — it does not verify the model exists or credentials are present. A job started with anthropic/nonexistent-model will pass /start and silently fill failed_count with "judge call failed" errors. A resolve-time credential check or at least a warning in the response would close this gap.

The architecture is solid enough that fixing (1) and (3) + completing staging would make this mergeable. As-is it has a real correctness bug in judge prompt sizing and an operational risk in the migration strategy.