feat(shadow-eval): skip self-shadowed requests - #36321
Conversation
A shadowed key's real traffic can be a mix of models, and per-tier win rates blend those incumbents together: if the router beats gpt-4o but loses to a fine-tune within the same tier, the tier rate hides both facts. The verdict rollup now groups by (tier, real_model) and the results carry a second stratification, by_current_model, listing win rates against each model the key actually uses. The UI renders it as a second table, only when the traffic really was mixed — a single incumbent would just repeat the tier table's totals. Tier confidence is now turn-weighted across the merged rows rather than averaged per row. Requests already served by the router being shadowed are now skipped before sampling: duplicating them compares the router to itself — guaranteed ties, judge spend for zero information. Traffic routed by a *different* auto-router still samples, which is a meaningful router-vs-router comparison. Skipped requests still count toward the job's request_count so the seen/judged ratio stays honest. Co-Authored-By: Claude <noreply@anthropic.com>
Greptile SummaryThe PR adds shadow-evaluation result breakdowns by incumbent model, turn-weighted confidence aggregation, and filtering of requests already served by the evaluated router
Confidence Score: 4/5The PR appears safe to merge after addressing the non-blocking repository-style violations in the new aggregation and tests The changed runtime behavior has no established functional or security failure, but the new accumulator uses prohibited mutation and several changed lines add prohibited comments Files Needing Attention: litellm/proxy/management_endpoints/auto_router_endpoints.py, tests/test_litellm/proxy/management_endpoints/test_auto_router_endpoints.py, ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.test.tsx
|
| Filename | Overview |
|---|---|
| litellm/integrations/shadow_eval_logger.py | Adds a routing-decision guard that skips requests already served by the evaluated router |
| litellm/proxy/management_endpoints/auto_router_endpoints.py | Adds dual tier/model aggregation and weighted confidence calculations, with non-blocking violations of the repository's immutability and comment conventions |
| litellm/types/management_endpoints/auto_router_endpoints.py | Extends the shadow-evaluation response contract with backward-compatible per-model results |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx | Renders per-model results only when multiple incumbent models are represented |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/useShadowEval.ts | Exposes the generated per-model result type to the shadow-evaluation UI |
| ui/litellm-dashboard/src/lib/http/schema.d.ts | Regenerates the OpenAPI declaration for the new per-model response field |
Reviews (1): Last reviewed commit: "feat: per-incumbent-model shadow eval re..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| def absorb(self, row: _VerdictAggRow) -> None: | ||
| self.turns += row.turn_count | ||
| self.real_wins += row.real_wins | ||
| self.shadow_wins += row.shadow_wins | ||
| self.ties += row.ties | ||
| self.confidence_weighted += (row.avg_confidence or 0.0) * row.turn_count |
There was a problem hiding this comment.
_tally_by mutates both its dictionary and _VerdictTally instances, making the aggregation harder to reason about and change safely
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| confidence_weighted: float = 0.0 | ||
|
|
||
| def absorb(self, row: _VerdictAggRow) -> None: | ||
| self.turns += row.turn_count |
There was a problem hiding this comment.
This inline comment and the new backend and UI test comments violate the repository's explicit prohibition on adding source-code comments
Context Used: CLAUDE.md (source)
# Conflicts: # tests/test_litellm/proxy/management_endpoints/test_auto_router_endpoints.py # ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx # ui/litellm-dashboard/tsconfig.tsbuildinfo
tin-berri
changed the title
2 participants
Stacked on #36250. One follow-up from review discussion (the per-incumbent-model results breakdown originally here moved into #36250 itself):
Skip self-shadowed requests
Requests already served by the router being shadowed are skipped before sampling: duplicating them compares the router to itself — guaranteed ties, judge spend for zero information. Traffic routed by a different auto-router still samples (meaningful router-vs-router comparison). Skipped requests still count toward
request_countso the seen/judged ratio stays honest.Tests
Two new logger tests (self-shadowed requests are skipped but counted; different-router traffic still samples); mutation-checked — disabling the guard fails the test.
🤖 Generated with Claude Code