feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates) - #36250

Closed

tin-berri wants to merge 34 commits into

litellm_internal_stagingfrom

shadow-eval-pre-adoption

Commits on Aug 8, 2026

feat: pre-adoption shadow eval foundation (schema, types, logger skeleton)

akapur99
and
claude
committed
5618bc9
feat: complete pre-adoption shadow eval (backend + UI)

akapur99
and
claude
committed
a50a391
shadow-eval: fix production issues & add comprehensive tests

akapur99
and
claude
committed
0b96dcc
shadow-eval: address bot review findings

akapur99
and
claude
committed
5f03456
Merge remote-tracking branch 'origin/main' into shadow-eval-pre-adoption
akapur99
committed
13fe386
Merge origin/litellm_internal_staging into shadow-eval-pre-adoption

akapur99
and
claude
committed
2092b80
Merge origin/litellm_internal_staging (catch up on further base moves)

akapur99
and
claude
committed
05aa3d4
Fix CI lint and test failures

akapur99
and
claude
committed
aba922f
fix: default shadow eval cost_actual to 0 so judge spend accumulates
akapur99
committed
1da3bdc
refactor: satisfy strict lint and type-discipline gates in shadow eval
akapur99
committed
ba4b521
fix: widen judge output budget and surface prior shadow eval jobs

akapur99
and
claude
committed
0b94603
fix: attribute shadow eval spend to the shadowed key's budget

akapur99
and
claude
committed
28878e8
Merge remote-tracking branch 'origin/litellm_internal_staging' into shadow-eval-pre-adoption
akapur99
committed
8485a0d
feat: time-bound shadow eval jobs with a real start form

akapur99
and
claude
committed
9f9ae2b
feat: jump-to-shadow-eval button; fix: block shadow/judge calls past budget

akapur99
and
claude
committed
cc61e12
Merge remote-tracking branch 'origin/litellm_internal_staging' into shadow-eval-pre-adoption
akapur99
committed
d302068
fix(ui): judge model field is a real selector, not a pre-filled default

akapur99
and
claude
committed
029be4d
perf: read shadow eval volume estimate from the daily rollup

akapur99
and
claude
committed
7050e0c
perf: cache the whole active shadow-eval job set, not per-key rows

akapur99
and
claude
committed
0fc75c2

Commits on Aug 9, 2026

fix: three review findings — cap bypass, start race, request-path DB read

akapur99
and
claude
committed
43265a5
test: cover shadow eval lifecycle endpoints and judge failure modes

akapur99
and
claude
committed
4d73db0
feat(ui): explain the shadow eval mechanism and clarify its metrics

akapur99
and
claude
committed
94ebbb9
feat(ui): drop the strictly-won hero; one metric plus the verdict bar

akapur99
and
claude
committed
f339938
feat(ui): lead shadow eval results with the models compared against

akapur99
and
claude
committed
b61404e

Commits on Aug 11, 2026