feat(ui): direction picker and reverse-mode display for shadow evals

[tin-berri]

TLDR

Problem this solves:

• feat(shadow_eval): add reverse-direction shadow eval jobs #36865 added reverse shadow evals, but only over the raw API
• The dashboard form pins direction to forward with no controls
• Reverse jobs would render with forward labels, inverting every win rate

How it solves it:

• Start form gains a direction picker and a baseline model picker
• Baseline lists configured model groups first, then the public catalog
• Baseline is required in reverse mode and never sent in forward
• Job cards, verdict bar, and slice tables label arms per direction

User Flow

Before: an operator whose key already runs on an auto-router wants to check the router still beats a strong fixed model, and the dashboard offers no way to ask

1. They open http://localhost:3000/cost-optimization and click the Shadow evals tab
2. The "Start a shadow eval" form shows key, auto-router, sampling, duration, turn budget, and judge fields only; every eval it can start asks the adoption question
3. To run a regression check they must hand-craft a POST to http://localhost:4000/auto_router/shadow_eval/start with "direction": "reverse" and "baseline_model": "..."
4. When that job appears on the page it reads "Shadowing 10% via my-router" with columns "Router wins" and "Current model wins", and the numbers under them are swapped: the router's own wins sit in the "Current model wins" column

After: the same operator starts and reads a reverse eval entirely from the page

1. They open http://localhost:3000/cost-optimization and click the Shadow evals tab
2. The form now leads with a Direction picker; they choose "Regression check: router's picks vs a baseline" and the copy under the title switches to describe re-running the router's traffic against a fixed baseline
3. A "Baseline model" picker appears, leading with the model groups configured on this gateway (their own deployments and aliases) ahead of the public catalog, and the Start button stays disabled until they choose one; forward mode never shows the picker
4. They pick a baseline, start the job, and the card reads "Comparing my-router to openai/gpt-4o on 10% of its traffic"
5. The headline says "Router matched or beat the baseline", the bar legend shows "Router won / Tie / Baseline won", and slice tables show "Router wins" and "Baseline wins" with each number under the arm that earned it

Relevant issues

• Frontend follow-up to feat(shadow_eval): add reverse-direction shadow eval jobs #36865, which added the reverse direction to the API and pinned the dashboard to forward

Linear ticket

Part of LIT-5538

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

UI change, QA runbook to capture screenshots:

1. Run the proxy with a config that defines an auto-router plus a plain model, then npm run dev in ui/litellm-dashboard
2. Open http://localhost:3000/cost-optimization and click the Shadow evals tab
3. In "Start a shadow eval", confirm the Direction field shows "Adoption check: key's traffic vs the router" and no Baseline model field is visible
4. Switch Direction to "Regression check: router's picks vs a baseline"; the description under the title changes and a Baseline model picker appears, listing the gateway's configured model groups on top, with the Start button disabled until it is filled
5. Fill key, auto-router, baseline, and judge, click Start shadow eval, then send a few chat completions through the auto-router with that key
6. The job card should read "Comparing to on 10% of its traffic" and, once verdicts land, the headline "Router matched or beat the baseline" with "Baseline wins" columns

Type

🆕 New Feature

---

Note

^{Cursor Bugbot is generating a summary for commit 0672aa1. Configure here.}

[greptile-apps]

Greptile Summary

The PR adds reverse shadow-evaluation controls and direction-aware result presentation to the cost-optimization dashboard.

• Adds direction and baseline selectors to the start form.
• Remaps job headings, aggregate verdicts, legends, and slice columns according to evaluation direction.
• Adds component tests for reverse submission and result rendering.

Confidence Score: 4/5

The reverse result rendering appears correct, but the baseline option source must be fixed so supported custom models and aliases do not make the new workflow unusable.

The API accepts configured router-resolvable models and aliases, while the new required picker exposes only models in the static public cost map, leaving valid reverse jobs impossible to start from the dashboard.

Files Needing Attention: ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx

Important Files Changed

Filename	Overview
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Adds reverse-mode form and display behavior, but sourcing baselines solely from the public cost map excludes valid configured models and aliases.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.test.tsx	Adds meaningful coverage for reverse submission and arm-label rendering, but uses only a public cost-map model and does not cover configured custom baselines.

_{Reviews (1): Last reviewed commit: "feat(ui): direction picker and reverse-m..." | Re-trigger Greptile}

[greptile-apps]

Baseline picker omits configured models

If the intended baseline is a configured custom model or model-group alias absent from the public cost map, useBaselineModelOptions omits that API-supported identifier, causing the required baseline field to remain empty and preventing the operator from starting the reverse evaluation.

Knowledge Base Used:

• Auto-router Shadow Evaluations
• Admin dashboard (ui/litellm-dashboard)

[tin-berri]

Fixed in e46def9: the picker now lists configured non-auto-router model groups first, then the public catalog.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[tin-berri]

@greptileai can you reply here with score & why

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The implementation is clean and well-tested. Here's the breakdown:

What's done well:

• The direction-aware helper functions (routerWinRate, otherArmWinRate, routerMatchedOrBeatPct) are correctly derived. In reverse mode shadow = baseline and real = router, so flipping the columns is arithmetically correct.
• VerdictBar and routerMatchedOrBeatPct are consistent: the headline percentage includes ties ("matched or beat"), while the bar distinguishes wins from ties — that's the right semantic split.
• usePlainModelGroups reuses autoRouterListKey, so no extra network request; it piggybacks on the existing query cache.
• Test coverage is solid — direction-toggle, baseline gating, label flipping, and selectPlainModelGroups edge cases are all exercised.

Minor issues holding it from 5/5:

1. Misleading fallback constant. usePlainModelGroups returns NO_AUTO_ROUTERS (an empty Set) as its default. Functionally correct, but the name is confusing in that context.

2. selectPlainModelGroups silently drops mixed-name groups. If a model_name fronts both an auto-router deployment and a plain one, it's excluded from baseline candidates entirely. The test documents this but there's no comment in the function explaining the intent — could surprise someone debugging a missing baseline.

3. direction is always sent in the start body for both forward and reverse. That's a silent behavioral change from the previous hardcoded "forward" as const — worth a comment confirming the backend accepts explicit "forward".

None of these are blockers — the feature logic and tests are solid.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0672aa1. Configure here.}