feat(ui): shadow evals tab beside auto-router usage

[tin-berri]

TLDR

Problem this solves:

• The shadow eval backend is API-only; admins need a place to start an eval, watch verdicts accumulate, and read win rates without curling endpoints

How it solves it:

• Adds Usage and Shadow Evals sub-tabs to Cost Optimization -> Auto-Router; shadow evals render independently of the benchmarks body so they work for pre-adoption keys with no router sessions yet
• Built from a pre-written state matrix covering every cell earlier review rounds found: list and detail queries each render explicit loading, error, and data states, every active job gets its own self-polling card (one per key can run concurrently), the newest finished job is showcased when none is active, the start form is always offered to admins and hidden for view-only sessions, and cards render from list rows until their detail arrives

User Flow

1. Admin opens Cost Optimization -> Auto-Router and picks the Shadow Evals sub-tab; the Usage sub-tab keeps the benchmarks view
2. Picks a key (paged picker), an auto-router, a sampling percentage, a duration, a turn budget, and a judge model, then starts; the card polls every 15s while running, and the job list polls on the same cadence while any running job is visible so finished jobs settle without a manual refresh
3. The card shows "X of max_turns turns judged, E errored, $Y judge spend" plus the matched-or-beat headline, verdict bar, and one shared table for both stratifications; Stop halts sampling and older evaluations stay expandable in the previous list

Relevant issues

• UI half of the shadow eval rebuild; the backend merged in feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state) #36587, and this supersedes the UI half of feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates) #36250 and feat(ui): shadow eval section on the auto-router usage tab #36572

Changes

• New ShadowEvalSection.tsx, useShadowEval.ts, and their tests under cost-optimization/_components; AutoRouterBenchmarksTab splits into Usage and Shadow Evals sub-tabs, lazy mounted and kept mounted after first visit like the parent view's tabs
• The shared PaginatedSearchSelect gains an optional errorText, and the key picker uses it to surface a failed key list instead of posing as no matching keys
• Reads via $api.useQuery against generated types; mutations via fetchClient plus useMutation with NotificationsManager, the repo's established shape; a new useInfiniteKeys hook pages /key/list for the picker
• Client-side gates mirror the API bounds (percentage 0.1 to 100, turn budget 1 to 2000) with inline hints; both queries retry once and never render an error as an empty state

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Verified in a live browser against a proxy running the backend branch: the section rendered a stopped job's card from live data, and expanding the previous evaluation fetched its detail on demand, showing 10 judged, $0.0010 judge spend, and the 70 percent matched-or-beat headline with both stratification tables. To reproduce for screenshots: run the backend branch proxy with an auto-router, NEXT_PUBLIC_BASE_URL=http://localhost:4000 npm run dev, open http://localhost:3000/?page=cost-optimization, Auto-Router tab, Shadow Evals sub-tab

Component tests: workflow tests over the section's state matrix plus both polling helpers; vitest on the cost-optimization tree is green (139 tests), eslint and prettier clean, npm run build passes

Type

🆕 New Feature

Caveats (if any)

• None

QA runbook

Follow the User Flow above; additionally verify a view-only admin sees the cards but no start form or stop button, a non-admin sees no section, and the Usage sub-tab still renders every benchmarks state on its own

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
New UI can start shadow evals that sample real key traffic and bill judge spend, so misconfiguration has cost impact. Risk is moderated by admin/view-only gates and client-side input bounds.

Overview
Adds a Shadow eval section to Cost Optimization → Auto-Router so admins can start and monitor live quality comparisons without using the API directly.

Admins pick a key, auto-router, sample %, duration, turn budget, and judge model, then watch win/tie rates accumulate on polling job cards. Active jobs can be stopped; finished ones stay expandable under previous evaluations. The section sits above the benchmarks body so it still works when usage data is empty or errored, and start/stop controls are hidden for view-only sessions (section hidden on 403).

Also adds useShadowEval query/mutation hooks with 15s polling while jobs run, plus useInfiniteKeys for the paged key picker.

^{Reviewed by Cursor Bugbot for commit ee238b9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The implementation is solid overall — clean separation of concerns, good type safety via generated schema types, proper RBAC gates, and thorough test coverage of the full state matrix. Here's what earns it a 4 rather than a 5:

What's well done:

• 19 tests covering every state cell described in the PR (error, loading, view-only, active/finished/showcased, stop flow, start form validation)
• useShadowEval.ts follows the repo's established $api.useQuery + fetchClient + useMutation + NotificationsManager pattern exactly
• 403 short-circuits to null cleanly, no flash of broken UI
• Lazy detail loading in PreviousJob (only fetches when expanded) is exactly the right call
• detail ?? job fallback renders the card from the list row while the detail is in flight

What holds it to 4:

1. List query doesn't poll. useShadowEvalJobs has no refetchInterval. Detail cards poll every 15s, so the card content stays current, but the list-level state (which drives active / finished / showcased / listed) only refreshes on mutations. If a job completes between the initial load and the user's next interaction, the "Previous evaluations" count and the start-form offer won't update until a stop/start fires invalidateShadowEval. Adding refetchInterval to the list query mirroring the detail's logic (poll while any job is active) would close this.

2. accessToken sourcing is inconsistent. useInfiniteKeys reads it from useAuthorized() internally, but useShadowEvalJobs, useShadowEvalJob, useStartShadowEval, and useStopShadowEval take it as a prop. ShadowEvalSection then threads it through every child. Pick one pattern; the hook-reads-from-context style in useInfiniteKeys is cleaner.

3. useStartShadowEval / useStopShadowEval ignore accessToken entirely — they call fetchClient.POST directly without any auth header, relying on whatever the fetchClient singleton configures. If that's fine for the rest of the repo it's fine here, but it's worth calling out for reviewers unfamiliar with the HTTP layer.

None of these are blockers — the feature works correctly and the tests will catch regressions — but item 1 is the kind of subtle staleness bug that would show up as a UX report ("the section didn't update after my job finished"), so it's worth addressing before merge.

[greptile-apps]

Greptile Summary

Adds an administrative Shadow eval workflow to the Auto-Router usage tab, including paginated key selection, start and stop controls, per-job polling, result summaries, and historical evaluations

• Keeps the section available across benchmark loading, empty, and error states
• Adds generated-client query and mutation hooks for shadow-evaluation endpoints
• Adds workflow tests covering permissions, polling, errors, active jobs, completed results, and form submission
• Adds an infinite-query key-list hook for the paginated picker

Confidence Score: 4/5

The PR appears safe to merge, with one non-blocking immutability cleanup in the router-option derivation

The shadow-evaluation lifecycle, endpoint error propagation, key identifier contract, authorization presentation, and polling behavior align with the backend contracts; only avoidable mutable option-building remains

Files Needing Attention: ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx

Important Files Changed

Filename	Overview
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Implements the complete shadow-evaluation UI and state matrix; router option derivation violates the repository's immutability convention
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/useShadowEval.ts	Adds typed list, detail, start, stop, invalidation, and active-job polling hooks with centralized HTTP error propagation
ui/litellm-dashboard/src/app/(dashboard)/hooks/keys/useKeys.ts	Adds correctly keyed and paginated infinite loading for the key picker
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksTab.tsx	Mounts the shadow-evaluation section outside benchmark-body state branches so it remains available independently
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.test.tsx	Adds substantial coverage for permissions, list and detail states, concurrent jobs, mutations, form payloads, and result rendering

_{Reviews (1): Last reviewed commit: "feat(ui): shadow eval section on the aut..." | Re-trigger Greptile}

[greptile-apps]

Avoid incremental option mutation

The router-option derivation incrementally mutates names, contrary to the repository's immutability convention and adding avoidable mutable local state

Suggested change

	const names = new Set<string>();
	for (const deployment of autoRouters ?? []) {
	if (deployment.model_name) names.add(deployment.model_name);
	}
	return [...names].sort().map((name) => ({ label: name, value: name }));
	const names = new Set(
	(autoRouters ?? []).flatMap((deployment) => (deployment.model_name ? [deployment.model_name] : [])),
	);
	return [...names].sort().map((name) => ({ label: name, value: name }));

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[tin-berri]

Fixed with the suggested immutable construction plus toSorted and a type predicate

[tin-berri]

Both findings addressed, rebased onto the backend tip. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's the breakdown:

What's solid:

• Follows the repo's established mutation shape (fetchClient + useMutation + NotificationsManager) and query shape ($api.useQuery against generated types) consistently
• The state matrix is genuinely complete — every cell (loading/error/data) × (list/detail) is handled explicitly, and the tests cover the tricky ones (null judged_count ≠ zero verdicts, detail-sees-completion-before-list, detail-error-over-collecting-copy)
• shown = detail ?? job means the card always renders the freshest data while gracefully degrading to the list row
• 403 → null, non-403 error → inline message (never an empty state) is correct and tested
• accessToken! is safe because enabled: Boolean(accessToken) guards both infinite and shadow-eval queries
• isActive(shown) — not isActive(job) — controls the Stop button, so a naturally-completed job correctly loses its Stop button even before the list re-fetches

The one gap holding it to 4:

useShadowEvalJobs (the list) has no refetchInterval. When a job completes naturally (not via the Stop button), invalidateShadowEval is never called, so the list entry stays status: "running" until the next manual refresh. The detail will correctly reflect completed and the card will hide the Stop button (good), but the card stays pinned in the showcased/active slot and never migrates into PreviousJobs — the "start a new eval" affordance stays visually buried behind a card that's actually done. A refetchInterval on useShadowEvalJobs matching ACTIVE_POLL_MS when any job in the list is active would close this gap:

export const useShadowEvalJobs = (accessToken: string | null) =>
  $api.useQuery("get", LIST_PATH, {}, {
    enabled: Boolean(accessToken),
    retry: 1,
    refetchInterval: (query) =>
      (query.state.data ?? []).some((job) => job.status === "running") ? ACTIVE_POLL_MS : false,
  });

Everything else is clean — the architecture, tests, and UX gates are production-ready. The natural-completion stale-list case is the only real remaining gap.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 52d0d09. Configure here.}

[tin-berri]

Rebased onto the backend tip, no UI changes. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

4/5

The implementation is solid overall. Here's the breakdown:

What's well done:

• State matrix coverage in tests is thorough — loading/error/data for both list and detail, active vs. finished job showcasing, view-only gating, 403 silencing, the race where polled detail sees completion before the list does
• shadowEvalPollMs returning false to halt polling on terminal states is clean
• Rendering from the list row while the detail loads (fallback detail ?? job) correctly prevents a blank card on first render
• error instanceof ApiError && error.status === 403 returning null before the section renders is the right auth gate
• useInfiniteKeys follows the existing useKeys pattern cleanly

The one notable gap — form doesn't reset on success:

After start.mutate(startBody) succeeds, onSuccess invalidates the queries but StartForm stays mounted with the previous field values. If the mutation completes and a new job card appears above, the user sees the form still showing the old key/router/judge selections, which looks like the submit didn't take. A reset to defaults on onSuccess would close this — the reset belongs in StartForm via a callback prop or by keying the form on a submit counter.

Minor:

• (v: string | null) in the Select onValueChange handler is wider than shadcn/ui's actual (value: string) => void signature; harmless but slightly misleading
• useStopShadowEval() is instantiated per JobCard, so stop.isPending is local to each card — correct for concurrent jobs, just worth noting it's intentional

The deduction is primarily for the missing form reset, which is a visible UX issue on first use. Everything else holds up well.

[tin-berri]

List staleness fixed, rebased onto the backend tip. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4.5 / 5

Why:

Strengths that push it up:

• State matrix coverage is genuinely complete: loading, error (including 403 → null return), running, completed, stopped, view-only, list-rows-before-detail, detail-error, concurrent active jobs — each has a test that actually asserts the right thing
• Component decomposition is clean and appropriately scoped (JobCard → JobResults → ResultsBody + VerdictBar + SliceTable); nothing in the tree is doing too much
• Hook shapes match the repo's established pattern: $api.useQuery for reads, fetchClient + useMutation + NotificationsManager for mutations
• Polling logic is correct: refetchInterval on the detail query gates on the detail's own status, while the list query polls only while any job is active — so finished jobs settle without a manual refresh
• useInfiniteKeys is minimal and follows existing useKeys conventions without duplicating the network layer

Minor nits keeping it from 5:

1. Duration Select type annotation is wrong. onValueChange={(v: string | null) => setDurationDays(v ?? "7")} — Radix/shadcn's Select.onValueChange signature is (value: string) => void; null is never passed. The ?? "7" fallback is dead code and the | null annotation is incorrect. Harmless at runtime but misleading.

2. toSorted() is ES2023. Used in useJudgeModelOptions and routerOptions. Works in Node 20+ and all modern browsers, so likely fine for this Next.js app — worth confirming the project's tsconfig lib target covers it, otherwise .slice().sort() is safer.

3. showcased.includes(job) uses reference equality, which works correctly here since both showcased and finished are derived from the same jobs ?? [] array in a single render. Subtly fragile if that changes (e.g. a selector that remaps objects), but not a bug today.

None of these are blocking. Implementation is solid and ready for a maintainer look pending the backend PR merge.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f6f290f. Configure here.}

[tin-berri]

Rebased onto the backend tip, no UI changes. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Solid implementation that follows the repo's established patterns well. Here's the breakdown:

What's done well:

• State matrix coverage is thorough — all 16 tests target distinct states (loading, error, 403, active/inactive, multiple concurrent jobs, view-only, race conditions, null field handling)
• Polling is correctly gated: the list only polls while any job is running (shadowEvalListPollMs), and individual detail queries self-stop once the job completes
• The 403 → null render correctly gates non-admins with zero flicker
• ShadowEvalSection is mounted outside BenchmarksBody's early returns, so it renders even when there's no benchmark data — architecturally correct
• Uses generated schema types (components["schemas"]["ShadowEvalJob"] etc.), $api.useQuery, and useMutation + NotificationsManager — all the repo's established shapes
• Client-side validation bounds (0.1–100%, 1–2000 turns) are tied to constants, not magic numbers scattered inline

Minor issues holding back a 5:

1. useInfiniteKeys missing staleTime — useKeys sets staleTime: 30000 and placeholderData: keepPreviousData; the new useInfiniteKeys omits both. Not a functional bug, but it will cause more re-fetches and UI flicker in the key picker than the rest of the key-list UI.

2. accessToken threading is inconsistent — ShadowEvalSection accepts an accessToken prop and threads it to the shadow eval hooks (where it's only used for the enabled guard, not added to headers). Meanwhile useInfiniteKeys and useAutoRouters read the token from context internally. Both patterns exist elsewhere in the codebase, but mixing them in the same component is worth cleaning up if the section grows.

3. showcased.includes(job) uses reference equality — works correctly today because listed is derived from the same jobs array slice, but it's a subtle invariant that could silently break if the data pipeline ever normalizes/reconstructs objects. job.job_id comparison would be more explicit.

None of these are blockers given this is a dashboard-only feature gated behind admin roles. The implementation is production-ready pending the backend PR merge.

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's the breakdown:

---

What's strong:

• Comprehensive test coverage (370-line test file) covering all the edge states from the state matrix: loading/error/data on both list and detail, auth gates, polling, stop-while-running, error vs. empty distinction, and the "null count ≠ zero count" case that previous rounds flagged
• keepMounted with lazy-visit tracking is correct — TabsContent wraps @base-ui/react/tabs Panel, which does accept keepMounted, and the approach avoids re-mounting while staying lazy until first visit
• 403 → null early return is clean; the section silently disappears for non-admins rather than showing an error card
• Polling architecture is right: detail cards poll individually based on their own status, and the list polls only while any job is running, so both settle without manual refresh
• showcased/listed partition handles all combinations (active jobs vs. none, newest finished job featured, rest collapsible) correctly per the tests

---

What holds it to 4:

1. handleTabChange(value: unknown) — Base UI's Tabs.Root.onValueChange types as (value: string | null) => void, not unknown. The guard is harmless but it's a type mismatch symptom; passing string | null directly would be accurate and avoid the defensive branch.

2. pageParam as number in useInfiniteKeys — pageParam is typed as unknown (TanStack Query's generic default), so the cast is silent. The initial page param is always 1, so it works, but the type could be parameterised: useInfiniteQuery<KeysResponse, Error, InfiniteData<KeysResponse>, QueryKey, number> to get proper typing through.

3. useInfiniteKeys query key excludes accessToken — consistent with useKeys, so it's a pre-existing pattern rather than a new regression, but it means a token rotation mid-session won't bust the cache. Worth noting since this hook is new.

4. startBody built at render time, not in the click handler — not a bug (it closes over the current render's values and the button is disabled while !valid), but it means every render allocates the object. Constructing it inside the onClick callback would be idiomatic.

None of these are blocking — the feature is well-scoped, the auth and billing-cost guards are in place, and the test matrix is thorough. Minor typing and pattern cleanup would push this to 5.

[tin-berri]

Fixed at the shared component: PaginatedSearchSelect gains errorText, the key picker passes its query error, regression test added

[tin-berri]

Installed base-ui types tab values as any, all three sibling hooks cast pageParam, and startBody now builds in the click handler. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

All three issues from the previous round are resolved cleanly:

• handleTabChange now takes value: unknown with a proper typeof guard — no cast, no any
• pageParam is annotated as unknown in the queryFn signature and cast at the call site, which is the standard pattern for useInfiniteQuery when the TPageParam isn't threaded through all five generic slots; initialPageParam: 1 guarantees it's always a number so the cast is safe
• startBody is built inside handleStart, so it always reads the current state values at click time

What's keeping it at 4 rather than 5:

1. onValueChange annotation on the Duration <Select> — you have (v: string | null) but shadcn's Select.onValueChange is (value: string) => void; the null branch and the ?? "7" fallback are dead code. It won't break anything, but it suggests a stale copy from an earlier draft.

2. pageParam as number could be eliminated by fully parameterising the query: useInfiniteQuery<KeysResponse, Error, InfiniteData<KeysResponse>, QueryKey, number>(...). With that fifth generic the queryFn receives { pageParam: number } and the cast disappears. Minor, and the current form is safe given initialPageParam: 1.

Everything else — the 403 guard, the polled showcased/listed split, the keepMounted/visited-tab pattern, and the 25-case test matrix — is solid. Fix the dead string | null annotation and the score is a clean 5.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 94f0ccb. Configure here.}

[tin-berri]

Select values are nullable in the installed base-ui typings (nine matching annotations exist), pageParam is now fully parameterised, and the twin mutations share one helper

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

Here's why:

What's strong:

• Test coverage is exceptional — 22 cases covering loading, error, 403, view-only, polling state transitions, the "list row before detail arrives" race, and the "detail finishes before list" race. This is exactly the kind of state matrix work that prevents regressions.
• Error states are never disguised as empty states — the errorText prop on PaginatedSearchSelect and the explicit error line in ShadowEvalSection follow the pattern from earlier review rounds correctly.
• Polling design is clean — shadowEvalPollMs / shadowEvalListPollMs as pure functions tested in isolation, refetchInterval fed from live query state, and detail queries stop polling once the job settles.
• Security gates are correct — 403 returns null (section hidden), view-only hides the start form and stop button, non-admin sees nothing.
• keepMounted via visitedTabs avoids double-fetch on tab switch without keeping the Shadow Evals tree alive until it's first visited — that's the right tradeoff.
• Types flow from generated schema — ShadowEvalJob, ShadowEvalSlice, StartShadowEvalRequest all come from components["schemas"], so drift is caught at compile time.

What holds it from 5/5:

1. handleTabChange accepts unknown instead of string. Radix onValueChange is typed (value: string) => void, so this is unnecessarily defensive:

  const handleTabChange = (value: string) => {
    setVisitedTabs((currentTabs) => (currentTabs.includes(value) ? currentTabs : [...currentTabs, value]));
  };

2. JobCard calls useStopShadowEval() once per mounted card, so each running job gets its own mutation instance. If two jobs are running simultaneously, each card's stop.isPending is independent — this is actually correct behavior, just worth being explicit that it's intentional since the PR mentions concurrent per-key jobs.

3. useInfiniteKeys reads accessToken internally via useAuthorized() while useShadowEvalJobs / useShadowEvalJob take it as a parameter. Not a bug, but the asymmetry makes the calling pattern slightly inconsistent within the same feature.

None of these are blockers. The implementation is solid, the test suite is the best part of this PR, and the component boundaries are clean. Merge-ready pending the maintainer's pass.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 83fbe6b. Configure here.}

[tin-berri]

Shadow eval hooks now read auth via useAuthorized like the hooks directory, removing the accessToken prop threading

[tin-berri]

Tab values are any in installed base-ui typings, the guard mirrors CostOptimizationView. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

That clarification is correct and resolves the concern. If base-ui types onValueChange as (value: any) => void, annotating the handler parameter as unknown and narrowing with typeof value !== "string" is strictly more type-safe than accepting the any directly, and mirroring CostOptimizationView is the right consistency call.

Score: 4/5

The implementation is solid across the board — comprehensive state matrix tests (139 passing), explicit loading/error/data states in every query, correct 15 s polling tied to job status, admin/view-only gates, and the keepMounted lazy-mount pattern is idiomatic for this tab structure. The useShadowEvalMutation abstraction is clean and invalidateShadowEval invalidates both keys to keep list and detail coherent after mutations.

The one remaining note keeping it from 5/5 is minor: ShadowEvalSection calls useAuthorized() twice — once at the top of the section and once inside StartForm — so any token rotation or role change would need to propagate through two separate hook instances rather than one. It works correctly today since both reads are reactive, but passing accessToken / isViewOnly as props would make the data flow explicit. That said, this is a style/consistency point, not a correctness issue given the hook is stable per render.

[tin-berri]

useAuthorized has 131 call sites reading one reactive store; passing it as props would reinstate the threading the previous round asked to remove

[tin-berri]

@greptileai can you reply here with a score and why that score?

[greptile-apps]

Score: 4/5

This is a well-executed PR. Here's the breakdown:

What's strong:

• Tests genuinely cover the state matrix — every meaningful cell (loading, error, empty, active, finished, view-only, 403, detail-load-error before list, etc.) gets its own it block. The intent is to prevent silent state collapses, and the tests enforce that.
• Polling stops correctly: shadowEvalPollMs and shadowEvalListPollMs return false for terminal states, so no unnecessary polling after jobs finish.
• The 403 → return null pattern is the right shape for section-level gating; it avoids cluttering the parent with permission checks.
• errorText extension on PaginatedSearchSelect is cleanly backwards-compatible and addresses the real UX problem (a failed key list silently posing as empty results).
• useShadowEvalMutation DRYs the invalidation and error notification pattern correctly.
• Types are derived from the OpenAPI schema (components["schemas"]), not hand-written — drift is structurally prevented.

What holds it to 4:

1. useInfiniteKeys has no tests of its own. The hook has non-trivial logic — getNextPageParam, the options passthrough, the staleTime — and none of it is exercised. ShadowEvalSection.test.tsx mocks the hook away entirely, so a bug in the pagination logic would go undetected.

2. JobCard creates a new useStopShadowEval instance per card. With multiple active jobs, each card holds its own mutation state. That's fine for isolation, but stop.isPending on one card won't block the other's Stop button, so a user can fire simultaneous stops on different jobs and both mutations race to invalidate. The backend may handle this gracefully, but the UI gives no indication that one stop is already in flight.

3. handleTabChange types value as unknown when shadcn/ui's Tabs.onValueChange is typed as (value: string) => void. The defensive check is harmless but signals the prop type wasn't consulted.

4. Minor: "judge calls bill to the shadowed key" is mentioned in the form description but there's no inline hint about cost magnitude, only the turn-budget cap. An admin misconfiguring 100% sampling for 30 days on a high-traffic key could accumulate meaningful judge spend before noticing. Not a blocking issue, but worth a follow-up UX consideration.

Overall this is clean, well-tested UI work — the state-matrix discipline in the tests is notably above average for frontend PRs.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 84ae02e. Configure here.}