feat(shadow-eval): name the shadowed key in job responses and the UI headline

[tin-berri]

TLDR

Problem this solves:

• Shadow eval cards never say whose traffic they sample
• Job responses carry only a 64-char key hash

How it solves it:

• Responses resolve the hash to key alias and masked name
• Card reads "Shadowing X% of <key> traffic via <router>"

User Flow

Before: an admin running shadow evals over several keys cannot tell which key each job samples

1. They open http://localhost:4000/ui/?page=cost-optimization and the active card says "Shadowing 25% via auto_router1"
2. With two jobs on two different keys, both cards name only the router, so the cards are indistinguishable
3. They send GET http://localhost:4000/auto_router/shadow_eval and each job carries only "api_key_id": "64c87d65f785...", a hash they must cross-reference against the key list by hand

After: every job names the key it samples, on the card and over the API

1. They open http://localhost:4000/ui/?page=cost-optimization and the active card says "Shadowing 25% of prod-batch-alpha traffic via auto_router1"
2. With two jobs on two different keys, each card names its own key; a reverse job reads "Comparing auto_router1 to gpt-5 on 10% of prod-batch-alpha traffic"
3. They send GET http://localhost:4000/auto_router/shadow_eval and each job carries "key_alias": "prod-batch-alpha" and "key_name": "sk-...ghYg" next to the hash; a key with no alias falls back to its masked name, and a deleted key returns null for both while the card falls back to the shortened hash

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• I have added a screenshot of my new test passing locally
• My PR passes all unit tests on make test-unit [https://docs.litellm.ai/docs/extras/contributing_code]
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature

Changes

Backend: ShadowEvalJobResponse gains nullable key_alias and key_name, resolved from the key row at read time. The start endpoint reads them off the key row it already fetched for validation; list, detail, and stop resolve through one batched find_many over the distinct hashes, so the list path adds a single query rather than one per job. Deleted keys resolve to null

UI: the shadow eval card headline names the shadowed key in both directions, falling back from alias to masked key name to a shortened hash. schema.d.ts regenerated via npm run gen:api

Heads up: open PR #36871 moves api_key_id into a per-job keys list; these labels rebase onto it as per-key fields

Proof of fix

Live proxy on localhost:4000 backed by a fresh Postgres database, an auto-router (auto_router1), and the local stub provider. Same commands at the merge base and at this PR's head

Before, at merge base c1fc598, the list response has no key fields:

$ curl -s http://localhost:4000/auto_router/shadow_eval -H "Authorization: Bearer sk-1234"
[{"job_id": "cmsxrq5sy0000mgpb0s5hdnqn", "api_key_id": "64c87d65f78523d2d20f1d607ed6a773455a9604c6269b725b4914fa1c20d6e8", "router_name": "auto_router1", "direction": "forward", ...}]

After, at this branch, create an aliased key and start a job on it:

$ curl -s http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" \
    -H "Content-Type: application/json" -d '{"key_alias": "prod-batch-alpha"}'
{"token_id": "64c87d65f785...", "key_alias": "prod-batch-alpha", ...}

$ curl -s http://localhost:4000/auto_router/shadow_eval/start -H "Authorization: Bearer sk-1234" \
    -H "Content-Type: application/json" \
    -d '{"api_key_id": "64c87d65f78523d2d20f1d607ed6a773455a9604c6269b725b4914fa1c20d6e8", "router_name": "auto_router1", "shadow_percentage": 25, "judge_model": "gpt-5-mini", "duration_days": 7, "max_turns": 100}'
{"job_id": "cmsxrq5sy0000mgpb0s5hdnqn", "api_key_id": "64c87d65f785...", "key_alias": "prod-batch-alpha", "key_name": "sk-...ghYg", "router_name": "auto_router1", "shadow_percentage": 25.0, "status": "running", ...}

List and detail both resolve the labels, and the fallbacks hold: a job on a key generated without an alias reports "key_alias": null, "key_name": "sk-...EY4A", and after POST /key/delete on that key the same job lists "key_alias": null, "key_name": null:

$ curl -s http://localhost:4000/auto_router/shadow_eval -H "Authorization: Bearer sk-1234"
[{"job_id": "cmsxrqngf0001...", "key_alias": null, "key_name": null, "router_name": "auto_router1", "shadow_percentage": 10.0, "status": "running", ...},
 {"job_id": "cmsxrq5sy0000...", "key_alias": "prod-batch-alpha", "key_name": "sk-...ghYg", "router_name": "auto_router1", "shadow_percentage": 25.0, "status": "running", ...}]

$ curl -s http://localhost:4000/auto_router/shadow_eval/cmsxrq5sy0000mgpb0s5hdnqn -H "Authorization: Bearer sk-1234"
{"job_id": "cmsxrq5sy0000...", "key_alias": "prod-batch-alpha", "key_name": "sk-...ghYg", "judged_count": 0, "status": "running", ...}

UI screenshots of the card headline to follow in a comment (http://localhost:4000/ui/?page=cost-optimization with the job above running)

[greptile-apps]

Greptile Summary

The PR enriches shadow-evaluation responses with the sampled key’s alias and masked name, then uses those labels in dashboard job headlines.

• Resolves key labels with one batched verification-token query for list responses and equivalent enrichment for start, detail, and stop responses.
• Extends the Pydantic and generated OpenAPI response types with nullable key-label fields.
• Updates dashboard headlines and fallback behavior, with backend and component tests covering labeled and deleted keys.

Confidence Score: 5/5

The PR appears safe to merge, with the backend and UI consistently handling present, missing, and deleted key labels.

The label lookup uses the same hashed token stored on each job, preserves existing authorization boundaries, batches list enrichment, and keeps the new response fields nullable for backward compatibility.

Important Files Changed

Filename	Overview
litellm/proxy/management_endpoints/auto_router_endpoints.py	Adds consistent key-label enrichment across all shadow-evaluation response endpoints, using a single batched query for lists and null fallbacks for deleted keys.
litellm/types/management_endpoints/auto_router_endpoints.py	Adds backward-compatible nullable key_alias and key_name fields to ShadowEvalJobResponse.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Displays the sampled key in forward and reverse job headlines with alias, masked-name, and shortened-hash fallbacks.
tests/test_litellm/proxy/management_endpoints/test_auto_router_endpoints.py	Verifies start, list, and detail label resolution, batched lookup behavior, and deleted-key null handling.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.test.tsx	Verifies updated headlines and the complete key-label fallback order.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Regenerates the OpenAPI declaration with optional nullable key-label fields.

_{Reviews (1): Last reviewed commit: "feat(shadow-eval): name the shadowed key..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...roxy/management_endpoints/auto_router_endpoints.py	92.30%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadoweval_card_copy (59476ae) with litellm_internal_staging (2bc87ec)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (3f4810b) during the generation of this report, so 2bc87ec was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩