test: remove the five test functions a later definition shadows - #37591
Conversation
Python binds a name once per scope, so when a module or class defines the same test twice only the last one exists. The earlier definitions are unreachable: pytest never collects them, and nothing that references them can fail. A sweep in August cleared nine of these. Five have appeared since, which is the argument for a rule rather than another sweep. Each survivor is the better version, so nothing is lost. The two SQS logger twins additionally stub `asyncio.create_task`, which the shadowed copies did not. The cost-calculator duplicate is a two-line stub that also takes a `model_item` parameter no fixture supplies, so it could not have run even unshadowed. The two `test_prompt_caching` bodies are both `pass`. Collecting the four files reports 416 tests before and after. `tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a file named exactly `conftest.py`, nothing imports this one, and the space in the name says what it was.
Greptile SummaryThis PR removes five Python test definitions that were unreachable because later definitions rebound the same names, plus an unused nonstandard conftest copy.
Confidence Score: 5/5The PR appears safe to merge because it removes only definitions and configuration-copy code that were not active in pytest collection. Each removed test was rebound by a later same-named definition in the same Python scope, while the deleted nonstandard conftest filename was neither automatically discovered nor explicitly loaded.
|
| Filename | Overview |
|---|---|
| tests/llm_translation/test_bedrock_completion.py | Removes an earlier module-level test definition already shadowed by a later definition, without changing collected coverage. |
| tests/llm_translation/test_openai.py | Removes the first of two same-named class methods; both were no-op test stubs and only the later binding was collectable. |
| tests/local_testing/test_completion_cost.py | Removes an uncollectable same-named test whose body only imported Router; the surviving parametrized test retains the actual behavior checks. |
| tests/logging_callback_tests/test_sqs_logger.py | Removes two shadowed queue tests while retaining later versions with equivalent assertions and safer task stubbing. |
| tests/proxy_unit_tests/conftest copy.py | Deletes a stray nonstandard conftest copy that pytest did not automatically discover and that repository tooling does not explicitly load. |
Reviews (1): Last reviewed commit: "test: remove the five test functions a l..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
tin-berri
left a comment
There was a problem hiding this comment.
done — #37591 approved. Deletes 5 test function definitions that a later same-named definition in the same scope shadows — Python's own name binding means the earlier definition never gets bound, so pytest never collects it, giving a false sense of coverage to anyone reading the dead one (plus a stray duplicate conftest copy.py). Proof-of-fix is exactly right for this claim: an AST-based detector (scripts/find_shadowed.py) names the precise dead definitions and their line numbers before the change, and pytest --collect-only reports the identical 416-test count before and after removal, confirming nothing that actually ran got deleted. Small, mechanical, part of the same CI-hygiene pass as #37586/#37588/#37590/#37605. CI green.
* fix: keep tool_use and tool_result adjacent when converting mid-conversation system turns
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
* fix(realtime): resolve the vertex token resolver at call time
Binding the bound method at import froze the module-level VertexBase
instance, so callers that swap it no longer reached their replacement.
* fix(cli): finish a refused token file rewrite in place
Taking the secret out of ~/.litellm/token.json stages a replacement and moves it into
place, which needs room for a second file and a directory that will accept a new entry.
A full disk refuses the first and a read-only ~/.litellm the second, and logout gave up
there: it removed the file when it could, dropping the record that the keychain had never
been confirmed clear, so the logout after it reported a clean keychain it never checked
Shortening the file already in place needs neither, so the logout scrub and the legacy
migration now fall back to overwriting it where it lies. On a read-only ~/.litellm the
logout the user asked for now happens, instead of coming back with instructions to delete
the file by hand
* fix(cli): stop whoami calling an unreadable credential authenticated
`lite whoami` led with "Authenticated" whenever a token file was on disk, even when the
keychain holding the credential would not give it up. The notice about that sat below the
account lines, so the session read as a working one and sent the user looking for the
problem anywhere but the keychain
* test(e2e): require consecutive cache reads before the reminder turn
* fix(router): let the routed deployment's own litellm_params beat forwarded auto_router marker params
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes #37613
* test(e2e): pin openai_passthrough routing, cost logging, and file list isolation
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
* fix: harden vertex live passthrough against client model forms and dict credentials
- accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model
- keep a dict service account intact instead of stringifying it
- treat same-target deployments holding different credentials as ambiguous
- guard both websocket states before every close so a second close cannot raise
- build the sendable close codes from the public CloseCode enum
* test(e2e): drop the passthrough streaming-cost test, it needs a config flag
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
* test(e2e): pin the unflagged Vertex cache test to us-east5
* fix(router): carry forwarded auto_router marker params per request and keep tier overrides and marker flags
The forwarded-keys record moves off the shared metadata dict onto the
per-request kwargs, where _update_kwargs_with_deployment consumes it, so
sibling requests that share a metadata dict (abatch_completion) can no
longer clear it mid-routing. Per-tier litellm_params from the hook
response are never treated as forwarded marker params, and a deployment
only beats a forwarded value when it sets its own, not when it carries a
LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
* docs(e2e): correct the passthrough-stream registry row's uncovered reason
* fix(cli): write the logout note again when the file holding it had to go
When a full disk refuses the replacement file and a read-only token file
refuses the rewrite in place, the only way left to get the secret off disk
is to remove the file carrying it. That file was also the note saying the
keychain went unchecked, so its absence made the next logout read a
keychain that was never confirmed as one already known to be clean.
Removing it is what frees the room the replacement was refused for, so the
note is written again on the way out and the logout after this one still
warns.
* test(cli): cover the keyless token record and keep keyring to the cli extra
`lite up` treats a token record whose key the keychain would not hand over as no
login at all, and that clause had no test: every existing freshness test passed a
record carrying a real key, so deleting the clause left the whole suite green
The base install smoke check now also asserts keyring is absent, which is what
makes the lazy import in cli_keyring meaningful. keyring ships in the cli extra
only, so a plain `pip install litellm` must not be able to reach it
* fix(anthropic): map metadata.user_id to prompt_cache_key on the /v1/messages bridge
Both /v1/messages bridges (Responses API adapter for openai/* and the
chat-completions adapter) now derive prompt_cache_key from the first 64
characters of metadata.user_id, next to the existing user mapping. The
chat bridge only sets it when the resolved provider advertises
prompt_cache_key in its supported params, so providers that reject
unknown params are unaffected. A prompt_cache_key sent explicitly by the
client always wins over the derived value.
Fixes #37508
* feat(proxy): native CLI login with OAuth authorization code + PKCE
The proxy's OAuth authorization server (dynamic registration, PKCE S256,
loopback redirects, single-use codes, refresh rotation) gains a proxy-API
audience: /authorize?resource=<proxy origin> renders a consent page with
team selection and /token mints the same per-user credential lite login
mints, so a native CLI can sign a user in through the system browser and
call /v1/* with user and team attribution. Adds GET /.well-known/litellm-cli-auth
as the versioned discovery contract for non-Python clients, POST /revoke
(RFC 7009) for logout, and lite login --pkce, lite logout, and
lite auth print-token on the CLI side. Proxy-API grants only ever redirect
to a loopback address and the server never picks a team on the user's behalf.
Fixes #37332
* test(e2e): request reasoning explicitly on the reasoning-cost assertions
The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
* fix(cli): stop asking a keychain that already stopped answering
A pre-flight that times out leaves its write parked inside the keychain, holding
it against every later call, so the next read blocks on the main thread with no
timeout of its own. Anything that resolves the credential more than once in a
process hits it: an SDK Client built a second time never returns.
The vault now remembers the silence and reports the keychain unreachable for the
rest of the process rather than queueing behind the parked call.
* Map cache_control_injection_points to OpenAI prompt_cache_breakpoint on GPT-5.6+ targets
When the resolved deployment is provider openai and the model is GPT-5.6 or
newer, the cache control hook now writes prompt_cache_breakpoint on the
targeted content block and sets prompt_cache_options to explicit mode unless
the caller already passed one. The /v1/messages bridges carry the marker
through (the Responses bridge moves a marked system prompt into a developer
message, since top-level instructions cannot hold one). Breakpoint counting
and the stand-down check recognise both marker kinds, and client breakpoints
already present in messages are no longer subtracted from the cap twice.
Fixes #37509
* fix(cli): pin PKCE discovery to the typed proxy and ignore foreign sibling records
The discovery document is accepted only when its issuer is the --base-url the user
typed and every endpoint and the resource share that origin (RFC 8414 section 3.3),
so a tampered or redirected document can no longer point the code, verifier, or
refresh token at another host. After a failed refresh the re-read token record is
used only when it continues the same credential (same proxy, token endpoint, and
resource) and has not expired, so a concurrent login against a different proxy can
never hand this one its key
* fix(cli): pick the credential by the sign-in it came from
A login the keychain accepted whose token file could not be replaced left the
superseded secret on disk, and the next load preferred the file unconditionally,
so it served the old credential and erased the new one from the keychain on the
way past. The keychain entry now carries the timestamp of the sign-in that minted
it, and the two stores are compared on that instead.
* docs(cli): say why a refused scrub does not roll the keychain back
The two stores hold different credentials on that path, so the rollback a
migration does would hand the superseded one back out. The login that could not
replace the file already named the state, and logout reports it too.
* fix(anthropic): skip the derived prompt_cache_key for litellm_proxy deployments
* fix(cli): bind the sibling refresh fallback to the same user and team
* fix(cli): keep each sign-in stamped past the one it replaces
The stamp in the keychain entry is what decides that secret against one still
sitting in the token file, and it came straight off the wall clock. A clock
that stepped backwards between two logins therefore handed the win to the
older of them: a login the keychain took but the token file could not be
pointed at was resolved back to the credential it replaced, and the fresh one
was erased from the keychain on the way past.
save_cli_token now reads the stamp already on disk and pins the new sign-in
just above it, so the ordering never depends on the clock having moved
forwards. On a clock that did, this changes nothing.
* docs(deps): say that the cli extra pulls cryptography on linux
The comment above the extra named cryptography as one of the heavy imports a
thin install leaves out. That stopped being true when keyring joined the
extra: on Linux it reaches the Secret Service through secretstorage, which
depends on cryptography.
* fix(cli): stamp each sign-in past the keychain as well as the file
A login the keychain took but the token file could not record leaves the
keychain naming a later sign-in than the file does. Reading only the file
then stamps the next login below that keychain entry, and a clock that
went back far enough puts the superseded credential back in use.
* fix(cli): renew a --pkce key wherever lite checks freshness and show its expiry in whoami
* Gate OpenAI prompt cache breakpoints on the real target and carry them through /v1/responses
The cache control hook also runs on litellm.responses() input. On a
GPT-5.6 deployment it wrapped a string-content item into a chat-shaped
{"type": "text"} part, which the Responses API rejects, and it never
marked input_text, input_image or input_file parts, so no breakpoint and
no prompt_cache_options reached the provider. Add the Responses part
types to the eligible block set and translate chat-shaped text parts on
non-assistant items to input_text in
ResponsesAPIRequestUtils.merge_prompt_management_input, which both the
async and the sync prompt management sites go through.
The dialect also fired for any GPT-5.6 name that resolved to provider
openai, including deployments pointed at a custom api_base that does not
understand prompt_cache_breakpoint. Decide it once per request from the
provider, the model map and the resolved api_base (request, then
litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE): only
api.openai.com and *.api.openai.com hosts speak the dialect, a top-level
prompt_cache_options opts a custom target in, and litellm_proxy/ targets
never get it. maybe_seed_default_injection_points takes api_base and
stamps the finished decision on the points as _litellm_openai_dialect so
the sync completion() path, whose hook params do not carry api_base,
honors it; maybe_inject_cache_control takes api_base from the
/v1/messages handler.
Eligibility now comes from a supports_prompt_cache_breakpoint model map
flag on the OpenAI gpt-5.6 entries, exposed through
litellm.utils.supports_prompt_cache_breakpoint, with the GPT version rule
kept only for models the map does not know. The OpenAI dialect no longer
reserves a slot for tool_config points, which OpenAI has no cache block
for, and with_prompt_cache_breakpoint plus the chat bridge helper return
a new block instead of mutating their input.
* fix(cli): tell whoami's expired PKCE key to sign in again instead of promising a renewal
* test(cli): model keyring's null backend in the vault test double
FakeSecretVault could only stand in for a discarding backend by passing
KeyringDiscardsWrites as its `failure`, which also made read() and erase()
hand it back. Neither SecretRead nor SecretErase admits that outcome and the
real KeyringVault never produces it there, so the login path's match was
falling through on a value it can never see. Give the double a `discards`
flag that reports it from write() alone, which is what the null backend does.
Also widen lint-format-check-changed's pathspec. Git wildmatch runs without
FNM_PATHNAME here, so 'litellm/**/*.py' still requires an intermediate
directory and silently skipped all 21 top-level modules, litellm/__init__.py
and litellm/main.py among them. All 21 already pass ruff format.
* fix(proxy): refuse a teamless native-client grant for a user who has teams
The consent page offers the team picker, but a form posted without a team
sealed a teamless grant and the token endpoint minted an unscoped
credential for a team member, escaping the team attribution classic lite
login always applies. The minter now refuses such a grant on redemption
and refresh alike; memberships whose team rows are gone still count as no
team so they cannot lock a user out
* Fall back to the GPT version rule when the cost map carries no breakpoint flag
A proxy on the default remote cost map never produced a prompt cache
breakpoint: the published map has the gpt-5.6 entries without
supports_prompt_cache_breakpoint, so the model-map gate returned False
for every listed model and only LITELLM_LOCAL_MODEL_COST_MAP=True (the
repo .env, hence the passing unit tests) made the feature work. The hook
now honors the flag when the entry carries one, True or False, and
otherwise applies the GPT-5.6+ version rule to the model name, so a map
that lags the flag still gets the OpenAI dialect. The model-map tests
pin litellm.model_cost to the bundled backup map and a new test drives
the hook against an unflagged gpt-5.6 entry.
completion() and acompletion() take base_url as an alias for api_base
that only lands on api_base after the cache control hook ran, so a
GPT-5.6 call at a non-OpenAI gateway given through base_url still got
the dialect. Both seed calls and the unstamped request-params read now
look at base_url too.
ResponsesAPIRequestUtils.merge_prompt_management_input reshaped hook
output in place, retyping text parts to input_text on the caller's own
message objects. The merge now shapes a copy of each message as it
emits it, so the identity-based merge keeps working on the hook's
objects and nothing the hook or the client owns is mutated.
* fix(cli): never follow a redirect when posting to the proxy's OAuth endpoints
Discovery checks that every endpoint sits on the proxy origin, but the CLI's
requests.Session followed redirects, and requests replays a POST body on
307 and 308, so a token or revocation endpoint answering with one of those
would have sent the code and verifier, or the refresh token, wherever
Location pointed. Every POST now goes out with allow_redirects=False and a
3xx answer fails the command with a message naming where it pointed
* test(cli): pin the shared stamp's effect on the freshness shortcut
The stamp both orders the two stores and drives is_cli_token_fresh, and
nothing tied the two together, so a login that inherits a stamp from the
future could stop being a deliberate trade without anything failing.
Also corrects the lint-format-check-changed comment: git pathspecs match
recursively, so the target checks a superset of the CI step rather than
an identical set.
* fix(cli): revoke the previous login's refresh token when lite login replaces a stored record
* fix(cli): name lite login --pkce and the refresh failure when a key cannot be renewed
A PKCE credential whose renewal is refused (for example after lite logout ran
on another copy of it) used to fail lite auth print-token with the classic
'Token expired. Run lite login again' hint and no reason, while lite whoami
already named lite login --pkce. fresh_api_key now reports why a renewal
failed through a warn callback whenever no sibling rotation rescued it, the
CLI prints that reason on stderr, and the expiry hint names the command that
produced the credential. Both READMEs document the admin revocation semantics
and the Redis precondition for refresh single use on several workers.
* fix(cli): renew the key once per lite auth print-token run
The cli group already resolves the stored key for the server it was pointed
at, so print-token re-ran the renewal and, when the refresh token had been
revoked, posted to /token twice and printed the reason twice. print-token now
reuses the group's result whenever the stored record was issued for that
server and no --api-key or LITELLM_PROXY_API_KEY took precedence, and only
resolves the key itself when invoked bare for a different server.
* fix(cli): let lite up trust the key the group already renewed
The lite group resolves the stored key once for every command and renews a --pkce key on the way in. lite up then asked the token file again, so every start sent a second refresh to the proxy, and once the refresh token was burned the refusal printed twice. _ensure_fresh_login now reuses the key the group resolved when the group read it from the token file, and only re-reads the file after the interactive login it starts itself. Also covers print-token through the group with a renewing session in the tests
* fix(mcp): answer 503 when a refresh-token burn cannot be recorded
revoke_refresh_token discarded the single-use claim result, so a revocation that arrived while Redis was unreachable answered 200 and left the refresh token live. The token endpoint reported the same outage as invalid_grant "already used". The guard now reports first, replayed, or unavailable, and both endpoints answer 503 temporarily_unavailable for an outage (RFC 7009 section 2.2.1, RFC 6749 section 5.2), which the CLI surfaces as a one-line warning while keeping the key it has
* fix(cli): keep the login record when the proxy cannot record a logout's revocation
A 503 from POST /revoke means the proxy could not write the single-use record, so clearing the local record left a live refresh token nobody could revoke and a hint to retry with nothing left to retry. lite logout now keeps the record, exits 1, and asks to be run again shortly. A refused or unreachable revocation still clears the record and warns as before, and a re-login that replaces a record keeps its existing warning because the new record already stands
* refactor(ci): fold the nine thin unit-shard callers into one matrix (#37590)
Nine workflow files existed only to make a single call to _test-unit-base.yml
with a different test-path. Adding a shard meant adding a file; changing
anything shared meant editing nine. One matrix caller replaces them, so a shard
is now one entry.
Check names are unchanged, which is the whole constraint. A reusable-workflow
job reports as "<job name> / <inner job name>", so setting `name` to the shard id
alone reproduces today's context strings exactly: the eleven the matrix produces
are eleven of the twenty-three "/ Run tests" contexts the branch ruleset
requires, matched string for string. No ruleset edit is needed and none should
be made for this.
Every matrix entry states its timeouts even where they equal the base defaults.
An absent matrix key renders as an empty string rather than falling back, and an
empty string is not a number, so a partially-specified entry would fail the call.
tests/proxy_unit_tests keeps test-unit-proxy-db.yml. It is already a matrix and
its shard-coverage guard reads that file by name, so folding it in belongs with
generalising that guard into assert_ci_coverage.py rather than here. Its twelve
shards are the remaining required contexts.
test-unit-documentation.yml stays too: it does not call the base workflow.
* chore(ci): close the test-census blind spots and move scripts out of workflows/ (#37586)
The agent job's CircleCI glob collected `tests/agent_tests/**/test_*.py` and then
piped it through `grep -v` to drop `local_only_agent_tests/`. `assert_ci_coverage.py`
reads the glob but not the pipeline, so those two files looked covered and were
invisible to the census. The glob now excludes them structurally and they carry an
allowlist entry instead, which is a decision on the record rather than a hidden
filter. The collected file set is unchanged: `tests/agent_tests/` holds exactly one
CI-runnable test at the top level.
`tests/scim_tests/` held a single JSON fixture and no tests, referenced from nowhere.
`.github/workflows/` is for workflows. Both stray scripts move to `.github/scripts/`
with their callers updated: the price-file updater is invoked by
`auto_update_price_and_context_window.yml`, and the translation-report runner by
`make test-llm-translation`. The audit listed the latter as orphaned, but Makefile
line 317 still runs it, so it moves rather than being deleted.
The rollout heads-up workflow was a deliberate one-shot for the agent-shin rollout.
That rollout is done, the triage and auto-close workflows have been running daily
since June, so the pre-flip warning window is long past. Its script and dedicated
test go with it, and the sibling workflow-invariant test drops its entry.
* test: retire tests/old_proxy_tests, which holds no tests (#37605)
* test: retire tests/old_proxy_tests, which holds no tests
Twenty files named test_*.py, and pytest collects nothing from any of them:
uv run pytest tests/old_proxy_tests --collect-only -q
no tests collected, 16 errors in 114.17s
They are manual snippets against a running proxy, written at module level with
no test function, no assertion and no entry point, so the only thing the name
buys them is a place on the coverage allowlist. Sixteen of the twenty cannot
even be imported in this environment, wanting langchain, llama_index or
google.api_core, and ten still point at 0.0.0.0:8000, which stopped being the
proxy's default port some time ago.
Nothing outside the directory refers to it apart from the allowlist entry, which
goes with it. The other loose contents go too: five load_test_*.py scripts, a
bursty variant, two committed log files, an essay fixture and a stray .js
snippet.
Allowlist paths 88 -> 68, test files 2422 -> 2402, and no job loses anything it
was running. Recoverable from history if a snippet turns out to be someone's
habit.
* test: drop the retired old_proxy_tests paths from the coverage allowlist
* feat(ci): ratchet the test suite's zero-assert, mock-echo and global-state debt (#37588)
* feat(ci): ratchet the test suite's zero-assert, mock-echo and global-state debt
The suite's dominant failure mode is tests that cannot fail for the reason anyone
would want them to. The testing-strategy audit measured five shapes of it, and
nothing mechanical stops any of them from reproducing, so they keep reproducing.
`scripts/check_test_quality.py` is an AST checker for those five, emitting the
same `path:line: CODE message` contract as `scripts/check_type_discipline.py`:
TQ001 a collectible test with no assertion of any kind
TQ002 mock-echo, where every assertion only inspects the mock that was patched
TQ003 sys.path.insert inside the test tree
TQ004 raw `os.environ[...] =`, which leaks into whatever runs next
TQ005 `litellm.<attr> =`, the process-wide leak the 491-line conftest undoes
`scripts/test_quality_gate.py` caps each rule against test-quality-budget.json,
seeded at exactly today's count, and fails only when a rule is both over its
limit and higher than the base being merged into, so a change is blamed for what
it adds and never for drift already in the base. `--update` lowers a limit by
what a branch cleared, so the ceilings only ever fall. It runs in the existing
required lint job, which means it enforces without a ruleset change.
TQ001 follows assertions into helpers defined in the same module, transitively.
Without that it flagged 111 tests in tests/e2e, the harness this program holds up
as the reference, because that suite factors its assertions into shared helpers
(`assert_auth_denied(result, ...)`). Following them leaves 25, all of which reach
their assertions across a module boundary; those are grandfathered and documented
rather than papered over.
The seeded counts land within about 10% of the audit's independent numbers for
every rule measured on the same subtree, which is the cross-check that the
definitions here match the ones the audit pinned.
* fix(ci): resolve test helpers per scope, not by bare name
The helper walk keyed every function in a module by its bare name, so two
same-named helpers in different classes collided and the last one parsed won.
A test calling `self._check()` could be cleared by a `_check` belonging to a
different class, or flagged because of one.
Resolution is now scoped: a bare name looks up the module-level functions, and
`self.<name>` looks up the enclosing class's own methods and no other class's.
Recursion is tracked by function identity rather than by name, so the cycle
guard cannot be confused by the same collision.
This surfaced one real zero-assert test that a same-named helper elsewhere had
been clearing, so TQ001 seeds at 750 rather than 749.
The test module has to register itself in sys.modules before exec_module:
`@dataclass(slots=True)` rebuilds its class through `sys.modules[__module__]`,
and Scope fails to construct without it. Recorded at the call site, since it
reads like avoidable global mutation otherwise.
* fix: register test-quality-budget.json with the ratchet alarm
The repo keeps one census over its budget files: every *-budget.json on disk
must appear in DEFAULT_BUDGETS, or its ceilings can be raised with no signal.
tests/test_litellm/test_budget_ratchet_check.py asserts that set equality and
caught the new budget on the way in.
Registering it also turns the alarm on for TQ001-TQ005, so a later PR cannot
quietly raise a test-quality ceiling. The file already uses the {limit: N}
schema the ratchet reads, so no other change was needed.
* test: remove the five test functions a later definition shadows (#37591)
Python binds a name once per scope, so when a module or class defines the same
test twice only the last one exists. The earlier definitions are unreachable:
pytest never collects them, and nothing that references them can fail.
A sweep in August cleared nine of these. Five have appeared since, which is the
argument for a rule rather than another sweep.
Each survivor is the better version, so nothing is lost. The two SQS logger
twins additionally stub `asyncio.create_task`, which the shadowed copies did
not. The cost-calculator duplicate is a two-line stub that also takes a
`model_item` parameter no fixture supplies, so it could not have run even
unshadowed. The two `test_prompt_caching` bodies are both `pass`.
Collecting the four files reports 416 tests before and after.
`tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a
file named exactly `conftest.py`, nothing imports this one, and the space in the
name says what it was.
* feat(ci): guard shard assignment across every sharded test tree (#37593)
tests/proxy_unit_tests had a 30-line YAML parser inlined in its workflow that
failed the run when a test file there belonged to no shard. tests/test_litellm
is sharded the same way, with no catch-all bucket, and had no such guard: a new
directory under it (or under its proxy subtree) is collected by nothing and runs
nowhere, and the coverage census cannot see it because a token like
tests/test_litellm/test_*.py already answers 'yes, that tree runs'.
The two questions differ. The census asks whether a file runs at all, so an
ancestor path standing in for everything beneath it is a fine answer. Shard
assignment asks which shard owns a child, and there that same ancestor path is
precisely the bug. _token_covers keeps the first meaning; _token_names adds the
second, and the guard now walks a list of sharded trees rather than one hardcoded
directory. Both read the same test-path keys, so there is one workflow parser.
A directory needs a shard when it holds a test file, not when it is named test_*.
That drops the hardcoded test_configs exception and keeps fixture directories
like expected_fine_tuning_api out on their own merits.
The job keeps its name and its workflow, since assert-shard-coverage is a
required status check on litellm_internal_staging.
Verified red-first: a planted directory under tests/test_litellm, a planted
directory under tests/test_litellm/proxy, and a planted file under
tests/proxy_unit_tests each fail the guard, while a fixture-only directory does
not. 327 children across the three trees are assigned today.
* feat(ui): multi-key shadow eval picker and per-key breakdown (#37389)
Stacked on the multi-key shadow eval backend. The key picker becomes a
paginated multi-select with chips, built on the base-ui combobox chips
primitives, with the pagination and debounced-search logic extracted into a
shared usePaginatedCombobox hook that PaginatedSearchSelect now also uses.
The detail view gains a per key table showing each key's own status, judged
turns against its budget, and win rates from the by_key slice, and the job
headline pluralises to "N keys" for multi-key jobs
* fix(mistral): correct zai-glm-5-2 limits, add cached-input price and glm-5-2 alias
Mistral's live /v1/models reports max_context_length 1048576 and capabilities.reasoning
true for zai-glm-5-2, and its docs price cached input at $0.14/M. Without
cache_read_input_token_cost LiteLLM billed every cached prompt token at $0, so a repeat
request against a 21k-token cached prefix logged $0.0000135 instead of its real cost.
Mistral also serves the model under the short glm-5-2 name, which had no cost map entry
at all and therefore no pricing, so add it alongside.
* fix(ui): make dark-mode form controls visible (#37648)
* fix(ui): make dark-mode form controls visible
Two dark-mode defects left form controls without any visual boundary or fill.
`--input` and `--border` share one value in `.dark`, oklch(0.309), which resolves to
rgb(48,48,48). Against `--background` (33) that is a 15-step stroke, and against `--popover` (42)
it collapses to 6 steps out of 255, so a control inside any dialog is effectively undrawn. The
controls also use `bg-transparent`, so there is no fill cue either and only the placeholder text
renders. Measured 1.09:1 against the dialog surface where WCAG 1.4.11 asks for 3.0:1 on the
boundary of a user interface component. Splitting `--input` off at oklch(0.56) restores 3.07:1
without touching `--border`, which stays where it is because it draws decorative separators rather
than control boundaries. 91 controls across 19 routes were measured at the collapsed value, every
one with an identical stroke and surface, so a single token covers all of them.
Separately, `@tailwindcss/forms` paints a white fill on every bare control. The block above
already neutralises that for `combobox-chip-input`, but its audit covered `components/ui` only,
and hand-rolled controls elsewhere still render white on a dark page: typed text lands at 1.11:1
and native selects at 2.19:1 on `/model-hub-table`, `/playground`, `/guardrails`, `/mcp-servers`
and `/models-and-endpoints`. Tracking `--background` fixes those at 14.51:1 and 7.34:1.
Light mode is unchanged by both. The token edit is scoped to `.dark`, and `--background` in
`:root` is the same white the plugin was already painting, verified control-by-control on a dev
server: backgrounds stay rgb(255,255,255) and ratios stay 20.13:1 and 4.84:1.
* fix(ui): keep the combobox chip input transparent under the bare-control fill
The new base rule matched at (0,2,1) while the combobox chip-input override
sits at (0,1,0), so ComboboxChipsInput lost its transparent background and
painted an opaque page-colored rectangle inside the chips container, which
carries its own bg-transparent / dark:bg-input/30 fill.
Folding the exclusions into one :not() list adds the chip input and drops the
selector to (0,1,1). Every @tailwindcss/forms base selector is wrapped in
:where(), so it lands at (0,0,1); (0,1,1) still outweighs it and bare inputs,
textareas and selects keep the fill this PR gives them.
* fix(ui): give status colours a readable foreground and drop the muted 70% step (#37649)
* fix(ui): give status colours a readable foreground and drop the muted 70% step
The four status tokens are lightened for dark mode, which is correct when they are used as text
and wrong for the 27 places that use them as a background under `text-white`. Every one of those
passes in light and fails in dark: success 1.78:1, warning 1.72:1, info 2.64:1, destructive
2.89:1. The cause is not 27 authoring mistakes, it is that no `--success-foreground` and no
sibling ever existed, so `text-white` was the only thing available to write. Adding the four
companions and registering them in `@theme` makes the correct pairing expressible, and the call
sites then read `text-success-foreground` instead of a hardcoded colour. Dark lands at 9.98, 10.31,
6.72 and 6.15.
Light is deliberately pure white rather than the near-white the other `-foreground` tokens use, so
the four ratios stay at exactly the 4.95, 5.03, 5.25 and 4.77 they are today instead of drifting
down to 4.73, 4.81, 5.02 and 4.56.
Separately `text-muted-foreground/70` measures 2.75:1 on a light page and 4.31:1 on a dark one,
so the same 183 occurrences fail AA in light and sit under it in dark. Dropping the opacity step
takes them to 4.84:1 and 7.34:1. The identical step on the placeholder base rule goes with them,
which is what put every input's placeholder at 2.75:1 in light.
Residual, not addressed here: `text-muted-foreground` over `bg-muted` reaches 4.39:1 in light,
still short of 4.5. Closing that needs `--muted-foreground` itself to move, which changes every
secondary label in the product and is a design call rather than a defect fix.
* fix(ui): finish the status-foreground swap and repoint no-op muted hovers
Four sites still forced text-white on a status fill because the class sat on
a child element rather than on the filled container, so the earlier sweep did
not reach them. The compliance quick-test bubble was worse: it paired bg-info
with text-success-foreground and its paragraph kept text-white on top, so the
dark-theme contrast the PR set out to fix was still reachable there
Dropping the /70 step also turned 21 existing "text-muted-foreground/70
hover:text-muted-foreground" pairs into hovers that change nothing, which
local/no-noop-hover-variant flags as an error. The affordance was "brighten on
hover", so these now hover to text-foreground, matching the 74 places that
already spell it that way
The remaining churn is prettier reflowing the handful of lines whose length
changed, since the token names are longer than text-white
* fix(ui): let the approve/reject confirm button pick the token its fill uses
Both submission review dialogs put text-success-foreground on the shared
button class while the fill below it swings between bg-success for Approve and
bg-destructive for Reject, so Reject drew a success token over a destructive
fill. The two tokens resolve to the same value today, so nothing looks wrong,
but the pairing only holds by coincidence and would break the moment either
token moves. Moving the token into the branch makes it track the fill
* fix(ui): drop the last 70% placeholders, still live on the legacy utility
Four inputs spell their placeholder colour with Tailwind's older
placeholder-<colour> utility rather than placeholder:text-<colour>, so the
sweep that dropped the 70% step passed over them. Tailwind 4.3 still emits
that utility, and utilities sit after base in the layer order, so those four
kept overriding the new input::placeholder rule and kept rendering at 70% in
dark mode, which is the contrast failure this PR set out to close
They now spell it the same way as the three placeholders the PR already
converted, which both removes the step and settles on one spelling
* fix(ui): make inline styles and code blocks follow the theme (#37651)
* fix(ui): make inline styles and code blocks follow the theme
Two families of colour that a stylesheet never gets to see, so dark mode could
not reach them.
The log details drawer paints most of its chrome through React inline style
objects holding raw hex: #f0f0f0 borders, #fafafa panels, #262626 body text,
the antd-era role accents on message cards, and a green/red guardrail summary
pill. Inline styles win over any class, so the drawer stayed light on a dark
page. Every one of those literals becomes the var(--color-*) it was already
imitating, which costs nothing in light mode and now tracks the theme. The
guardrail pill keeps its layout inline and moves its three colours onto the
success and destructive tokens the rest of the dashboard uses.
The eleven code blocks pass a prism stylesheet as a prop, so the theme has to be
picked in JavaScript. There is no dark-mode toggle in the app yet, only the
`dark` class the design system keys off, so useIsDarkMode subscribes to that
class through useSyncExternalStore and useSyntaxTheme swaps in oneDark when it
is set. Each call site keeps the light stylesheet it already had, including the
two that were relying on the prism default and now name it, so light mode is
unchanged everywhere.
Six of those call sites were casting the stylesheet to `any` or re-declaring its
type to get past the prop signature; the hook returns the right type, so the
casts are gone.
* fix(ui): let the markdown code renderer keep its own syntax theme
The three ReactMarkdown code renderers spread the remaining code element
props after style, so the incoming style attribute widened the prop type
and next build's type check rejected the hook's return value. The old
`coy as any` cast hid the same conflict. Spreading first lets the
explicit props win, which is what every one of these call sites meant.
* test(ui): cover the dark-mode hooks that pick a syntax stylesheet
useIsDarkMode carries the only real logic in this change: an external
store over the root element's class list. Cover the three things that can
regress, the class already being present at mount, the class being
toggled later, and the observer being disconnected on unmount, then cover
useSyntaxTheme handing back the caller's own stylesheet in light mode and
oneDark in dark. The assertions are on which stylesheet object comes
back, by identity, not on any colour it holds.
* refactor(ui): drop the last stylesheet cast in the chat code renderer
This was the one markdown code renderer still spreading the code element
props over its style, so an incoming style attribute would have won over
the theme, and the cast on the spread was what kept that compiling.
Spreading first lets the theme win and the cast go.
* fix(ui): move the policy flow builder onto theme tokens (#37654)
* fix(ui): move the policy flow builder onto theme tokens
The flow builder carried its own private palette: 126 raw literals across a
1644-line file, hardcoded into React inline style objects and SVG presentation
attributes. Inline styles beat every class, so the whole page, its version
sidebar, its step cards and its test panel stayed light no matter what the
theme said.
Each literal now resolves through the token it was already imitating. The greys
map onto card, muted, border, muted-foreground and foreground; the indigo and
blue accents onto info; the pass, fail and API-failure accents onto success,
destructive and warning; and the pale status washes become a color-mix of the
same token so they track it in both themes. Six icons carried their colour as
an SVG presentation attribute, where custom properties do not substitute, so
those switch to currentColor with the token set alongside.
Light mode is not byte-identical, and that is the point: the file stops keeping
a second palette. Of the mappings, card, muted and border land on the exact same
rgb they had, covering most of the file. The rest snap to the dashboard's
canonical shade, which mostly means slightly darker text and deeper status
colours: the gray-400 labels pick up real contrast, the soft red on the fail
icon becomes the destructive red every other failure indicator uses, and the
indigo accent becomes the blue that info resolves to.
Verified in a browser on both themes. In dark mode nothing on the page paints a
light background any more; the six that still do are shadcn's inverted primary
buttons and badges, which are meant to.
* fix(ui): token the flow builder test textarea fill
The quick-chat textarea is the one bare form control left in the file, so the
@tailwindcss/forms base layer still paints it `background-color: #fff`. The
inline style overrode the plugin's border but not its fill, which left a white
box inside the now-dark test panel, and its text inherits the near-white
foreground, so the typed message was invisible in dark mode.
Pin both halves of the pair on the element the plugin styles: the card token it
sits on, and the foreground token it was already inheriting.
* test: settle three allowlist entries that were open questions (#37598)
* test: settle three allowlist entries that were open questions
The allowlist is meant to hold decisions, not deferrals, so an entry reading
'needs moving' or 'referenced by no job' is a gap wearing an exemption. These
three each get an answer.
The two prompt-factory tests move into the mirror, which is what their own entry
said they needed. Both were passing the whole time, so the 23 tests they hold
start running and the entry goes away rather than getting reworded.
test_aio_http_image_conversion.py is not a test. It fetches live image URLs,
times aiohttp against httpx, prints the ratio, and asserts nothing, and pytest
cannot collect it because its functions take arguments rather than fixtures.
Running it beside its siblings would buy CI a network dependency and a number
nothing reads, so it stays exempt with that written down.
test_litellm_proxy_extras_utils.py stays exempt with a measured reason. 24 of
its 28 tests pass; the 4 in TestMigrationSQLIdempotency fail because nine
migrations from 2026-04 onward use bare CREATE TABLE, ADD COLUMN and CREATE
INDEX where that file requires guarded forms. The convention eroded quietly
precisely because the test enforcing it has never run. Wiring it up is blocked
on what to do about those migrations, and editing them is not the answer, since
Prisma checksums an applied migration and a changed one breaks migrate deploy
for existing installs.
Allowlist entries 10 -> 9, paths 88 -> 86.
* docs(ci): correct the migration count in the proxy-extras allowlist reason
* feat(ci): ratchet tests that skip themselves when a credential is absent (#37612)
* feat(ci): ratchet tests that skip themselves when a credential is absent
* docs(ci): name the new rule where the gate's rules are listed
* fix(ci): require the condition to test for absence before TQ006 fires
* feat(ci): catch files a -k expression deselects from every job (#37601)
* feat(ci): catch files a -k expression deselects from every job
The coverage census asks whether some job names a file. It cannot ask what that
job's -k then does with it, and the gap is not hypothetical: tests/local_testing
is globbed by five jobs, two of which carry
-k "... and not router and not assistants and not langfuse and not caching and not cache"
while the other three keep one keyword each. Any file whose path holds an
excluded term is dropped by the first two and matched by none of the rest, so it
runs nowhere while the census counts it as covered. 118 tests across eight
caching files sit in exactly that hole today.
The new mode reads the same CircleCI jobs the census already parses and asks
whether each globbed file survives its job's selector. Two facts about -k make
that decidable without running pytest: it matches an item's own name and its
parents', so a term appearing in the module path deselects the whole file; and
the names it can match are otherwise the classes and functions in the file,
which ast reads. A positive term is therefore satisfied by the path or by a name
inside, which is what keeps a langfuse-named test inside test_logging.py from
being reported.
Where the parser is unsure it stays quiet. An expression with or, parentheses,
or a negated group is left unmodelled and its job is treated as claiming
everything it globs, so an unparsed selector can never raise a false alarm.
Glob translation learned character classes, without which
tests/local_testing/**/test_[a-mA-M]*.py matches nothing and the guard would
report that whole directory. The census and shard counts are unchanged by it,
2423 files and 327 shard children before and after.
Validated against the real thing: collecting tests/local_testing under each
job's own selector leaves 175 of 1577 tests unselected, in exactly the ten files
this check derives statically, no more and no fewer. Two of the ten are named
outright by other jobs, which the check credits, leaving the eight now recorded
in the allowlist as a decision rather than an accident.
Verified red-first: dropping one of those eight from the allowlist reports it,
and adding 'and not embedding' to the two part jobs reports test_embedding.py
and test_get_optional_params_embeddings.py.
* fix(ci): keep the slice guard from pairing one command's -k with another's glob
Two accuracy notes from review, both about the parser's model rather than its
current verdicts.
A job that runs several pytest commands offers no way to tell which glob a -k
belongs to, since both are read out of the same flattened job text. Combining
them could pair one command's exclusion with another command's glob and report a
file that in fact runs. Such a job is now left unmodelled, which means it claims
everything it globs, matching how the parser already treats an expression it
cannot read. Only one job in the config has two globs today and it carries no
-k at all, so no verdict changes.
The second is a deliberate limit, now stated where it lives: an excluded term is
only honoured when it sits in the module path, because that is the case that
takes the whole file with it. A term matching one function inside drops that
test and leaves the file running, and reporting it would be a false alarm.
Answering per-test instead would need a baseline of test ids that churns on
every rename, for a smaller failure than a file going dark.
Both are pinned by tests.
* test: run the 30 test files stranded in the second mirror (#37595)
* test: run the 30 test files stranded in the second mirror
tests/litellm sat beside tests/test_litellm, which is the mirror the repo
convention names, and no job collected it. The allowlist called the directory
unresolved and assumed it was a duplicate. It is not: 30 of its 34 files have no
counterpart in the real mirror, so they are tests nobody has run since they were
written, not copies of tests that run elsewhere.
Moving them in is byte-identical, and it is what makes them run. Every one is
now claimed by a shard's test-path rather than by an allowlist entry, and the
216 tests they hold pass. Directories that needed to become packages did, since
several files are named test_transformation.py and pytest cannot import two of
those from non-package directories in one session.
Never running is why three assertions had drifted away from the code:
* nvidia.nemotron-super-3-120b max_output_tokens, 32000 -> 32768
* sambanova/MiniMax-M2.7 max_input_tokens, 204800 -> 196608
* the Vertex text-to-speech handler moved from data= to json=, so the test
reads the decoded body off the json kwarg instead of parsing the data one
The first two follow model_prices_and_context_window.json, which the catalog
sync keeps current; the third follows the handler. In all three the test was the
stale side.
The lint workflow ran test_no_hardcoded_secrets.py by path and now points at the
new one.
Four files stay behind. Each shares a filename with a live test whose contents
are disjoint from it, so landing those means merging test bodies, which is a
content review rather than a move. The allowlist entry now names those four and
records how many tests each would bring, in place of calling the whole
directory unresolved.
* fix(ci): keep the secret scan out of the mirror's conftest
The secret-scan job runs pytest under uv run --no-project, so its environment
holds pytest and nothing else. That worked while the file sat in tests/litellm,
which has no conftest, and broke the moment it moved into tests/test_litellm,
whose conftest imports litellm on collection: ModuleNotFoundError: No module
named 'dotenv', before a single test ran.
The file is a repo-wide static scan that imports only base64, os, re and pytest,
so it belongs with the other repo-wide checks in tests/code_coverage_tests,
which has no conftest, rather than in the package mirror. Installing the full
dependency set into a 15-second job to satisfy a conftest it does not use would
be the wrong trade.
Verified with the job's exact command:
uv run --no-project --with 'pytest==9.0.2' pytest \
tests/code_coverage_tests/test_no_hardcoded_secrets.py -q
1 passed in 0.47s
* fix(ui): make hardcoded palette surfaces theme-aware (#37650)
* fix(ui): make hardcoded palette surfaces theme-aware
Twenty-one dashboard files painted fills from the raw Tailwind palette with no
dark counterpart, so in dark mode they rendered as near-white islands carrying
dark text: unreadable. The route sweep caught them on teams, access-groups,
policies, users, skills, guardrails-monitor, logs, compliance, playground,
fallbacks and the AI hub.
Where the hue already had a semantic token, the surface moves onto it. Every one
of these lines had a token on its border and a palette class on its fill, so
this finishes a migration that had stalled halfway: bg-blue-50 next to
border-info/20 becomes bg-info, bg-gray-50 becomes bg-muted, DocLink's bg-white
becomes bg-card, and the Alert error variant drops text-red-800 and text-red-600
for the destructive token its sibling variants already use.
Purple, violet and indigo have no token in the system, which is exactly why the
maps in PluginTableColumns, GuardrailsOverview, AccessGroupsTableColumns and
teamTableColumns had migrated every other entry and left those behind. Rather
than mint a brand token here, they take the dark palette step, matching what
TeamGuardrailsTab, add_agent_form, MCPToolsetsTab and mcp_connect already do.
Gradient stops get the same treatment since bg-linear stops have no token form.
Light mode is unchanged apart from the four surfaces that moved onto a token,
and those stay inside the same colour family.
The #1e1e1e code slabs in guardrail_info and CustomCodeModal are deliberately
left alone: they are intentionally dark editors in both themes, and their
gray-200 text stays legible either way.
* fix(ui): give dark surfaces a readable foreground step
The dark fills added for the purple and indigo surfaces left three nested
foregrounds on their original light-palette step, so the text and icon sitting
on those new fills dropped below readable contrast in dark mode.
text-purple-800 on purple-950 measured 1.72:1, text-indigo-600 on indigo-950
2.54:1, and text-purple-600 on the blue-950 gradient stop 2.73:1. Each now
takes the purple-300 / indigo-300 step this PR already uses elsewhere, which
lands them at 8.48:1, 8.02:1 and 8.31:1.
The pricing calculator renders the same cost expression twice, so both copies
move together rather than leaving one half-migrated.
* fix(ui): keep the guardrail chip remove button visible on hover
The chip itself moved to the dark indigo fill, but its remove button still
darkened to indigo-900 on hover, which against indigo-950 measures 1.40:1 and
makes the X vanish under the cursor in dark mode.
Dark mode now brightens to indigo-100 on hover instead, mirroring the light
theme where hover darkens away from the resting colour.
* fix(model_prices): consolidate nine open registry audits into one changeset
Combines the model-cost-map data from #35911, #36017, #36080, #36113, #36188, #36444, #37029, #37252 and #37632 onto current litellm_internal_staging, merged per entry field so older branches no longer revert fields the base has gained since they were opened. Drops the Gemini deprecation dates from #36188 and the text-embedding-004 date from #36080 that the official docs contradict.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): draw one Per Day savings bar per date on Cost Optimization (#37643)
* fix(ui): draw one Per Day savings bar per date on Cost Optimization
The page paged /user/daily/activity over raw rows, so a date spanning
pages arrived N times with partial metrics and rendered as N thin bars.
Switch to the single-shot aggregated endpoint, thread
include_current_utc_day through it to keep the live-end extension from
PR #36051, and merge the paginated fallback by date.
* fix(ui): keep aggregated call at four params and mock it in view tests
Trailing userId and includeCurrentUtcDay ride a named rest tuple so the
eslint max-params baseline stays at 23, and the CostOptimizationView
suites mock the new networking export their render now reaches.
* feat(complexity_router): add business classification rubric preset (#37534)
* feat(complexity_router): add business classification rubric preset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api schema for business rubric
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): suppress preexisting antd import violations in touched files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: satisfy test-quality gate in consolidated registry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): serve a dark-mode variant of the LiteLLM logo (#37656)
The bundled logo is a JPEG, so it carries no alpha and its white
background renders as a bright slab against a dark sidebar. Making it
transparent alone would not be enough either: the wordmark is near-black
and would disappear on dark.
Adds logo_dark.png, derived from the light logo. The sky-blue disc and
train are kept as they are behind a circular alpha mask, and the
wordmark's antialiasing is un-flattened from white into straight alpha
and repainted in the dark theme's own foreground colour. Both files are
1000x257, so swapping between them cannot shift the sidebar header.
/get_image gains a theme query param. The default response is byte for
byte what it was, and a logo configured through UI_LOGO_PATH is served
unchanged in both themes, since custom logos have no dark variant yet.
* fix(model_prices): drop duplicate zai-glm-5-2 entry superseded by staging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): route Phoenix traces to per-key/team projects under otel v2 (#36706)
* feat(otel): route Phoenix traces to per-key/team projects under otel v2
The v2 arize_phoenix preset read PHOENIX_PROJECT_NAME once at startup into a
static resource attribute, silently dropping the per-key/team project routing
v1 supported. Route it via Phoenix's x-project-name OTLP/HTTP header instead:
the env var stays the global default, and a phoenix_project_name (or
phoenix_project_name_override) in key/team metadata sends that key's traces
to the named project.
The project comes only from user_api_key_auth_metadata (server-set at auth),
never from client request metadata or StandardCallbackDynamicParams, since
choosing the telemetry destination is a data-exfiltration primitive. The
header is appended to the exporter's static headers rather than replacing
them, so the preset's Authorization survives, and it is gated to OTLP/HTTP
exporters because Phoenix only reads it on /v1/traces.
Also unban the bare phoenix_project_name fields from the request-body gate:
the proxy integrations ignore them (only user_api_key_auth_metadata routes,
and that stays banned), so rejecting them just broke SDK-style callers.
* fix(otel): root project-routed Phoenix spans in their own trace
Phoenix assigns a whole trace to one project by whichever span arrives
first. The request's auth/db/root spans always export through the default
provider without the project header, so a project-routed LLM span parented
into that trace got dragged back into the default project and the header
did nothing (verified against a live Phoenix instance). Detach the routed
span into its own trace with a link back to the request trace, mirroring
how the v1 Phoenix logger exported each request under its own local parent.
* fix(otel): drain in-flight spans before shutting down evicted providers
LRU eviction shut a routed provider down immediately, but an LLM span
opened at pre_call stays open until the later success or failure callback;
with more than 256 overlapping credential/project routes that in-flight
span was silently dropped instead of exported. Refcount open spans per
provider (hold at span open, release when the carrier is removed on close,
carrier-map eviction, or MCP stray-carrier cleanup) and defer a retired
provider's shutdown until its last open span closes.
* fix(otel): take the provider hold inside route_for to close the eviction race
pre_call can run on thread-pool workers, so between route_for returning a
provider and the caller recording its open span, a concurrent request could
overflow the LRU and shut that provider down with a zero span count, dropping
the routed trace. route_for now increments the open-span count in the same
locked critical section as the cache update and hands back an already-held
provider; every caller releases it once its span has landed. The lock also
makes the cache mutations safe under that same thread-pool concurrency.
* fix(otel): skip tenant routing on deferred pre_call
route_for ran before the recordable-parent check, so a thread-pool
pre_call still built or LRU-touched a tenant provider and could evict
an idle one even though the hold was released immediately and close
re-routed. Only route when the span actually opens
* add somethign
* Revert "add somethign"
This reverts commit 2f2cf84c5a049f0e287efd18a664479bd827fe2c.
* fix(otel): cap retired tenant providers draining open spans
* docs(otel): justify the retired-provider cap
* fix(cli): keep the refresh token in the OS keychain, not in token.json
`lite login --pkce` mints a refresh token that buys a fresh key from the
proxy on demand, so it is the credential just as much as the key is. Moving
the key into the keychain left it behind in ~/.litellm/token.json, where any
process running as the user can read it and renew the login for itself.
It now travels with the key: `save_cli_token` writes both into the keychain
entry, the token file keeps only metadata, and `lite logout` takes it out of
the file whether or not the keychain answers.
Upgrading finds one sign-in split across the two stores, the key already in
the keychain and the refresh token still on disk. That case rejoins the two
halves into a single entry before scrubbing the file, so the write never
replaces a live key with nothing, and a machine that refuses the scrub keeps
what it has rather than having the key rolled back out from under it.
* test: replace blind sleeps with deadline waits in callback and caching tests (#37660)
* test: replace blind sleeps with deadline waits in callback and caching tests
tests/local_testing/test_custom_callback_input.py slept a fixed 1-3s after
every call and then asserted the callback handler recorded no errors. Because
the handler only appends to `states` when a callback actually fires, an assert
of `len(errors) == 0` passes just as happily when nothing fired at all, so the
sleep was buying flakiness in exchange for a vacuous check. The async tests
were worse: `time.sleep` blocks the event loop, so the success/failure tasks
scheduled on it could not run before the assertion.
Adds tests/_wait_helpers.py with `wait_until` / `await_until`, which poll a
predicate against a deadline, and converts all 17 sites to wait on the thing
the test actually cares about (the terminal state landing in `states`, or the
patched log hook being called). The waits assert the callback fired, so these
tests now fail on a dropped callback instead of passing silently.
The three sleeps in test_caching_handler.py sat between `sync_set_cache` and
`_sync_get_cache`, both fully synchronous against a local in-memory cache, so
they are just deleted.
* fix(test): wait on the priming call's own logging in the cache-hit test
The 3s sleep in test_logging_async_cache_hit_sync_call was not waiting for the
cache write, which lands before the stream iterator is exhausted. It was
waiting for the priming call's success callback to drain, so the handler
installed right after it only ever sees the second, cache-hit call. Waiting on
a populated cache_dict let the priming call's still-pending log_success_event
reach the new mock, and the test then read cache_hit off the wrong payload.
Waits on the priming handler's own sync_success state instead.
…
TLDR
Problem this solves:
conftest copy.pyis a duplicate pytest never loadsHow it solves it:
User Flow
Before: someone reads a test, believes it runs, and it does not
tests/logging_callback_tests/test_sqs_logger.pylooking for what covers the SQS success pathtest_async_log_success_event_adds_to_queueat line 154 and read its assertionsAfter: the file contains only definitions that run
test_async_log_success_event_adds_to_queueasyncio.create_taskpytest --collect-onlyreports the same 416 tests it did before, because the deleted definitions were never among themRelevant issues
Linear ticket
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -vScreenshots / Proof of Fix
The claim is that these definitions never ran, so the proof is that removing them changes nothing pytest collects. The detector below walks every scope in the test tree and reports each name a later definition rebinds.
Shared setup, the four files touched:
Before (5290150)
The shadowed definitions exist and are unreachable
scripts/find_shadowed.pybeing the AST detector described above:After (c3545ee)
Nothing is shadowed any more
The collected set is unchanged
Same number, so no test was removed from the run. That is the whole safety argument.
The one keyless file still passes
Every test file still has a runner
Type
🧹 Refactoring
✅ Test
Caveats
Final Attestation