feat(model_prices): add gemini-3.1-flash-lite-image - #222
Open
emerzon wants to merge 616 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Adds the gemini-3.1-flash-lite-image model to LiteLLM’s pricing/catalog maps so image-generation spend is correctly computed and exposed via model metadata, including prefixed variants for Gemini and Vertex
Changes:
- Adds unprefixed,
gemini/, andvertex_ai/catalog entries with image-token pricing and cache read pricing - Introduces dedicated tests validating registration, routing, and cost calculation for image tokens
- Updates existing Gemini image-model test parameter lists to include the new model
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_litellm/test_utils.py | Extends Gemini image model reasoning-capability assertions to include the new model keys |
| tests/test_litellm/test_gemini_3_1_flash_lite_image_model_metadata.py | New tests covering cost-map registration, routing, and spend math for the new image model |
| tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py | Adds the model to existing Gemini image-generation cost fallback tests |
| model_prices_and_context_window.json | Adds pricing and metadata entries for the new model across unprefixed and prefixed keys |
| litellm/model_prices_and_context_window_backup.json | Mirrors the same model additions into the backup cost map used for local fallbacks |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+123
to
+124
| assert TOKENS_PER_1K_IMAGE * OUTPUT_IMAGE_TOKEN_COST == OUTPUT_COST_PER_1K_IMAGE | ||
| assert TOKENS_PER_1K_IMAGE * INPUT_COST == INPUT_COST_PER_IMAGE |
…eplaces a stored record
…annot be renewed A PKCE credential whose renewal is refused (for example after lite logout ran on another copy of it) used to fail lite auth print-token with the classic 'Token expired. Run lite login again' hint and no reason, while lite whoami already named lite login --pkce. fresh_api_key now reports why a renewal failed through a warn callback whenever no sibling rotation rescued it, the CLI prints that reason on stderr, and the expiry hint names the command that produced the credential. Both READMEs document the admin revocation semantics and the Redis precondition for refresh single use on several workers.
The cli group already resolves the stored key for the server it was pointed at, so print-token re-ran the renewal and, when the refresh token had been revoked, posted to /token twice and printed the reason twice. print-token now reuses the group's result whenever the stored record was issued for that server and no --api-key or LITELLM_PROXY_API_KEY took precedence, and only resolves the key itself when invoked bare for a different server.
The lite group resolves the stored key once for every command and renews a --pkce key on the way in. lite up then asked the token file again, so every start sent a second refresh to the proxy, and once the refresh token was burned the refusal printed twice. _ensure_fresh_login now reuses the key the group resolved when the group read it from the token file, and only re-reads the file after the interactive login it starts itself. Also covers print-token through the group with a renewing session in the tests
revoke_refresh_token discarded the single-use claim result, so a revocation that arrived while Redis was unreachable answered 200 and left the refresh token live. The token endpoint reported the same outage as invalid_grant "already used". The guard now reports first, replayed, or unavailable, and both endpoints answer 503 temporarily_unavailable for an outage (RFC 7009 section 2.2.1, RFC 6749 section 5.2), which the CLI surfaces as a one-line warning while keeping the key it has
…'s revocation A 503 from POST /revoke means the proxy could not write the single-use record, so clearing the local record left a live refresh token nobody could revoke and a hint to retry with nothing left to retry. lite logout now keeps the record, exits 1, and asks to be run again shortly. A refused or unreachable revocation still clears the record and warns as before, and a re-login that replaces a record keeps its existing warning because the new record already stands
…er_region_leak fix(router): routed deployment's own litellm_params beat forwarded auto_router marker params
…erriAI#37590) Nine workflow files existed only to make a single call to _test-unit-base.yml with a different test-path. Adding a shard meant adding a file; changing anything shared meant editing nine. One matrix caller replaces them, so a shard is now one entry. Check names are unchanged, which is the whole constraint. A reusable-workflow job reports as "<job name> / <inner job name>", so setting `name` to the shard id alone reproduces today's context strings exactly: the eleven the matrix produces are eleven of the twenty-three "/ Run tests" contexts the branch ruleset requires, matched string for string. No ruleset edit is needed and none should be made for this. Every matrix entry states its timeouts even where they equal the base defaults. An absent matrix key renders as an empty string rather than falling back, and an empty string is not a number, so a partially-specified entry would fail the call. tests/proxy_unit_tests keeps test-unit-proxy-db.yml. It is already a matrix and its shard-coverage guard reads that file by name, so folding it in belongs with generalising that guard into assert_ci_coverage.py rather than here. Its twelve shards are the remaining required contexts. test-unit-documentation.yml stays too: it does not call the base workflow.
…workflows/ (BerriAI#37586) The agent job's CircleCI glob collected `tests/agent_tests/**/test_*.py` and then piped it through `grep -v` to drop `local_only_agent_tests/`. `assert_ci_coverage.py` reads the glob but not the pipeline, so those two files looked covered and were invisible to the census. The glob now excludes them structurally and they carry an allowlist entry instead, which is a decision on the record rather than a hidden filter. The collected file set is unchanged: `tests/agent_tests/` holds exactly one CI-runnable test at the top level. `tests/scim_tests/` held a single JSON fixture and no tests, referenced from nowhere. `.github/workflows/` is for workflows. Both stray scripts move to `.github/scripts/` with their callers updated: the price-file updater is invoked by `auto_update_price_and_context_window.yml`, and the translation-report runner by `make test-llm-translation`. The audit listed the latter as orphaned, but Makefile line 317 still runs it, so it moves rather than being deleted. The rollout heads-up workflow was a deliberate one-shot for the agent-shin rollout. That rollout is done, the triage and auto-close workflows have been running daily since June, so the pre-flip warning window is long past. Its script and dedicated test go with it, and the sibling workflow-invariant test drops its entry.
* test: retire tests/old_proxy_tests, which holds no tests Twenty files named test_*.py, and pytest collects nothing from any of them: uv run pytest tests/old_proxy_tests --collect-only -q no tests collected, 16 errors in 114.17s They are manual snippets against a running proxy, written at module level with no test function, no assertion and no entry point, so the only thing the name buys them is a place on the coverage allowlist. Sixteen of the twenty cannot even be imported in this environment, wanting langchain, llama_index or google.api_core, and ten still point at 0.0.0.0:8000, which stopped being the proxy's default port some time ago. Nothing outside the directory refers to it apart from the allowlist entry, which goes with it. The other loose contents go too: five load_test_*.py scripts, a bursty variant, two committed log files, an essay fixture and a stray .js snippet. Allowlist paths 88 -> 68, test files 2422 -> 2402, and no job loses anything it was running. Recoverable from history if a snippet turns out to be someone's habit. * test: drop the retired old_proxy_tests paths from the coverage allowlist
…state debt (BerriAI#37588) * feat(ci): ratchet the test suite's zero-assert, mock-echo and global-state debt The suite's dominant failure mode is tests that cannot fail for the reason anyone would want them to. The testing-strategy audit measured five shapes of it, and nothing mechanical stops any of them from reproducing, so they keep reproducing. `scripts/check_test_quality.py` is an AST checker for those five, emitting the same `path:line: CODE message` contract as `scripts/check_type_discipline.py`: TQ001 a collectible test with no assertion of any kind TQ002 mock-echo, where every assertion only inspects the mock that was patched TQ003 sys.path.insert inside the test tree TQ004 raw `os.environ[...] =`, which leaks into whatever runs next TQ005 `litellm.<attr> =`, the process-wide leak the 491-line conftest undoes `scripts/test_quality_gate.py` caps each rule against test-quality-budget.json, seeded at exactly today's count, and fails only when a rule is both over its limit and higher than the base being merged into, so a change is blamed for what it adds and never for drift already in the base. `--update` lowers a limit by what a branch cleared, so the ceilings only ever fall. It runs in the existing required lint job, which means it enforces without a ruleset change. TQ001 follows assertions into helpers defined in the same module, transitively. Without that it flagged 111 tests in tests/e2e, the harness this program holds up as the reference, because that suite factors its assertions into shared helpers (`assert_auth_denied(result, ...)`). Following them leaves 25, all of which reach their assertions across a module boundary; those are grandfathered and documented rather than papered over. The seeded counts land within about 10% of the audit's independent numbers for every rule measured on the same subtree, which is the cross-check that the definitions here match the ones the audit pinned. * fix(ci): resolve test helpers per scope, not by bare name The helper walk keyed every function in a module by its bare name, so two same-named helpers in different classes collided and the last one parsed won. A test calling `self._check()` could be cleared by a `_check` belonging to a different class, or flagged because of one. Resolution is now scoped: a bare name looks up the module-level functions, and `self.<name>` looks up the enclosing class's own methods and no other class's. Recursion is tracked by function identity rather than by name, so the cycle guard cannot be confused by the same collision. This surfaced one real zero-assert test that a same-named helper elsewhere had been clearing, so TQ001 seeds at 750 rather than 749. The test module has to register itself in sys.modules before exec_module: `@dataclass(slots=True)` rebuilds its class through `sys.modules[__module__]`, and Scope fails to construct without it. Recorded at the call site, since it reads like avoidable global mutation otherwise. * fix: register test-quality-budget.json with the ratchet alarm The repo keeps one census over its budget files: every *-budget.json on disk must appear in DEFAULT_BUDGETS, or its ceilings can be raised with no signal. tests/test_litellm/test_budget_ratchet_check.py asserts that set equality and caught the new budget on the way in. Registering it also turns the alarm on for TQ001-TQ005, so a later PR cannot quietly raise a test-quality ceiling. The file already uses the {limit: N} schema the ratchet reads, so no other change was needed.
…cli_oauth_pkce feat(proxy): native CLI login with OAuth authorization code + PKCE
…prompt_cache_breakpoint feat(prompt-caching): map cache_control_injection_points to OpenAI prompt_cache_breakpoint on GPT-5.6+ targets
…e_silent_hang fix(realtime): bound Vertex credential resolution and make realtime failures loud
…pic_prompt_cache_key fix(anthropic): map metadata.user_id to prompt_cache_key on the /v1/messages bridge
…credentials fix(passthrough): resolve vertex live credentials from db model deployments
…s_without_prompt_id fix(prompt_management): don't route no-prompt_id requests to prompt managers that can't run them
…iAI#37591) Python binds a name once per scope, so when a module or class defines the same test twice only the last one exists. The earlier definitions are unreachable: pytest never collects them, and nothing that references them can fail. A sweep in August cleared nine of these. Five have appeared since, which is the argument for a rule rather than another sweep. Each survivor is the better version, so nothing is lost. The two SQS logger twins additionally stub `asyncio.create_task`, which the shadowed copies did not. The cost-calculator duplicate is a two-line stub that also takes a `model_item` parameter no fixture supplies, so it could not have run even unshadowed. The two `test_prompt_caching` bodies are both `pass`. Collecting the four files reports 416 tests before and after. `tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a file named exactly `conftest.py`, nothing imports this one, and the space in the name says what it was.
…iAI#37593) tests/proxy_unit_tests had a 30-line YAML parser inlined in its workflow that failed the run when a test file there belonged to no shard. tests/test_litellm is sharded the same way, with no catch-all bucket, and had no such guard: a new directory under it (or under its proxy subtree) is collected by nothing and runs nowhere, and the coverage census cannot see it because a token like tests/test_litellm/test_*.py already answers 'yes, that tree runs'. The two questions differ. The census asks whether a file runs at all, so an ancestor path standing in for everything beneath it is a fine answer. Shard assignment asks which shard owns a child, and there that same ancestor path is precisely the bug. _token_covers keeps the first meaning; _token_names adds the second, and the guard now walks a list of sharded trees rather than one hardcoded directory. Both read the same test-path keys, so there is one workflow parser. A directory needs a shard when it holds a test file, not when it is named test_*. That drops the hardcoded test_configs exception and keeps fixture directories like expected_fine_tuning_api out on their own merits. The job keeps its name and its workflow, since assert-shard-coverage is a required status check on litellm_internal_staging. Verified red-first: a planted directory under tests/test_litellm, a planted directory under tests/test_litellm/proxy, and a planted file under tests/proxy_unit_tests each fail the guard, while a fixture-only directory does not. 327 children across the three trees are assigned today.
…#37389) Stacked on the multi-key shadow eval backend. The key picker becomes a paginated multi-select with chips, built on the base-ui combobox chips primitives, with the pagination and debounced-search logic extracted into a shared usePaginatedCombobox hook that PaginatedSearchSelect now also uses. The detail view gains a per key table showing each key's own status, judged turns against its budget, and win rates from the by_key slice, and the job headline pluralises to "N keys" for multi-key jobs
…glm-5-2 alias Mistral's live /v1/models reports max_context_length 1048576 and capabilities.reasoning true for zai-glm-5-2, and its docs price cached input at $0.14/M. Without cache_read_input_token_cost LiteLLM billed every cached prompt token at $0, so a repeat request against a 21k-token cached prefix logged $0.0000135 instead of its real cost. Mistral also serves the model under the short glm-5-2 name, which had no cost map entry at all and therefore no pricing, so add it alongside.
Base landed the native CLI OAuth + PKCE login, which added its own token storage and a silent refresh that wrote the key straight to token.json. This branch had already moved that secret into the OS keychain, so the two had to be joined rather than picked between. auth.py now keeps one pair of record helpers, load_token and save_token, that read and write through the vault and hand the PKCE layer the plain mapping it works with. fresh_api_key and revoke_stored_credential get vault-bound save and reload callables, so a renewed key is stored in the keychain like any other and a sibling process's rotation is still seen. login goes through _replace_stored_token on both paths, so the credential it replaces is revoked on the proxy and the user is still told where the new one landed. logout revokes first, then reports what the clear actually managed to do.
* fix(ui): make dark-mode form controls visible Two dark-mode defects left form controls without any visual boundary or fill. `--input` and `--border` share one value in `.dark`, oklch(0.309), which resolves to rgb(48,48,48). Against `--background` (33) that is a 15-step stroke, and against `--popover` (42) it collapses to 6 steps out of 255, so a control inside any dialog is effectively undrawn. The controls also use `bg-transparent`, so there is no fill cue either and only the placeholder text renders. Measured 1.09:1 against the dialog surface where WCAG 1.4.11 asks for 3.0:1 on the boundary of a user interface component. Splitting `--input` off at oklch(0.56) restores 3.07:1 without touching `--border`, which stays where it is because it draws decorative separators rather than control boundaries. 91 controls across 19 routes were measured at the collapsed value, every one with an identical stroke and surface, so a single token covers all of them. Separately, `@tailwindcss/forms` paints a white fill on every bare control. The block above already neutralises that for `combobox-chip-input`, but its audit covered `components/ui` only, and hand-rolled controls elsewhere still render white on a dark page: typed text lands at 1.11:1 and native selects at 2.19:1 on `/model-hub-table`, `/playground`, `/guardrails`, `/mcp-servers` and `/models-and-endpoints`. Tracking `--background` fixes those at 14.51:1 and 7.34:1. Light mode is unchanged by both. The token edit is scoped to `.dark`, and `--background` in `:root` is the same white the plugin was already painting, verified control-by-control on a dev server: backgrounds stay rgb(255,255,255) and ratios stay 20.13:1 and 4.84:1. * fix(ui): keep the combobox chip input transparent under the bare-control fill The new base rule matched at (0,2,1) while the combobox chip-input override sits at (0,1,0), so ComboboxChipsInput lost its transparent background and painted an opaque page-colored rectangle inside the chips container, which carries its own bg-transparent / dark:bg-input/30 fill. Folding the exclusions into one :not() list adds the chip input and drops the selector to (0,1,1). Every @tailwindcss/forms base selector is wrapped in :where(), so it lands at (0,0,1); (0,1,1) still outweighs it and bare inputs, textareas and selects keep the fill this PR gives them.
… 70% step (BerriAI#37649) * fix(ui): give status colours a readable foreground and drop the muted 70% step The four status tokens are lightened for dark mode, which is correct when they are used as text and wrong for the 27 places that use them as a background under `text-white`. Every one of those passes in light and fails in dark: success 1.78:1, warning 1.72:1, info 2.64:1, destructive 2.89:1. The cause is not 27 authoring mistakes, it is that no `--success-foreground` and no sibling ever existed, so `text-white` was the only thing available to write. Adding the four companions and registering them in `@theme` makes the correct pairing expressible, and the call sites then read `text-success-foreground` instead of a hardcoded colour. Dark lands at 9.98, 10.31, 6.72 and 6.15. Light is deliberately pure white rather than the near-white the other `-foreground` tokens use, so the four ratios stay at exactly the 4.95, 5.03, 5.25 and 4.77 they are today instead of drifting down to 4.73, 4.81, 5.02 and 4.56. Separately `text-muted-foreground/70` measures 2.75:1 on a light page and 4.31:1 on a dark one, so the same 183 occurrences fail AA in light and sit under it in dark. Dropping the opacity step takes them to 4.84:1 and 7.34:1. The identical step on the placeholder base rule goes with them, which is what put every input's placeholder at 2.75:1 in light. Residual, not addressed here: `text-muted-foreground` over `bg-muted` reaches 4.39:1 in light, still short of 4.5. Closing that needs `--muted-foreground` itself to move, which changes every secondary label in the product and is a design call rather than a defect fix. * fix(ui): finish the status-foreground swap and repoint no-op muted hovers Four sites still forced text-white on a status fill because the class sat on a child element rather than on the filled container, so the earlier sweep did not reach them. The compliance quick-test bubble was worse: it paired bg-info with text-success-foreground and its paragraph kept text-white on top, so the dark-theme contrast the PR set out to fix was still reachable there Dropping the /70 step also turned 21 existing "text-muted-foreground/70 hover:text-muted-foreground" pairs into hovers that change nothing, which local/no-noop-hover-variant flags as an error. The affordance was "brighten on hover", so these now hover to text-foreground, matching the 74 places that already spell it that way The remaining churn is prettier reflowing the handful of lines whose length changed, since the token names are longer than text-white * fix(ui): let the approve/reject confirm button pick the token its fill uses Both submission review dialogs put text-success-foreground on the shared button class while the fill below it swings between bg-success for Approve and bg-destructive for Reject, so Reject drew a success token over a destructive fill. The two tokens resolve to the same value today, so nothing looks wrong, but the pairing only holds by coincidence and would break the moment either token moves. Moving the token into the branch makes it track the fill * fix(ui): drop the last 70% placeholders, still live on the legacy utility Four inputs spell their placeholder colour with Tailwind's older placeholder-<colour> utility rather than placeholder:text-<colour>, so the sweep that dropped the 70% step passed over them. Tailwind 4.3 still emits that utility, and utilities sit after base in the layer order, so those four kept overriding the new input::placeholder rule and kept rendering at 70% in dark mode, which is the contrast failure this PR set out to close They now spell it the same way as the three placeholders the PR already converted, which both removes the step and settles on one spelling
…#37651) * fix(ui): make inline styles and code blocks follow the theme Two families of colour that a stylesheet never gets to see, so dark mode could not reach them. The log details drawer paints most of its chrome through React inline style objects holding raw hex: #f0f0f0 borders, #fafafa panels, #262626 body text, the antd-era role accents on message cards, and a green/red guardrail summary pill. Inline styles win over any class, so the drawer stayed light on a dark page. Every one of those literals becomes the var(--color-*) it was already imitating, which costs nothing in light mode and now tracks the theme. The guardrail pill keeps its layout inline and moves its three colours onto the success and destructive tokens the rest of the dashboard uses. The eleven code blocks pass a prism stylesheet as a prop, so the theme has to be picked in JavaScript. There is no dark-mode toggle in the app yet, only the `dark` class the design system keys off, so useIsDarkMode subscribes to that class through useSyncExternalStore and useSyntaxTheme swaps in oneDark when it is set. Each call site keeps the light stylesheet it already had, including the two that were relying on the prism default and now name it, so light mode is unchanged everywhere. Six of those call sites were casting the stylesheet to `any` or re-declaring its type to get past the prop signature; the hook returns the right type, so the casts are gone. * fix(ui): let the markdown code renderer keep its own syntax theme The three ReactMarkdown code renderers spread the remaining code element props after style, so the incoming style attribute widened the prop type and next build's type check rejected the hook's return value. The old `coy as any` cast hid the same conflict. Spreading first lets the explicit props win, which is what every one of these call sites meant. * test(ui): cover the dark-mode hooks that pick a syntax stylesheet useIsDarkMode carries the only real logic in this change: an external store over the root element's class list. Cover the three things that can regress, the class already being present at mount, the class being toggled later, and the observer being disconnected on unmount, then cover useSyntaxTheme handing back the caller's own stylesheet in light mode and oneDark in dark. The assertions are on which stylesheet object comes back, by identity, not on any colour it holds. * refactor(ui): drop the last stylesheet cast in the chat code renderer This was the one markdown code renderer still spreading the code element props over its style, so an incoming style attribute would have won over the theme, and the cast on the spread was what kept that compiling. Spreading first lets the theme win and the cast go.
* fix(ui): move the policy flow builder onto theme tokens The flow builder carried its own private palette: 126 raw literals across a 1644-line file, hardcoded into React inline style objects and SVG presentation attributes. Inline styles beat every class, so the whole page, its version sidebar, its step cards and its test panel stayed light no matter what the theme said. Each literal now resolves through the token it was already imitating. The greys map onto card, muted, border, muted-foreground and foreground; the indigo and blue accents onto info; the pass, fail and API-failure accents onto success, destructive and warning; and the pale status washes become a color-mix of the same token so they track it in both themes. Six icons carried their colour as an SVG presentation attribute, where custom properties do not substitute, so those switch to currentColor with the token set alongside. Light mode is not byte-identical, and that is the point: the file stops keeping a second palette. Of the mappings, card, muted and border land on the exact same rgb they had, covering most of the file. The rest snap to the dashboard's canonical shade, which mostly means slightly darker text and deeper status colours: the gray-400 labels pick up real contrast, the soft red on the fail icon becomes the destructive red every other failure indicator uses, and the indigo accent becomes the blue that info resolves to. Verified in a browser on both themes. In dark mode nothing on the page paints a light background any more; the six that still do are shadcn's inverted primary buttons and badges, which are meant to. * fix(ui): token the flow builder test textarea fill The quick-chat textarea is the one bare form control left in the file, so the @tailwindcss/forms base layer still paints it `background-color: #fff`. The inline style overrode the plugin's border but not its fill, which left a white box inside the now-dark test panel, and its text inherits the near-white foreground, so the typed message was invisible in dark mode. Pin both halves of the pair on the element the plugin styles: the card token it sits on, and the foreground token it was already inheriting.
…I#37598) * test: settle three allowlist entries that were open questions The allowlist is meant to hold decisions, not deferrals, so an entry reading 'needs moving' or 'referenced by no job' is a gap wearing an exemption. These three each get an answer. The two prompt-factory tests move into the mirror, which is what their own entry said they needed. Both were passing the whole time, so the 23 tests they hold start running and the entry goes away rather than getting reworded. test_aio_http_image_conversion.py is not a test. It fetches live image URLs, times aiohttp against httpx, prints the ratio, and asserts nothing, and pytest cannot collect it because its functions take arguments rather than fixtures. Running it beside its siblings would buy CI a network dependency and a number nothing reads, so it stays exempt with that written down. test_litellm_proxy_extras_utils.py stays exempt with a measured reason. 24 of its 28 tests pass; the 4 in TestMigrationSQLIdempotency fail because nine migrations from 2026-04 onward use bare CREATE TABLE, ADD COLUMN and CREATE INDEX where that file requires guarded forms. The convention eroded quietly precisely because the test enforcing it has never run. Wiring it up is blocked on what to do about those migrations, and editing them is not the answer, since Prisma checksums an applied migration and a changed one breaks migrate deploy for existing installs. Allowlist entries 10 -> 9, paths 88 -> 86. * docs(ci): correct the migration count in the proxy-extras allowlist reason
… see (BerriAI#37793) The sweep ran unbounded whenever no config.yaml deployment was present, deleting the day's sentinel rows for deployments absent from the run's own view. A written charge records capacity that was reserved, so the only rows a run may retract are the ones it can reassess: a deployment it scanned and then declined to charge, because the window closed or the PTU config was removed. It is now always bounded to the ids it scanned
…itellm_/revert-pr-37554-migration-5af8ea
Adds ownership of litellm-proxy-extras/litellm_proxy_extras/migrations so schema migration changes get a review request. Also repoints the two existing entries at @yuneng-berri. GitHub's CODEOWNERS validator was rejecting @yuneng-jiang as an unknown owner, which left the /ui/ and _experimental/out/ rules inert.
…e-owner-c09d78 chore(codeowners): own the proxy-extras migrations directory
…the first A name bound twice keeps only the second binding. In `tests/` that is nearly always a repeated import, harmless but misleading, and the same rule is what catches the cases that are not harmless: a local that shadows an import the module still calls, and a second `def test_x` that quietly replaces the first. 311 of the 344 sites were repeated imports and came out with ruff's own fix. The remaining 33 needed a decision. Four modules imported a name they never used because a local definition below already shadowed it. Two comprehensions bound `call` over `unittest.mock.call`, which those modules import and use. One test rebound the two module handles its nested reload closure had captured. One class attribute shadowed an unused `status` import. The load-test fixtures move to a conftest, which is how pytest is meant to share them, so the test module no longer imports three fixture names it never calls. The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that fixture by name before the body runs, so the parameter never shadows anything.
…4-migration-5af8ea revert(spend-logs): drop the endTime backfill migration for spend log timestamps
…t server access (BerriAI#37865) The dcr_bridge oauth_delegate connect flow completed for a signed-in user with no litellm-side grant to the target server: every leg returned 200, the DCR client showed connected, and tools/list then fail-closed to an empty list with the upstream never contacted (BerriAI#36358). The authorize leg now admits the user the way MCP egress will (same reload_admitted_user constructor, same get_allowed_mcp_servers resolver) and refuses with an RFC 6749 access_denied redirect naming the remedy, before any upstream OAuth runs or an envelope is minted. Availability faults (5xx) propagate; unknown or deactivated users deny fail-closed Promotes MCPRequestHandler reload_admitted_user to public: it already had a cross-module consumer in ui_session_utils, and this gate adds a second, so the private name no longer reflected its use. Ratchets the freed reportPrivateUsage budget headroom down
…xceededError (BerriAI#37862) Mantle reports context overflow as a structured 400 validation_error rather than the plain-text patterns Bedrock itself uses, so callers such as Claude Code that key reactive compaction off the phrase "prompt is too long" never see it. Detect the pattern and normalize the message to that phrase.
…te_definitions test: enforce F811 so a duplicate definition cannot silently replace the first
…etadata (/v1/messages, Claude Code) (BerriAI#36889) * fix(guardrails): resolve guardrail pipelines from the canonical metadata bucket Policy-resolved pipelines are stored in litellm_metadata on routes like /v1/messages, but the pre_call reader fell back to the caller-supplied metadata field first, so a request that sends its own top-level metadata (Claude Code sends metadata.user_id) skipped every pipeline-managed guardrail. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): drive the pipeline regression through a registered guardrail Exercise the real executor with a guardrail in litellm.callbacks instead of patching PipelineExecutor.execute_steps at class scope. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): read pipeline state from the bucket the policy engine wrote Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): type the policy pipeline state accessors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): annotate policy pipeline state casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…erriAI#36981) * fix(vertex_ai): convert messages to contents in gemini count_tokens acount_tokens passed contents=None to the Vertex Gemini countTokens endpoint when called with messages=, causing a silent zero token count. The Gemini branch of VertexAITokenCounter.count_tokens never read the messages parameter, so the request body was {"contents": null}, which Vertex accepts with HTTP 200 and no totalTokens field. Convert messages to Gemini contents format using the existing _gemini_convert_messages_with_history helper when contents is None. Treat a response without totalTokens as a failure so the caller falls back to local token counting instead of returning a silent zero. Fixes BerriAI#36921 * style: apply ruff format to common_utils.py Resolves lint CI failure on PR BerriAI#36981. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: suppress LIT002 on messages fallback for Gemini token counter Adds `# mutable-ok:` suppression on the `messages or []` fallback passed to `_gemini_convert_messages_with_history`. The [] is a None-fallback; the helper signature requires list[AllMessageValues], so a tuple would violate the type contract. Resolves type-discipline-budget CI failure on PR BerriAI#36981. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: suppress reportPrivateUsage on _gemini_convert_messages_with_history import Adds `# pyright: ignore[reportPrivateUsage]` on the import of the shared `_gemini_convert_messages_with_history` helper. The function is already used by gemini/chat, context_caching, and vertex_and_google_ai_studio_gemini; reusing it here avoids duplicating the OpenAI-to-Gemini message conversion. Resolves basedpyright budget CI failure on PR BerriAI#36981. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A test that asserts on the error inside its own except block passes when the call stops raising, because nothing runs the handler. That is the exact case the test exists to catch, so the regression lands green. Rewrites all 111 such blocks into pytest.raises, which fails when the call succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
…oyment name resolves from the cost map (BerriAI#37869) * fix(router): don't log 'Could not identify azure model' when the deployment name resolves from the cost map get_router_model_info already falls back to resolving the azure deployment's model name against the model cost map when base_model is unset — and for deployments named after real azure models (e.g. azure/gpt-4o) that resolution returns correct max tokens and costs. The unconditional ERROR was therefore spurious for exactly the deployments that need no operator action, and on busy proxies it logs thousands of times per day per multi-deployment group. Log at debug when the fallback entry carries usable limits/costs (membership alone is not enough: Router init auto-registers every deployment name as a zeroed stub), keep the ERROR otherwise. Fixes BerriAI#33172 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(router): use consistent positive checks in azure base_model fallback gate Review follow-up: token-limit fields used 'is not None' while the cost field used '> 0' — a cost-map entry explicitly storing 0 limits could suppress the error log without carrying usable resolution data. All three checks now require a positive value. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(router): trim fallback gate comment and reuse the shared local_model_cost_map fixture --------- Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…me picker (BerriAI#37871) * fix(ui): drive auto-router usage from the shared cost-optimization time picker * fix(ui): extend a live-ending benchmarks range to the current UTC day
* feat(ui): add per-key Savings tab to key detail page
Adds a "Savings" tab to the key detail view, showing the same four metrics
and time-series chart as the proxy-wide Cost Optimization view, but scoped
to a single API key.
For org admins, the tab shows the key's full savings across all requests.
Non-admins see only their own requests on the key, with a scope note
explaining the limitation.
Root cause: userDailyActivityCall and userDailyActivityAggregatedCall
never forwarded an api_key query parameter to the backend, even though
both handlers already accept and filter by it.
Changes:
- networking.tsx: Add optional apiKey param to both daily activity call
wrappers (appended to variadic options tuple for backward compatibility).
- costOptimizationUtils.ts: Extract shared metrics helpers (compressionOf,
cachingOf, autorouterOf, savedTokensOf, cacheHitRatio) and shortDate
so both UsageTab and KeySavingsTab use the same formulas and prevent
divergence.
- useDailyActivityRange.ts: Refactor into useScopedDailyActivityRange(
accessToken, scope: {userId, apiKey?}) for reuse-by-parameter unbundling.
Role resolution stays at the entry point (useDailyActivityRange), not in
a scoped caller. Update test expectations for new 6-arg tuple.
- UsageTab.tsx: Simplify by importing extracted helpers and SummaryCard
component instead of defining them inline. No behavioral change.
- key_info_view.tsx: Insert "Savings" tab trigger between "Overview" and
"Settings"; wire TabsContent to new KeySavingsTab component with lazy
mounting (no keepMounted) to defer daily-activity fetch until tab opened.
- NEW: components/shared/SummaryCard.tsx — Shared presenter for four-tile
summary row (label + value + hint + optional info popover). Extracted
from UsageTab so both surfaces show identical tile layout without CSS
divergence.
- NEW: components/templates/KeySavingsTab.tsx — Per-key view with admin/
non-admin scope branching, empty-state messaging, same chart toggles
and info popovers as UsageTab.
- NEW: components/templates/KeySavingsTab.test.tsx — 7 tests covering mount,
loading state, empty state, scoping, and scope-note visibility.
Authorization: No new permission check. Both backends gate api_key filter
by the same user role check that governs the request itself. Non-admins
must send their own user_id and can only see their own keys.
Tests: 6121 pass (1 pre-existing failure unrelated to this change).
Prior art / collision note:
- PR BerriAI#37570 (budgets tab) lands in same TabsList hunks as "Savings" tab,
but different tab names so conflict trivial if both merge.
- PR BerriAI#37659 (my own) adds progress/cancelled/cancel to DailyActivityRange,
but this PR uses stable three-field interface from staging.
* fix(ui): scope spend view by the backend's admin-view contract, not all_admin_roles
Greptile flagged org admin handling on the key savings tab. The live bug it
described does not fire today: useAuthorized supplies session-role labels and
all_admin_roles only carries the raw org_admin spelling, so an org admin was
already scoped. That safety was accidental, so replace the predicate with
spendScopeUserId / hasProxyWideSpendView in utils/roles.ts, mirroring the
backend's user_api_key_has_admin_view (proxy admin and admin viewer only, org
admin excluded in both spellings), and use it in both useDailyActivityRange
and KeySavingsTab
Reclassify the KeySavingsTab render test as an integration test per the
repo's unit/integration split, move scope-resolution coverage to roles.test.ts
as a full role matrix, use real session-role values instead of raw ones, and
assert tile totals against non-empty metrics. Replace the nested ternary in
the chart body (frontend-lint error) with flat conditional rendering
* fix(ui): show auto-router savings as the fourth key-savings tile
Cache hit rate had displaced auto-router savings from the fourth slot,
diverging from the org-wide Cost Optimization page's tile order. Match
it: Total / Compression / Prompt caching / Auto-router, with cache hit
rate as a fifth tile.
* fix(ui): drop cache hit rate from the key savings tiles
Keep the four tiles this page is meant to show: total, compression,
prompt caching, and auto-router savings.
* fix(ui): stop an empty api_key from widening a key-scoped activity read
The paginated and aggregated daily-activity wrappers disagreed on an
empty filter value: the paginated one appended it, the aggregated one
coerced it to undefined with || and dropped it. Since the aggregated
call is the one tried first, an empty key hash would have silently
turned a key-scoped read into a proxy-wide one and reported every
key's savings as this key's. Use ?? so both send the filter through
and it matches nothing instead.
* style(ui): satisfy prettier and the inline-object lint rule in key savings tests
* refactor(ui): drop the cacheHitRatio extraction left over from the removed tile
* fix(ui): pass daily-activity filters raw so both transports agree at the null boundary
* refactor(ui): share the savings tiles and totals between both surfaces
The per-key Savings tab and the proxy-wide Cost Optimization tab carried a byte-identical
four-tile block, three long metric-definition strings included, and five identical useMemo
totals. Both now render SavingsTiles and total through useSavingsTotals, so the donut cannot
slice numbers the tile above it disagrees with.
* docs(ui): say request, not mount, in the savings tab comment
The comment claimed mounting eagerly would fire the rollup sweep, which reads as a claim about
the bundle. Only the request is deferred; the module ships with the key page either way.
* test(ui): pin the daily-activity args array against the real caller signatures
The sibling unit test mocks networking, so it checks the positional array against itself and
stays green when the array and a networking signature drift apart. Swapping user_id and api_key
in the aggregated signature alone passes there and fails here on user_id=hash-abc.
* style(ui): hoist the daily-activity query options out of the call argument
The four-property object literal tripped local/no-large-inline-object-arg. The violation predates
this branch, which only moved the line into the annotated range, and the rule count drops 550 to 549.
…ot the whole client (BerriAI#37863) A ConnectionError/TimeoutError on one node of the async Redis Cluster client made redis-py tear down every node's connections and force every other concurrent caller through the shared reinit lock, turning one client-side timeout under event-loop saturation into a proxy-wide latency spike while Redis itself stayed healthy. Confirmed live against a local 3-master cluster: pausing one node made 100% of concurrent commands to the other two, untouched nodes stall for the full pause duration; after this change, zero. LiteLLMAsyncRedisCluster overrides only the ConnectionError/TimeoutError branch of _execute_command to reset the one node that failed, mirroring what a plain non-cluster Redis client already does when a pooled connection errors. Every other branch (MOVED, ASK, CLUSTERDOWN, slot-not-covered) is unchanged, since those already carry real evidence the topology changed.
…in_except test: reject assertions on a caught error inside except (ruff PT017)
…elling (BerriAI#35576) * fix(mcp): strip root_path before matching the per-server MCP route spelling The 401 challenge for a gateway-managed oauth2 MCP server advertises the protected-resource metadata URL in the spelling the client connected on, so a strict RFC 9728 section 3 client lands on a document whose `resource` equals the URL it actually called. That spelling test compared `_original_path` against the root-relative `/{server}/mcp` shape, but `_original_path` and `scope["path"]` are raw request-line paths that still carry the deployment's `root_path` On a SERVER_ROOT_PATH deployment the prefix therefore made the legacy test fail and every request fell through to the standard `/mcp/{server}` branch. A client connecting on `/litellm/github/mcp` was pointed at the standard-pattern document, which serves `resource = {base}/litellm/mcp/github`; that is not the URL the client called, so a strict client aborts discovery before the MCP request fires Route the path through `get_route_relative_request_path` first, which removes `root_path` on a segment boundary the same way `litellm.proxy.auth.auth_utils.get_request_route` already does for the rest of the MCP auth path, so `/litellmfoo` is not truncated under `root_path=/litellm` * fix(mcp): make the gateway-managed 401 challenge root-path aware The gateway-managed authorization_code challenge in process_mcp_request built its AS-metadata URL from two root-path-unaware pieces: - it matched the caller's spelling against `scope["_original_path"]`, a raw request-line path that still carries the deployment prefix, so on a SERVER_ROOT_PATH deployment the `/mcp/{server}` branch never matched and every request fell through to the legacy one-segment form - it hardcoded `/.well-known/oauth-authorization-server` without the root-path segment the discovery route decorators bake in, so the URL 404'd under a sub-path deployment regardless of which branch was taken Route the spelling match through get_route_relative_request_path and the well-known root through well_known_root_suffix, the same two helpers the discovery route registrations derive their paths from, so the advertised URL cannot drift from the route that serves it. Root-mounted deployments are unaffected: both helpers are no-ops when SERVER_ROOT_PATH is unset. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…acks (BerriAI#37894) The auto-router savings figure was computed only inside the spend-update writer, downstream of where logging callbacks consume the standard logging payload, so Datadog-style callbacks never received it. Compute it once in the payload builder, stamp it as a top-level payload field beside cost_breakdown, thread it into the spend log metadata, and have both spend-writer call sites read the recorded value with recomputation as the fallback for rows written before the field shipped. Internal sub-calls (classifier, shadow eval) are never stamped, and a caller-forged metadata value is discarded by the unconditional overwrite. Resolves LIT-5973
* fix(proxy): group Codex turns under one session id
Codex puts its conversation uuid in an unprefixed `session-id` header
(`session_id` on builds before the codex-api split), so
`get_chain_id_from_headers` never matched it: the `x-<vendor>-session-id`
regex requires an `x-` prefix. Codex also sends no request metadata the
Anthropic `metadata.user_id` path could parse and no traceparent, so every
turn fell through to a freshly generated per-call trace id and landed as its
own row in the logs.
Read the unprefixed `session-id` / `thread-id` (and the older `session_id` /
`conversation_id`) names, gated on the Codex user agent. Those names are
generic enough that an unrelated client could send one meaning something
else, and colliding values across callers would merge their traces, so the
bare-header path stays Codex-only.
* fix(proxy): match every first-party Codex originator
`is_codex_user_agent` tested `startswith("codex_")`, but the Codex TUI sends
`codex-tui` with a hyphen, and often bare with no version at all. Real values
seen in the wild are `codex-tui` and
`codex-tui/0.149.0 (Mac OS 26.5.1; arm64) ghostty/1.3.1 (codex-tui; 0.149.0)`.
codex-rs's own `is_first_party_originator` lists `codex-tui`, `codex_cli_rs`,
`codex_vscode` and a `Codex ` prefix, which agree only on the `codex` stem.
Match that stem plus a separator so no spelling is missed and an unrelated
`codexfoo` client still is. This also repairs the pre-existing gap where
`should_auto_drop_params_for_agentic_cli` (called on the request path at
litellm_pre_call_utils.py:2049) never fired for the Codex TUI.
* refactor(proxy): take headers as a read-only Mapping in the Codex session lookup
…tion (BerriAI#37122) * fix(logging): preserve uvicorn color_message args during secret redaction SecretRedactionFilter clears record.args after substituting record.msg, but uvicorn's colorized formatter re-renders the separate color_message extra field against record.args at emit time. With args cleared, uvicorn prints the raw "%s://%s:%d" template instead of the actual startup URL whenever output goes to a TTY (colors on). * fix(logging): narrow color_message fallback to TypeError Bare except-Exception-pass on the color_message substitution pushed the BLE001 and S110 strict-rule budgets over their ceiling. Narrow to the one exception the %-format can actually raise and give it a real fallback instead of silently swallowing it. * refactor(logging): move color_message substitution into a helper The two record.color_message stores put LIT011 over its ceiling. Building the value in a pure helper leaves one store, marked rebind-ok since scrubbing a record in place is the logging.Filter contract. Also pins the ordering that makes the substitution safe: it has to run before args are cleared, which puts it before the extra-field loop that redacts the result, so a secret arriving through record.args is still scrubbed out of color_message.
…cing feat(llm): add moonshot/kimi-k3 to model prices and context window map
…itellm_add_gemini_3_1_flash_lite_image
…provider feat(providers): add SCX.ai as a JSON-configured OpenAI-compatible provider
…apabilities The three lite-image keys landed on the deploy branch separately while this branch was open, so merging left every key defined twice in both price maps. The merge is clean as text and the file still parses, but JSON keeps the last occurrence of a repeated key, so the first copy's supported_endpoints, supported_modalities and supports_system_messages were being dropped without any error. Each key is now one entry, placed next to its gemini-3.1-flash-image sibling rather than at the end of the file. supports_reasoning goes to false on all three, matching every other Gemini image model. Leaving it off is not neutral: _supports_factory falls through to the vertex_ai provider config, which answers true, and reasoning_effort then gets forwarded to an image endpoint that rejects it. That was fixed for the rest of the family in 75dd70a and these entries had drifted back. Also fills in what the entries were missing against Google's published pricing: the Vertex implicit cache read rate, batch rates on the Vertex routes, and the pdf/video input flags. The two overlapping test files are folded into one, and the price map suite grows a duplicate-key guard so the next clean-but-lossy merge fails loudly.
…itellm_add_gemini_3_1_flash_lite_image
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
gemini-3.1-flash-lite-imageis missing from the model catalogHow it solves it:
gemini/, andvertex_ai/catalog entriesUser Flow
Before: a proxy admin serving Nano Banana 2 Lite sees image traffic logged at $0
gemini/gemini-3.1-flash-lite-image{"model":"nano-banana-2-lite","messages":[{"role":"user","content":"Generate a picture of a banana wearing sunglasses"}],"modalities":["image","text"]}usage.completion_tokens_details.image_tokensof 1120spend=0.0After: the same image call is billed at Google's published Lite image rates
gemini/gemini-3.1-flash-lite-image9 * $0.25/1M + 1120 * $30/1M + text tokens * $1.50/1Mmax_output_tokens: 4096,output_cost_per_image_token: 3e-05, andoutput_cost_per_image: 0.0336Relevant issues
Fixes BerriAI#31945
Fixes BerriAI#32650
Supersedes BerriAI#31946 and BerriAI#36113
Linear ticket
Resolves LIT-5286
Pre-Submission checklist
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
This environment has no Gemini or Vertex credentials, so I could not recapture a live billed image call. Reviewer can replay the same e2e used on BerriAI#36113 after starting the proxy with a
nano-banana-2-lite -> gemini/gemini-3.1-flash-lite-imagedeployment andLITELLM_LOCAL_MODEL_COST_MAP=TrueExpected spend for a 1K image is
1120 * $30/1Mplus text and thinking tokens at$1.50/1Mand input at$0.25/1M. On BerriAI#36113 a live AI Studio call billed$0.03415725for 9 input, 370 text, and 1120 image tokens, which matches that mathCommit for this branch:
0e3f52a4c0Type
🆕 New Feature
Caveats (if any)
gemini/rpm/tpm copied fromgemini-3.1-flash-imagebecause Google only publishes tiered limitssupports_reasoningis false to match the other Gemini image catalog rowsFinal Attestation