chore(ci): promote internal staging to main - #37913
Merged
Merged
Conversation
#30670) The Request Logs "Key Hash" column binds to the SpendLogs metadata user_api_key field, which _get_spend_logs_metadata copied verbatim from request metadata without hashing. The top-level api_key column only hashed values starting with sk-. So a non-sk- passthrough provider key, or a Bearer-prefixed key on paths that do not strip it, could be persisted and shown in plaintext. A single helper now redacts both fields at the SpendLogs builder: it strips a Bearer prefix, passes through values already a sha256 hash or hashed-jwt- identifier, and otherwise hashes with hash_token. sk- keys still map to the same canonical hash, so log and spend correlation is unchanged
Replaces the five launch models with the two that SCX.ai now leads on. Both are live on api.scx.ai and both were verified against it for tool calling, json_object and json_schema output, reasoning, prompt caching, and, for Qwen3.8 Max, image input Pricing follows SCX's published USD rates. GLM-5.2 lands at $0.55/M input and $1.9255/M output, tracking the recent GLM-5.2 market repricing; Qwen3.8 Max at $1.815/M and $5.4461/M sits under the only other seller of that model, and is the first Qwen3.8 Max entry in the catalog Also corrects a metadata bug the removed entries carried: they set max_tokens equal to max_input_tokens, conflating the context window with the output cap. Both new entries declare a max_output_tokens of 131072, which is what the endpoint's own validator enforces The Add Model placeholder moves to scx-ai/GLM-5.2 now that MiniMax-M2.7 is no longer in the catalog
…itellm_scx_ai_provider
The constraint was 1.0, so anything above that was silently clamped down. SCX accepts [0.0, 2.0), verified live against both GLM-5.2 and Qwen3.8 Max: 1.5, 1.99 and 1.999 all return 200, while 2.0 returns 400 with "Temperature should be in [0.0, 2.0)" Since the clamp is an inclusive min(), 2.0 cannot be the ceiling or it would pass through a value the endpoint rejects. 1.99 is the practical maximum The clamp test now pins both ends: 2.5 comes back as 1.99, and 1.7 rides through untouched where it used to be flattened to 1.0
Register Nano Banana 2 Lite on the unprefixed, gemini/, and vertex_ai/ keys so completion_cost and pass-through spend tracking no longer treat the model as unmapped
…itellm_scx_ai_provider # Conflicts: # ui/litellm-dashboard/src/components/provider_info_helpers.tsx
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…logging obj Streamed `/v1/messages` `usage.cost` disagreed with the cost the logging callback recorded in three ways: `input_tokens` was read as the whole prompt total, but Anthropic reports it excluding cache tokens, so the non-cached input went unbilled on cache hits; the `cache_creation` 5m/1h split was dropped, billing 1h writes at the 5m rate; and costing by model name alone ignored the deployment's custom pricing, so a negotiated discount still streamed sticker price. Anthropic usage now goes through `AnthropicConfig.calculate_usage`, the same transformation the non-streaming path uses, and the chunk is priced through the call's logging object when there is one so it inherits `custom_pricing`, `custom_llm_provider`, `base_model` and `router_model_id`, falling back to `completion_cost` by model name. `calculate_usage` only reads its `usage_object`, so it now takes a `Mapping`. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…us bits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…already has reasoning_content
…s book spend Adds e2e coverage for batches terminal state and cost write-back, failure paths, per-backend file content downloads, and two-gateway routing (LIT-5730).
Pricing per https://platform.kimi.ai/docs/pricing/chat-k3: - $3.00/M input (cache miss), $0.30/M cache read, $15.00/M output - 1,048,576 context window; max_completion_tokens settable up to 1,048,576 - Supports reasoning (reasoning_effort low/high/max), tool calling, structured output, vision and video input Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ile_id is still lagging
…ng existing _completed_batch
… keep prompt and cache fields A streaming chat completion that ends early (client disconnect, or the proxy cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields in usage_object. The proxy restamps chunk.model in place to the client-facing alias, so the partial response rebuilt from those chunks priced the unmapped alias and came out at 0. The failure path also rebuilt usage without the request messages, so prompt tokens counted to 0, and a cut stream never sees the final usage event that normally zero-fills the cache fields. Restamp the rebuilt partial response with the wrapper's real model before cost calculation on both the disconnect and the failure paths, pass the request messages when rebuilding usage on the failure path, and zero-fill missing cache usage fields the way completed streams already do.
cache_read_input_tokens and cache_creation_input_tokens are pydantic extras on Usage, not declared fields, so filling them in created keys that were not there before rather than replacing a None. Readers that test for presence then took the new zero as authoritative: the spend log writer skipped its own copy from prompt_tokens_details, turning a real cache read of 500 into 0, and the prometheus provider cache counters stopped incrementing. Carry the prompt_tokens_details counts up before defaulting to zero, so a partial row reports the same cache numbers a complete one does. Renamed the helper to say what it now does.
…eam_cost_cache_tokens
… mutating items Align the response.completed item IDs by copying each output item rather than writing to it in place, and move the regression cases into the existing completion-response and image-generation test modules.
The ceiling used to go through `int(... or 3)`, so anything `int()` accepted worked. Tightening the new shared validator to `isinstance(int)` turned a config that boots today into a proxy that refuses to start, because `max_agentic_loops: os.environ/MAX_AGENTIC_LOOPS` is resolved to a string before it reaches either check, and a YAML-quoted "5" is a string too. Accept ints, integral floats, and strings that parse to a whole number. Keep refusing bools, fractional floats, words, and anything below 1.
…37954) PR #36130 added a KNOWN_MODEL_MODES guard to isModelCompatibleWithEndpoint that hides any model whose mode isn't in the ModelMode enum, to keep rerank/ocr/batch/etc. models out of chat-style endpoints. mode: completion (legacy text-completion models) wasn't in that enum, so it got caught by the same guard and disappeared from every endpoint, including chat, where it routes fine. Add ModelMode.COMPLETION and map it to EndpointType.CHAT like the other chat-compatible modes.
… callers Guardrails, token counting and rate limiting share the input transform with the provider path, so moving reasoning onto reasoning_content hid it from them. Provider-bound callers opt in with replay_reasoning.
…warning fix(types): silence pydantic ReadOnly warning on StandardLoggingRoutingDecision
…esponse fix(websearch_interception): end the turn when the agentic loop hits its ceiling
Write the fallback reasoning item id back to the cache so the reasoning-done path and the completed snapshot cannot drift apart, and cover the shared delta id and the snapshot alignment with tests.
…entrypoint (#37947) The standalone migration entrypoint re-runs `prisma generate` after the migration completes. That refresh writes into the installed prisma package in site-packages, which an arbitrary non-root uid cannot do, and which no uid can do under a read-only root filesystem. Both are supported configurations of the migrations Job: helm/litellm-helm/tests/migrations-job_tests.yaml asserts runAsNonRoot, runAsUser and readOnlyRootFilesystem all render. The write has always failed there, but the failure used to be swallowed. Making migration failures fatal turned it into a hard exit 1, so a Job that applied every migration correctly now reports Failed and blocks the rollout it was supposed to gate. The refresh is redundant in the shipped images: every Dockerfile generates the client at build time from the same baked schema, copies it into the runtime stage, and asserts it resolves there. It stays load-bearing only for a source checkout, where CircleCI runs the entrypoint under `set +e` and ignores the exit code anyway. So the call stays and only its exit code stops propagating; migration failures are still fatal. image-scan never ran on the change that introduced this, because its path filter did not list the entrypoint it exercises. Add prisma_migration.py and entrypoint.sh so the non-root offline migration test gates them from now on.
… 3.6.3-r5 (#37950) The pinned base (built 2026-07-02) ships busybox 1.37.0-r61 and libcrypto3/libssl3 3.6.3-r3. Grype reports 16 fixable findings against those revisions, 8 of them High, so the image-scan gate fails once it gets past the migration step. The runtime stage's `apk upgrade` cannot clear them. wolfi-base writes an exact `=version` constraint for every package it ships into /etc/apk/world, so `apk upgrade` is a no-op even though the fixed revisions are in the repo. Advancing them means moving the digest. The new digest carries busybox 1.38.0-r1, libcrypto3/libssl3 3.6.3-r5 and glibc 2.43-r15, which is at or above the fix revision Wolfi's secdb records for every finding. Verified with cosign against chainguard-images/images release.yaml, and grype reports no fixable findings on the rebuilt image. CVE-2026-14456, CVE-2026-54876, CVE-2026-38752, CVE-2026-38753, CVE-2026-38754, CVE-2026-38755
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| cursor_id = after | ||
| chunk_size = page_size + 1 | ||
|
|
||
| while len(matches) <= page_size: |
Contributor
There was a problem hiding this comment.
Low: Filtered file listing scans an unbounded row set
When purpose matches no files, this loop reads and parses every row owned by the caller because the page limit bounds matches rather than scanned rows. An authenticated user can create many managed files and repeatedly request a different valid purpose to consume unbounded database and application work; filter on an indexed purpose column or impose a per-request scan ceiling.
fix: don't retire a completed batch from cost recovery while output_file_id is lagging
…and logging_only response (#37965) The Presidio guardrail masks messages in place inside pre_call_hook, but three paths independently persisted or emitted the raw pre-guardrail data: the SpendLogs proxy_server_request body snapshot (taken before the hook runs), a verbose_proxy_logger.debug dump of the raw request, and logging_only mode's async_logging_hook, which never masked the model's response before it reached external logging callbacks. Resolves LIT-6015
…ctor The image generation item ID no longer comes from the chat completion response, so the extractor does not need it.
The team roster update, the user.teams update, the team membership delete, and the team-scoped verification token delete ran as four sequential writes with no transaction around them, so a failure between any two left the removal half applied. Thread a single prisma transaction through all four writes, following the same tx.<table> pattern /team/member_add and /team/member_update already use, so either all four land or none do.
…lvable-team fallback (#37960) * fix(auth): resolve team object_permission independently in the unresolvable-team fallback When get_team_object fails for a token's team_id, _user_api_key_auth_builder reconstructs a LiteLLM_TeamTableCachedObj from the token's own cached fields, carrying team_object_permission_id but leaving object_permission unset. That silently dropped any vector-store or MCP restriction the team carried, granting more access than the token's own object_permission_id vouches for. Resolve the object permission by its id directly via get_object_permission, independent of the unreadable team row, matching how every other consumer of a team's object_permission (vector store access checks, MCP tool/server resolvers) already treats an unresolvable team as "no restriction at this level" and re-resolves on its own. * fix(auth): trim ticket references and narrative docstrings per Greptile review Drop the LIT-5539 ticket id from test names and fixture strings, and shorten both the new helper's docstring and the regression test docstrings to their contracts rather than restating the fix's history.
When get_team_object fails, the centralized auth gate rebuilds the team from the token's own fields. A token whose team row was missing when the key was read carries team_models=[] and team_blocked=False, and the model-access check reads an empty model list as every model, so the rebuilt team grants more than the real team ever did. get_team_object reported a deleted team and a database that would not answer as the same 404, so the fallback could not tell a definitive answer from a degraded read. Raise a TeamNotFoundError subclass, still a 404 with the same detail so every other caller is unaffected, only when the database answers and the row is absent. A team that is provably gone now refuses, and no setting overrides that. Otherwise the grant is merely unknown: a token carrying one may vouch, since replaying a recorded grant cannot widen it, and a token carrying none may not. allow_requests_on_db_unavailable still opts back out there, and is only consulted once the failure is known to be a degraded read. The Admin UI mints every session key against the UI_TEAM_ID sentinel, which by design never has a team row, so every UI request hit the new refusal with no override. Exempt UI_TEAM_ID explicitly so it keeps reconstructing from the token unconditionally, matching how the MCP handler and agent_permission_handler already special-case it. Resolves LIT-5522
…atency_metric (#37958) litellm_request_total_latency_metric's start_time is set inside common_processing_pre_call_logic, which only runs after user_api_key_auth has already succeeded, so the metric silently excluded authentication and pre-call setup time despite being documented as total request latency. The sibling litellm_request_queue_time_seconds metric had the same problem: its arrival_time was captured after auth too, despite its own comment claiming to track when the request arrived at the proxy. request.state.litellm_received_at is now stamped unconditionally at the very first line of user_api_key_auth (previously only when OTEL was configured), giving a timestamp that precedes all auth work. Both metrics now derive from it: queue_time_seconds genuinely spans arrival through the start of pre-call processing, and the total-latency metric adds that queue time on top of its existing start/end window so it becomes true end-to-end latency. queue_time_seconds ends exactly at start_time rather than a separately captured timestamp, so its window and the total-latency window share a boundary instead of overlapping and double-counting a few lines of setup work on every request.
A reasoning input item that carries only summary text is replayed to the provider as reasoning_content, so inspection-only callers must see that text too. They used to fall through to the generic content branch, which reads content and drops a summary-only item, leaving guardrails and token counters blind to text the model still receives.
fix(responses): mint Responses API item IDs in the completion bridge
…roviders and single upstream blips (#37957) * test(e2e): send no-cache on every cacheable request body, opt in only where a hit is the assertion The e2e proxy runs with the response cache on, so any test that re-sends an identical chat, messages, responses, completions, embeddings or rerank body reads back a redis copy of an earlier call instead of reaching the provider. Five tests in the last week failed that way. Default cache: {"no-cache": true} on those request models and pass cache=None only in the two tests whose assertion is the cache hit itself. * test(e2e): give image edits and OCR a 180s client timeout Both routes wait on providers that can legitimately take longer than the 60s transport-wide request timeout (gpt-image edits, Azure Document Intelligence), and a client-side read timeout there fails a green request. post/upload now accept a per-call timeout like get already does; only those two call sites use it. * test(e2e): rerun once on network errors and upstream 5xx only Assertion failures still fail on the first attempt; only an outcome whose error string carries the e2e_http network kind or a 5xx status gets one more try. Test Engine records every attempt, so the flake rate stays visible while a single provider blip no longer reds the rc run. * test(e2e): let the reseed burst survive one upstream failure and print why The burst is the precondition, not the property: one 5xx among six concurrent calls still leaves five workers racing the cold counter, which is what the reseed assertion measures. Two or more failures still abort, and the failing bodies are now in the message instead of only the status codes. * test(e2e): keep polling Jaeger through a transient query failure poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land, but a single refused connection to the query API failed the test on the spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41 UTC, each under a minute) and took ten and three otel tests with it while the same tests passed on the rc build minutes later. A network failure now counts as not-yet inside the same deadline; if Jaeger is still unreachable when the deadline passes the test fails with that error, and any non-network failure still fails immediately.
) The field is declared optional on OpenAIFileObject and its own docstring says it is absent on every upload guardrails did not touch, but the /v1/files routes have no response_model, so FastAPI falls through to jsonable_encoder with exclude_none off and serialises the unset default as an explicit null. Every create and retrieve response on a proxy with no guardrails configured at all picked up a litellm_batch_guardrail: null it never had before, and so did every row of a file list, since those rows are the same object. A wrap serializer drops the key only when nothing set it, so the populated report still reaches the wire intact, including a record whose guardrail is null. The managed-files list route spreads a stored file_object blob rather than the model, so rows persisted before this lands keep their null until it is dropped there too.
… none An empty content list, or one holding only opaque blocks, still lets the provider-bound branch replay the summary text. The inspection path treated any non-None content as final, so that replayed text stayed invisible to guardrails and token counting.
…serve-reasoning-input-items fix(responses-bridge): preserve reasoning input items and signed thinking blocks
* test: add regression coverage for twelve closed issues Adds targeted regression tests for behavior that was fixed but left ungated, so the fixes cannot silently regress: - #33772 openai cache_write_tokens cost - #34309 Responses API cache cost_breakdown - #35363 /v1/responses batch spend - #36619 auto-router api_base/api_key leak on a shared model name - #35359 batch fallbacks within the owning model group - #36523 passthrough streamed Responses spend log - #36646 passthrough embeddings spend log - #37147 non-object metadata on create_batch is a 400 - #35362 unscoped list files reads the managed-file store - #33221 gpt-5.6 bridges to Responses on function tools alone - #34487 LLM complexity classifier runs for every caller metadata shape - #35124 streamed /v1/messages emits success logging on both bridges Cost assertions read rates from litellm.model_cost rather than hardcoding dollar amounts, so they do not drift on repricing. * fix: stop the new regression tests polluting and tripping over shared global state Two shard failures, both from global state the new tests share with their neighbours rather than from the behaviour under test. test_main.py's local_cost_map pinned litellm.model_cost but left the get_model_info lru_cache warm, so completion_cost billed at whatever prices were cached earlier in the process while the assertions read the pinned map. Clear the cache on both sides of the fixture, matching the local_model_cost_map fixture in tests/test_litellm/conftest.py. The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush() on whatever queue happened to be around. A queue left non-empty by an earlier test is still bound to that test's loop, so join() either hangs or raises "bound to a different event loop". Rebind to the running loop before the call and wait for the captured payload instead of a fixed sleep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
User Flow
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Caveats (if any)
QA runbook
Final Attestation