chore(ci): promote internal staging to main - #38293
Merged
Merged
Conversation
…s to litellm_team (#37918) * fix(terraform): add soft_budget, tags, and soft_budget_alerting_emails to litellm_team The team resource rejected soft_budget and tags at plan time and had no way to express the list-valued metadata.soft_budget_alerting_emails the proxy reads for soft-budget alerts, even though /team/new and /team/update accept all three. Add the attributes, forward them in buildTeamData (alert emails merged under metadata, where the proxy stores them), and send the full metadata map whenever either half changes because /team/update replaces metadata wholesale. Read was decoding /team/info as if the team fields were top-level, but the proxy nests them under team_info, so every attribute silently fell back to prior state. Decode the envelope and split the proxy's metadata back into tags / soft_budget_alerting_emails / string metadata, dropping the server-managed team_member_budget_id. Verified with OpenTofu plan/apply against a live proxy: the attributes are accepted, land on the proxy, refresh into state, re-plan clean, propagate on update, and clear when removed from HCL. * fix(terraform): clear litellm_team.soft_budget in state when the proxy returns null Read only wrote soft_budget when the proxy returned a value, so a soft budget cleared outside Terraform stayed in state and never surfaced as drift. Set it from the response unconditionally so a null clears it.
The virtual key shown after creating a key sits in a div with a hardcoded #f8f8f8 inline background, so in dark mode the box keeps the light background while the key text inherits the light foreground color, leaving the key nearly unreadable. Swap the inline styles for the bg-muted and text-foreground tokens, which resolve per theme.
…l flow (#37986) The tooltip popup is an inline-flex row, so the four sibling blocks passed as a fragment laid out side by side in four columns. Wrap them in a single flex-col container instead. The inline code samples also used bg-muted, which is defined against the page surface, not the inverted tooltip surface, so they rendered as near-white chips carrying near-white text. Tint them from the popup's own token instead.
…mited (#37916) * fix(ui): render team and org tpm/rpm limits of 0 as 0 instead of Unlimited A tpm_limit or rpm_limit of 0 is a hard block on the backend (every request 429s) and only null means unlimited, but the team and organization views rendered both as "Unlimited" (and a team-member limit of 0 as "No Limit") because every display site used a falsy || fallback. The team member edit dialog also seeded its form with `tpm_limit || null`, so opening Edit Member on a member stored with 0 and clicking Save sent null to /team/member_update and silently turned the hard block into unlimited Every limit display site in TeamInfo, organization_view, the organizations list cell and the team members table now uses a nullish check, and both member form seeding paths keep 0 for max_budget_in_team, tpm_limit and rpm_limit. Regression tests cover each site and the existing memberFormValues test that asserted 0 -> null is flipped to assert 0 survives Resolves LIT-5760 * test(ui): assert a stored 0 member limit survives an untouched save The EditMembership integration test named the old 0 -> null collapse as the expected payload, so the related-tests CI job went red once the form kept 0. It now asserts 0 survives and only the empty budget_duration collapses to null. The TeamMemberTab fixture is built with a map instead of mutating the nested membership
…ating (#37968) Regenerating a key from the key info page left the ?key= query param on the old hash, so dismissing the dialog or reloading landed on a key that no longer exists and the page rendered "Key not found". Two defects had to line up. POST /key/{key}/regenerate returns the rotated hash in token_id and leaves token null, but RegenerateKeyModal read response.token || response.key_id, neither of which the endpoint populates, so it always reported the old hash back to its parent. And KeyInfoView's onKeyDataUpdate prop had no caller anywhere in the tree: VirtualKeysTable owns the ?key= param and mounts the view but never passed it, so even a correct hash went nowhere. VirtualKeysTable now handles the update by pointing ?key= at the rotated hash and refetching. KeyInfoView holds that callback until the regenerate dialog is dismissed rather than firing it on the API response, because swapping the selected key mid-dialog unmounts the view and tears down the one-time plaintext key before the user can copy it.
The playground message bubble painted its fill, border and avatar circle from inline hex values, so in dark mode both bubbles stayed near-white while the text inherited the dark foreground: the message body was unreadable. The MCP-events placeholder bubble in ChatUI carried the same three fills. They move onto the tokens the rest of the sweep already uses, so the assistant surface is bg-card over border-border and the user surface is the info tint at the same weight the other selected-state surfaces take. Light mode keeps the same colour family it had. The regression test asserts the token classes and that no inline style survives on either surface, which is the exact shape the bug took.
…commands_v2 fix(UI): correct skill install command and marketplace setup UX
… comments in restored bodies
…uter_health_check fix(proxy): skip health checks for strategy routers
…_pricing fix(databricks): bill cached tokens at cache rates and add missing Claude pricing
…_no_trailing_empty_chunk Resolves the test-file conflict by keeping both sides, extends the finish-reason gate to trace-bearing metadata events so guardrail trace chunks keep their pre-regression delta shape, parametrizes the regression test over tool-call, mixed, and reasoning streams, and repairs the one ant-design icon usage the lucide-react migration left behind in skill_detail.tsx (semantic conflict on the base branch)
…ndtrip fix(anthropic): round-trip thinking blocks to OpenAI backends on /v1/messages
POST /v1/videos without an input_reference file now goes out as multipart/form-data the way the OpenAI SDK always sends it, instead of a JSON body that OpenAI-compatible backends (SGLang Diffusion, vLLM-Omni) reject; gemini, vertex, and runwayml keep their JSON bodies /v1/images/edits on the openai/azure/openai-compatible path now forwards unknown provider params (e.g. seed) and honors extra_body, matching /v1/images/generations, and aimage_edit forwards extra_headers/extra_query/extra_body instead of dropping them Generic pass-through no longer downgrades a file-less multipart form to application/x-www-form-urlencoded
…asing fix(a2a): normalize agent card protocolBinding casing before transport match
…outer-model /vllm and /azure routes The /vllm and /azure router-model passthrough branches called llm_router.allm_passthrough_route directly with no request metadata, so the cost callback saw no user_api_key and no user_api_key_budget_reservation. Spend for a budgeted virtual key hit neither the key's spend nor the spend logs, and the reservation minted at auth into the shared Redis counter was never released, drifting the counter up until the key falsely tripped BudgetExceededError. Thread the authenticated key's attribution metadata into both calls via the same builder add_litellm_data_to_request uses, so the cost callback attributes spend and reconciles the reservation. Regression tests cover both branches.
fix(interactions): track cost and spend for Google Interactions API requests
…-less Vertex passthrough When no Vertex credential is configured (no default_vertex_config, no matching use_in_pass_through deployment, no vector-store credential), the Vertex passthrough took the bring-your-own-credentials branch and forwarded the entire incoming header set upstream to Google. That set included whichever header carried the caller's LiteLLM virtual key: x-litellm-api-key, or Authorization when get_litellm_virtual_key read the key from there. The proxy's own secret was sent to a third-party provider. The credential-less branch now drops x-litellm-api-key and the Authorization value that equals the virtual key, keeping a genuine bring-your-own Google credential (an OAuth token in Authorization, or x-goog-api-key) so real BYO passthrough still works. When neither survives, the request fails with a clean 401 telling the operator no credential is configured, instead of forwarding the virtual key. Regression coverage in the mapped test path asserts the 401-and-never-forwarded behavior for both leak vectors and that a real Google credential still passes through with the virtual key stripped.
…cking Six defects in the RunwayML video provider: - transform_video_create_request hardcoded /image_to_video, so text-to-video 400'd and video-to-video was unreachable; the endpoint is now selected from the inputs present (promptVideo/videoUri, promptImage, or text only) - get_error_class raised instead of returning, turning a provider 4xx into a proxy 500 APIConnectionError; it now returns a RunwayMLError - VideoObject.progress was typed int while Runway sends a 0..1 float, 500'ing status polls while RUNNING; it is now scaled to a 0..100 percent - custom per-deployment pricing stored under litellm_metadata was ignored for video; the deployment model_info lookup now checks both metadata keys - stale cost-map entries (gen3a_turbo, gen4_aleph) were removed and current models added, with output_cost_per_second_480p/_4k tier keys plumbed through the model-info and router types - video cost now falls back to Runway's estimatedCost from the create response when no custom pricing is configured, and custom pricing always wins over it Fixes #36483
Move _update_litellm_params_for_health_check before can_user_make_model_call so health_check_params cannot retarget the probe after the auth check. Type the Pegasus test helper signature and drop the redundant test narrative.
…dy-36487-8fc4 Pulls in the detect-changes CI action and the test-unit job timeout bump, which clears the red lint and code-quality checks on this PR The merged, tightened lint budgets flag this PR's own code, so this merge also makes video_reference_to_id a pure function instead of a helper that mutates its input dict, and rewrites the form-body regression test to call the video_edit and video_extension handlers directly rather than patching an internal class method. Adds pure-logic unit tests for video_reference_to_id
…itellm_health_test_connection_health_check_params # Conflicts: # tests/test_litellm/proxy/test_health_check_max_tokens.py
Bring the Entra ID / OAuth auth work for Azure AI Foundry routes up to date with staging and fix the lint-budget regressions the merge surfaced: - widen get_azure_ai_auth_headers return type to Mapping[str, str] (LIT001) - build the azure_ai image_generation request headers into a new Final local instead of rebinding the Final headers dict (reportGeneralTypeIssues) - order HuggingFace rerank validate_environment params to match BaseRerankConfig so litellm_params lines up positionally (reportIncompatibleMethodOverride) - add a match= to the credential-error test and document the handler-boundary patches the auth wiring tests rely on
…und2 refactor(repositories): type prisma table access with one generic protocol
…c stream cache writer
…ges surfaces Adds streaming, async, /v1/responses, and /v1/messages coverage for the Together AI overhaul (#38233, #38248, #38230, #38265, #38275), plus the legacy api.together.xyz host and TOGETHER_AI_API_BASE through litellm.completion. Each new test fails under a one-line mutation of the merged code.
fix(anthropic): buffer streamed responses carrying server-fulfilled tools so retrieval tool calls never reach the client
test(together_ai): regression suite across chat, responses, and messages surfaces
…and cost through a live proxy
* feat(logging): add async_post_call_failure_deployment_hook CustomLogger already has async_pre_call_deployment_hook and async_post_call_success_deployment_hook, both firing once per real deployment attempt from wrapper_async since the router re-enters that wrapper fresh on every retry and fallback step. There was no failure-side counterpart; the only failure signal, async_log_failure_event, fires once per logical client request behind a dedup gate, so fallback chain attempts 2+ were invisible to callbacks needing per-deployment-attempt granularity. Adds async_post_call_failure_deployment_hook(request_data, exception, call_type) to CustomLogger and a matching dispatcher in utils.py, called from wrapper_async's except block. It needs no dedup coordination since each real attempt naturally re-enters the wrapper once. Unlike its two siblings, the dispatcher wraps each callback call in its own try/except since it runs on the wrapper's own exception path and a broken callback must never mask the exception about to be re-raised to the caller. * feat(logging): pass fallback_depth through to async_post_call_failure_deployment_hook Router already tracks fallback_depth internally on each fallback hop (litellm/router_utils/fallback_event_handlers.py), incrementing it once per target tried, but nothing surfaced it to CustomLogger callbacks. Reads it off request_data in the dispatcher and passes it through as a best-effort int | None keyword: None on the first, pre-fallback attempt or a bare SDK call with no router, 1 on the first fallback hop, 2 on the second, and so on. Verified live against a real multi-hop Router fallback chain before adding the regression tests. * fix(logging): fire async_post_call_failure_deployment_hook on internal calls too The failure hook was gated behind the same not _is_litellm_internal_call check as the request-level dedup-gated failure logging, so a failed internal sub-call (e.g. an emulated file-search step) never reached it, even though its async_pre_call_deployment_hook and async_post_call_success_deployment_hook siblings already fire unconditionally for such calls. * chore: retrigger CI (lint job hit a transient GitHub Actions infra outage on the prior push) * chore: retrigger CI (lint job hit the same GitHub Actions infra outage again) * fix(logging): scope async_post_call_failure_deployment_hook to the actual model call The hook was dispatched from the wrapper's broad outer except, which also catches BudgetExceededError (raised before any deployment attempt), errors from async_pre_call_deployment_hook, and errors raised after a successful model call (post_call_processing, async_post_call_success_deployment_hook, caching). None of those are a deployment attempt failing, so the hook misreported them as one. Scoped the hook to a try/except around the model call itself, so it only fires when that specific call raises, matching its own documented contract. * test: assert the callback actually ran in the failure-hook error-isolation test An upstream test-quality gate (TQ001) flagged this test for asserting nothing, so it could only fail by raising. Track whether the exploding callback actually ran and assert on it, so the test would catch a dispatcher that silently skipped every callback instead of isolating a raising one. * fix(logging): harden async_post_call_failure_deployment_hook against 5 maintainer-verified issues A maintainer's live-proxy A/B review against base found five real problems with the failure hook, all reproduced and fixed: - The dispatcher called overrides with fallback_depth as a required keyword, so an override matching this PR's own earlier 3-arg proof-of-fix example raised TypeError, swallowed at debug level, on every call. Now checks the override's signature once per class and omits the keyword when unsupported. - A callback mutating the exception it receives (e.g. status_code) changed what the real caller got back, since it was the same live object about to be re-raised. Callbacks now receive a same-class snapshot instead. - request_data exposed attempted_targets, the router's own live fallback-walk bookkeeping shared by reference across every hop, so a callback calling .record() on it could make the router skip a deployment it never actually tried. Now excluded from what the hook receives. - The hook's own await sat directly in the model-call except block, so a caller-side cancellation landing mid-await (e.g. asyncio.wait_for) replaced the real deployment exception with CancelledError/ TimeoutError. Now isolated so hook dispatch can never mask the real failure. - The timestamp used for the reported failure duration was captured after the hook ran, so a slow callback inflated async_log_failure_event's duration. Now captured before the hook dispatches. * fix(logging): preserve traceback/cause/context on the failure-hook exception snapshot Bugbot found a real gap in the previous round's exception-mutation fix: _snapshot_exception_for_hook only copied __dict__ and args, so a callback formatting or inspecting the failure chain saw an empty traceback and lost chained-exception context, even though the live exception still has them. __traceback__/__cause__/__context__ aren't stored in __dict__, so they need copying explicitly. * fix(logging): preserve __suppress_context__ on the failure-hook exception snapshot Setting __cause__ has a documented CPython side effect of implicitly forcing __suppress_context__ to True. Since the previous round's traceback fix set __cause__ before __suppress_context__, a normal implicit-chaining exception (no `raise ... from`, __suppress_context__ naturally False) got its context wrongly suppressed on the snapshot. Now __suppress_context__ is set explicitly, after __cause__, so it always reflects the real exception. * fix(logging): use MappingProxyType for the failure-hook's sanitized request_data A LIT002 budget check (surfaced by rebasing onto a moved base) flagged the dict comprehension building safe_request_data as mutable construction. MappingProxyType is also a strictly better fit here: a genuinely read-only view, not just an immutable-looking dict, matching the intent that callbacks should never be able to mutate what they're handed. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
yuneng-berri
requested review from
mateo-berri and
ryan-crabbe-berri
as code owners
August 26, 2026 02:39
Contributor
|
Too many files changed for review (615 files, 100 file limit). |
|
|
mateo-berri
approved these changes
Aug 26, 2026
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| if response.status in _TERMINAL_STATUSES and response.usage is not None: | ||
| await _bill_settled_interaction(logging_obj=context.logging_obj, response=response) | ||
| return | ||
| await _release_open_budget_reservation(logging_obj=context.logging_obj) |
Contributor
There was a problem hiding this comment.
Medium: Budget bypass when deleting active interactions
An authenticated user can start a background interaction and delete it while its status is still in_progress; this branch releases the full reservation and permanently stops the poller before final usage is known. Because deletion is separate from cancellation, provider work already performed can remain unbilled. Keep the reservation until deletion succeeds and reconcile it conservatively, or cancel and poll to a terminal state before releasing it; prefetch failures must not fail open either.
Contributor
PR overviewThis PR promotes internal staging changes to main, including updates to background cost polling for interactions. One security issue remains open, with none addressed so far. An authenticated user can delete an in-progress background interaction to release its budget reservation before final provider usage is reconciled, allowing completed work to remain unbilled and budget limits to be bypassed. Open issues (1)
Fixed/addressed: 0 · PR risk: 6/10 |
is_expected_client_error treated every HTTP 4xx as a rejection the proxy issued itself, so a 401 or 429 the provider returned lost its traceback in the standard logging payload and the OTel error span dropped litellm.provider.error.stack_trace. An exception carrying llm_provider is an upstream or deployment problem and keeps its traceback; the proxy's own pre-call rejections still skip it
test(e2e): cover Together AI reasoning, tool calls, template kwargs, and cost through a live proxy
The /v1/messages route logs the provider's raw BaseLLMException, which carries no llm_provider, so its 4xx still counted as an expected client error and lost its traceback. Treat BaseLLMException as provider-originated as well.
ProxyRateLimitError derives from HTTPException but carries an llm_provider, so it read as provider-originated and regained its traceback. Any HTTPException is a proxy rejection regardless of llm_provider.
…t error The auth handler stamps the requested model's provider onto BudgetExceededError before logging it, which made a key-over-budget 429 look provider-originated and regain its traceback (and an OTel stack_trace) after the provider 4xx carve-out. Any exception whose unified rate-limit category names litellm's own limiter is now a proxy rejection, matching the HTTPException rule.
…r_stack_trace fix(logging): keep tracebacks for provider-originated 4xx errors
shin-berri
approved these changes
Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
User Flow
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Caveats (if any)
QA runbook
Final Attestation