[Infra] Promote Internal Staging to main

[ryan-crabbe-berri]

Promote litellm_internal_staging → main to cut a nightly release.

Type

🚄 Infrastructure

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
6 out of 11 committers have signed the CLA.

✅ milan-berri
✅ shivamrawat1
✅ SHARP155
✅ ryan-crabbe-berri
✅ yuneng-berri
✅ Michael-RZ-Berri
❌ Michael Riad Zaky
❌ cursoragent
❌ ishaan-berri
❌ oss-agent-shin
❌ yassin-berriai

---

Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_internal_staging (b888177) with main (fa81017)}

[codecov]

Codecov Report

❌ Patch coverage is 89.32384% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/_experimental/mcp_server/server.py	25.00%	12 Missing ⚠️
litellm/proxy/db/prisma_client.py	83.82%	11 Missing ⚠️
litellm/integrations/opentelemetry.py	89.74%	4 Missing ⚠️
...ms/anthropic/chat/guardrail_translation/handler.py	71.42%	2 Missing ⚠️
litellm/proxy/proxy_cli.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This is a nightly staging-to-main promotion covering several independent features: read-replica routing for Prisma via DATABASE_URL_READ_REPLICA, a structured OTel capture-mode enum replacing the message_logging boolean, skip_tool_message_in_guardrail support mirroring the existing system-message flag, budget-reservation rework with image-gen cost estimation and a fixed output-token fallback, HTTP status-code correctness fixes across auth, and MCP extra-header forwarding for OpenAPI tool upstreams.

• Read-replica routing (routing_prisma_wrapper.py, utils.py, prisma_client.py): new RoutingPrismaWrapper routes reads to a reader endpoint and gracefully falls back to the writer on reader failure; IAM token refresh is wired for both endpoints independently.
• Budget reservation overhaul (budget_reservation.py): removes _get_smallest_remaining_budget in favor of DEFAULT_MAX_OUTPUT_TOKENS_FALLBACK (16 384 tokens) when the caller omits max_tokens, adds image-generation cost estimation, and clamps adversarial max_tokens values at the model ceiling.
• Auth status codes (auth_checks.py, user_api_key_auth.py, auth_exception_handler.py): model-access denials changed from 401 \u2192 403, expired-key from 400 \u2192 401, budget-exceeded from 400 \u2192 429.

Confidence Score: 3/5

The read-replica routing and OTel changes look solid, but the budget-reservation rework removes the concurrent-admission guard for uncapped requests and replaces the tests that covered that race path, leaving the behaviour unverified in CI.

The removal of _get_smallest_remaining_budget eliminates the live-counter budget pre-check for requests that omit max_tokens. The two tests that validated that concurrent-admission path were renamed and replaced with tests for entirely different scenarios, so the race-condition guard is now untested. The HTTP status-code changes are semantically correct but will silently change behaviour for any operator client that branches on the numeric code.

litellm/proxy/spend_tracking/budget_reservation.py and its test file warrant the closest look. The auth status-code changes span auth_checks.py, user_api_key_auth.py, and auth_exception_handler.py.

Important Files Changed

Filename	Overview
litellm/proxy/spend_tracking/budget_reservation.py	Removed _get_smallest_remaining_budget and replaced uncapped-output estimation with DEFAULT_MAX_OUTPUT_TOKENS_FALLBACK (16384). Image-generation cost estimation added. Race-condition guard for uncapped requests is no longer verified by tests.
tests/test_litellm/proxy/test_budget_reservation.py	Several race-condition tests renamed and replaced with tests for different scenarios (adversarial max_tokens, image-gen budget). The removed tests verified concurrent-admission guard behavior that is now untested.
litellm/proxy/db/routing_prisma_wrapper.py	New file implementing read-replica routing for Prisma. Writer-first fallback on reader failure is well handled; _RoutedActions lazily consults _should_use_reader() so mid-call reader degradation is picked up correctly.
litellm/proxy/db/prisma_client.py	Added IAMEndpoint dataclass and reader-wrapper support. DATABASE_PORT now defaults to '5432' fixing a botocore presigned-URL bug. Writer/reader PrismaWrapper construction refactored cleanly.
litellm/proxy/utils.py	PrismaClient now wires DATABASE_URL_READ_REPLICA into a RoutingPrismaWrapper; smoke-test after reconnect targets writer_db directly so the SELECT 1 validates the writer engine specifically.
litellm/proxy/auth/auth_checks.py	HTTP 401→403 for model-access denials; semantically correct but potentially breaking for existing clients.
litellm/proxy/auth/user_api_key_auth.py	Expired-key error code changed from 400 to HTTP 401; correct semantics, but backwards-incompatible for clients checking the numeric code.
litellm/proxy/auth/auth_exception_handler.py	BudgetExceededError now returns 429 (or the exception's own status_code) instead of hardcoded 400; semantically correct.
litellm/integrations/opentelemetry.py	Adds structured capture-mode enum (NO_CONTENT/SPAN_ONLY/EVENT_ONLY/SPAN_AND_EVENT) sampled at init; resolves before every request. Backwards-compatible with legacy message_logging bool.
litellm/proxy/_experimental/mcp_server/openapi_to_mcp_generator.py	Adds _request_extra_headers ContextVar and _merge_openapi_tool_request_headers() to forward allowlisted caller headers to OpenAPI tool upstreams. Precedence rules prevent callers from overriding static operator headers.
litellm/proxy/_experimental/mcp_server/server.py	Populates _request_extra_headers ContextVar from MCPServer.extra_headers allowlist; OAuth2 M2M path correctly skips forwarding caller's Authorization header.
litellm/proxy/management_endpoints/ui_sso.py	Debug SSO callback now surfaces raw_claims and access_token_claims alongside parsed result; token fields are scrubbed from both before rendering to prevent bearer-token leakage into HTML.
litellm/proxy/hooks/model_max_budget_limiter.py	After recording spend, flushes in-memory increments to Redis when a Redis cache is configured, preventing stale in-memory counters from lagging behind other replicas.
ui/litellm-dashboard/src/components/networking.tsx	teamInfoCall now URL-encodes the teamID parameter, preventing injection via special characters in team IDs.
litellm/proxy/_types.py	Adds /openai/v1/realtime routes to LiteLLMRoutes and /health/test_connection to non-admin routes.

Comments Outside Diff (1)

1. tests/test_litellm/proxy/test_budget_reservation.py, line 609-660 (link)

   Race-condition tests replaced without equivalent coverage

   Two tests that validated race-condition safety were removed and not re-added:

   • test_should_shrink_uncapped_reservation_when_counter_advances — verified that when a concurrent reservation advances the counter while a second reservation is in flight, the second reservation correctly shrinks to the remaining budget. The replacement (test_should_clamp_reservation_to_model_ceiling_when_caller_overrequests) tests a different scenario (adversarial max_tokens clamping).
   • test_should_shrink_uncapped_reservation_multiple_times — verified that multiple budget counters share the smallest remaining budget. This is gone entirely.

   Removing _get_smallest_remaining_budget eliminated the budget pre-check that happened when no max_tokens was specified. Now, uncapped requests use DEFAULT_MAX_OUTPUT_TOKENS_FALLBACK × cost_per_token as the reservation estimate, which may exceed or fall below the actual remaining budget, leaving the race-condition guard unverified.

   Rule Used: What: Flag any modifications to existing tests and... (source)

_{Reviews (1): Last reviewed commit: "Fix proxy auth status code tests (#27555..." | Re-trigger Greptile}

[greptile-apps]

HTTP status code changes are backwards-incompatible without a flag

Three status codes changed in this PR: model access denied 401 → 403, expired-key errors 400 → 401, and budget-exceeded errors 400 → 429. Each change is semantically correct per RFC 9110, but any client that parses the numeric code to branch its retry or error-display logic will silently change behavior. Per the team's rule on backwards-incompatible changes, these should either be gated behind a user-controllable flag or the changelog should call them out prominently so operators know to update their error-handling code.

Rule Used: What: avoid backwards-incompatible changes without... (source)

[veria-ai]

High: Budget reservation bypass for unpriced requests

Returning without a reservation for models/routes that cannot be priced lets a budgeted user start many concurrent image/audio/unknown-model requests: each request sees the same pre-call spend, skips the in-flight counter, and reaches the provider before post-call spend is recorded. Keep a conservative reservation for None estimates, or block budgeted requests whose maximum cost cannot be estimated.

[yuneng-berri]

Pushing back - the fail close method is too restrictive for our use case.

[veria-ai]

High: Stale auth and authorization reads on read replicas

Routing every find_* call to the replica includes reads of litellm_verificationtoken, users, teams, and model permissions used during request authentication. With replica lag, a user whose key or team/model access was just revoked can continue authenticating after cache invalidation because the auth path rehydrates from the stale replica; keep security-sensitive auth, authorization, and budget tables on the writer and only route non-admission reads to the replica.

[yuneng-berri]

Pushing back - we do not support multi db setups, this is for AWS RDS where there are 2 endpoints with different IAM roles. There will be no replication lag introduced by us

[veria-ai]

Medium: Team admins can test unauthorized model credentials

This PR adds /health/test_connection to the non-proxy-admin self-managed route set. The endpoint loads configured model credentials before checking authorization, then authorizes team admins based on caller-supplied model_info.team_id, so a team admin can run a health call against a proxy-configured model they do not otherwise have access to.

---

Status: 1 new · 3 open
Risk: 6/10

[veria-ai]

Medium: Unauthorized model health checks

Allowing non-proxy-admin callers through to /health/test_connection lets a team admin set litellm_params.model to any configured deployment and set model_info.team_id to a team they administer. The endpoint then loads that deployment's configured credentials before can_user_make_model_call() runs, so the caller can trigger a provider health request and receive routing metadata for models outside their team; gate this route to proxy admins, or verify that the requested configured model actually belongs to the caller's team before merging and using its credentials.

[yuneng-berri]

Pushing back - team admins are vetted and trusted people within the organization