chore(ci): promote internal staging to main - #36560
Merged
Merged
Conversation
Generated with AI Co-Authored-By: Claude Code
The native files and batches routes declare /{provider}/v1/... and their routers are mounted before the passthrough router, so /openai_passthrough/v1/files and /openai_passthrough/v1/batches matched them with provider="openai_passthrough" and 500'd on the LlmProviders lookup instead of reaching openai_proxy_route.
Move the dedicated /openai_passthrough prefix onto its own router mounted ahead of the batches and files routers. /openai/... and every other provider prefix keep their current behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GET /v1/files filters data down to the caller's own managed files but left first_id and last_id as the upstream page's, so a non-owner got back file ids belonging to other users even with an empty data array
fastapi 0.140.7 removed get_flat_dependant(), which broke the import in management_v1/common.py and took down every /management/v1 route. Switch to get_flat_params() and filter to ParamTypes.query so unknown-query-param rejection keeps matching the old behavior.
Guards _declared_query_params against a regression in the get_flat_params migration: the flatten step returns path, query, header and cookie params together, so a dropped ParamTypes.query filter would wrongly treat path or header names as declared query params and accept unknown ones. Removing the filter fails these tests.
…e file A batch or fine-tuning job is created from a file the caller already uploaded, and that file only exists under the credentials of the deployment that stored it. When the router fell back to a different model group it handed that file id to a provider that has never seen it, so the caller got the second provider's complaint about the file id instead of the error that explains what was actually wrong with their request. run_async_fallback now skips fallback targets outside the original model group whenever the request carries input_file_id or training_file. Order-based fallbacks stay inside the group, so retrying across deployments still works. The same handler also crashed with "'NoneType' object has no attribute 'update'" whenever a fallback fired on a request with metadata set to None, which /v1/batches always does when the caller sends no metadata, turning the provider's 400 into a 500. Record the model group with a merge instead of setdefault, and write it to litellm_metadata on the endpoints that use it so the router's bookkeeping no longer lands in the metadata stored on the provider's batch.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ribution (#34456) A poll of a Vertex passthrough batch wrote nothing to the managed-object row, so status and file_object stayed frozen at the create-time snapshot and GET /v1/batches served a stale status and an empty output file id for the life of the batch. Only the create may claim a batch, but every observation of one may refresh its state. store_unified_object_id takes create_if_missing, which the poll clears: it refreshes status and file_object through update_many, and leaves a row that is absent absent rather than creating one owned by the observer, since created_by and team_id are written by whoever reaches the create branch. The update payload is now shared with the upsert so it cannot drift into writing api_key, request_tags, created_by or team_id. The passthrough identity re-assertion that was previously part of this PR ships separately in #36121, so this PR keeps only the batch attribution work. The creating key owns user_api_key_alias only when it actually has one. Guarding the overwrite on the presence of a key rather than on a resolved alias nulled the field out for every key generated without key_alias, and for any key rotated or deleted before its batch finished, losing the creating user's alias that the spend row previously carried. The guard now matches the team-alias line below it.
…evin_ai_fix_file_list_cursor_leak_36087
…ptions docs: rewrite the CLAUDE.md comment rule with explicit exceptions
…leak_36087 fix(proxy): scope file list pagination cursors to the caller
fix(proxy): skip prisma-dependent hooks when no database is attached
…oped fix(proxy): report has_more false on caller-scoped file list pages
…evin_ai_fix_openai_passthrough_files_route_36086
…itellm_fix_batch_group_fallback # Conflicts: # litellm/router_utils/fallback_event_handlers.py # tests/test_litellm/router_utils/test_fallback_event_handlers.py
…endpoints The Logs nav entry is open to internal users so they can read their own request logs, but the page rendered all four tabs unconditionally. Audit Logs calls GET /audit and Deleted Teams calls GET /v2/team/list?status=deleted, neither of which an internal user is permitted to call, so the page fired requests that came back 401. Gate both tabs on new viewAuditLogs / viewDeletedTeams capabilities, using the same CAPABILITY_ROLES map and useCan hook introduced for Tool Policies. Hiding a tab drops its panel from the tree entirely, so the request is never issued rather than issued and rejected. Selecting a tab also mapped index 0 to "request logs" and every other index to "audit logs", which activated the audit panel whenever a user opened Deleted Keys or Deleted Teams. Derive the active tab from the visible tab list instead, so the mapping survives tabs being filtered out.
The Usage page admits internal users because their own usage view works, but the entity breakdown selector inside it also offered Organization Usage, so picking it fired /organization/daily/activity and collected a 401. Neither that route nor /agent/daily/activity appears in any non-admin route list, so both are default-deny. The team breakdown leaked the second one too: it fetches agent activity unconditionally to fill its Top Agents card, which 401s for the same roles. Adds viewOrganizationUsage and viewAgentUsage to the existing capability map and points the selector option, the page section, and the fetch's enabled flag at the same capability, so a role that cannot call the endpoint never sees the breakdown and never issues the request. The team and tag breakdowns, which internal users can read, are untouched, and the default Usage view was already one of those.
/policies/list and /prompts/list are default-deny for internal_user, but the Virtual Keys create/edit flow, the Teams forms and the Playground called them on mount, so every internal user landing on the dashboard fired two requests that 401. Add viewPolicies and viewPrompts to the capability map and use them to gate the nav entry, the form field and the fetch together, following the pattern from the Tool Policies migration. Non-admins now see no policy or prompt selector at all rather than an empty dropdown.
…get_flat_params fix(proxy): restore management_v1 query-param validation under fastapi>=0.140.7
docs: require a user flow and a stuck-at proof in feature requests
#36193) * feat(router): add required-AND (&) tag prefix and allow_fail_open flag Tag routing supported inclusion-OR and independent "!" negation, but had no way to express a hard "must match all of these" constraint per request, and no way for a model group to opt into degrading gracefully instead of raising when a constraint eliminates every deployment. Adds a "&tag" prefix for required-AND inclusion, composing with existing plain (OR) and "!" (negate) tags: negation still applies first, then required tags narrow the survivors, then plain tags apply today's OR/AND preference logic unchanged. Adds model_info.allow_fail_open (default false) so a chain can opt into falling back to the default-tagged pool instead of raising no_deployments_with_tag_routing when "!" or "&" empties the candidate set; existing chains without the flag keep today's fail-closed behavior exactly. * fix(router): gate mixed negation on allow_fail_open and stop diluting required-only requests Two gaps in the initial required-AND/allow_fail_open change: a "!" exclusion combined with a plain positive tag that emptied the candidate set raised unconditionally, bypassing allow_fail_open entirely, since the fail-open check only looked at required-AND exhaustion. And a request using only "&" tags could get narrowed down to just the deployment matching an incidental tag_regex/User-Agent preference, silently dropping other deployments that satisfied the required tags but had no tag_regex at all. Fixes both: the fail-open check now fires whenever either "!" or "&" leaves the candidate set empty, not just "&". And regex/header preference no longer counts as a positive filter when a required-AND ask is present, so a required-only request returns every deployment satisfying the required tags regardless of regex/header matching. Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new model_info.allow_fail_open field, and removes source comments explaining the router logic per repository convention. * fix(router): let allow_fail_open cover a non-empty !/& survivor set that fails the plain-tag preference The unconditional raise inside the has_positive_filter loop was the one remaining path a chain could hit despite setting allow_fail_open: when "!" or "&" leaves a non-empty candidate set but none of the survivors match the request's plain preference tag or carry "default", the request still raised instead of degrading. Routes that raise through the same allow_fail_open check used everywhere else, so it now falls back to the default-tagged pool for opted-in chains and keeps raising unconditionally for everyone else. This also let the now-redundant pre-loop empty-candidates shortcut be removed, since the loop reaches the same outcome on its own. * fix(router): deny allow_fail_open when an unrecognized required tag is masking a satisfiable answer A caller could add a single "&" tag no deployment in the group has ever carried to force an empty required-AND set on demand. On a chain with allow_fail_open, that emptied set fell back to the default-tagged pool unconditionally, discarding every other constraint merged into the same request, including ones inherited from key/team policy, even when the rest of those constraints were still individually satisfiable. Before falling back, drop any required tag not carried by any deployment in the group and recompute: if a specific, non-empty answer exists using only the recognized tags, the unrecognized tag was the actual cause of the exhaustion, and fail-open must not paper over it. If every required tag is already recognized, or none are, there's nothing hidden behind an invented tag, and fail-open proceeds exactly as before; this keeps a single opted-in deployment's legitimate catch-all behavior working when a caller's tag simply doesn't exist anywhere in that group. Ratchets ANN401 and LIT001 budgets down to reflect fixes already earned in this branch. * test(router): cover required-AND, allow_fail_open, and unknown-tag denial across fallback chains and model groups Extends coverage beyond single-hop scenarios: & exhausting a primary group falls through to a fallback group exactly like ! already does; !, &, and allow_fail_open composed together across three chained model groups each raise or fall back independently per-hop; and the unknown-tag denial from the previous commit is evaluated fresh per hop rather than leaking state across groups in a fallback chain. * feat(router): add model_info.enable_tag_filtering per-model-group override enable_tag_filtering was router-wide only: an operator turning it on for one model group that needs tag-driven routing exposed every other model group on the same proxy to the same tag evaluation, even ones that never use tags. Adds model_info.enable_tag_filtering, checked against any deployment sharing a model_name, so a chain can flip the router-wide default in either direction for itself alone: opt a specific group into filtering while the rest of the proxy stays off, or opt a group out (e.g. an incident-response catch-all) while the rest of the proxy enforces it. Precedence, low to high: router-wide default, then the chain override if set, then the existing request-level escalation (from key/team settings), which still only ever turns filtering on, never off, over whatever the router and chain already decided. Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new field. * fix(router): gate plain-tag exhaustion on allow_fail_open when the tag is known to the group A model group where every deployment is tagged "default" (a legitimate cross-cutting safety-net pattern) never has an empty default_deployments list, so the existing exhaustion check (len(new)==0 and len(default)==0) never fired for a plain positive tag that matched nothing among the currently healthy candidates. The request silently fell through to whatever "default"-tagged deployment happened to survive, even when allow_fail_open was never set and the caller's intent (e.g. quality:high) was never honored. Adds a check for whether the requested tag is part of the group's real vocabulary at all: if some deployment configured under this model_name (regardless of current health) genuinely carries the tag, and nothing healthy currently matches it, the request now raises by default or falls back per allow_fail_open, through the same _resolve_or_fail_open gate every other exhaustion path already uses. A tag that's foreign to the group entirely (e.g. one meant for an unrelated mechanism sharing the same request-tags list) keeps falling back to the default pool unconditionally, unchanged, since there's nothing this group's own routing intent could be violating. * fix(router): preserve inherited tag constraints when allow_fail_open discards a caller-caused exhaustion Adds metadata.caller_tags in litellm_pre_call_utils.py, populated only from what the request itself supplied (header, body tags, body metadata.tags), never from key/team metadata merged into the same metadata.tags list. get_deployments_for_tag now uses it to compute a trusted-only pool before falling open: a required/excluded tag attributable to the caller can be discarded on fail-open, one inherited from key/team policy cannot. If the trusted-only pool is itself empty, allow_fail_open raises instead of silently routing around an unsatisfiable inherited constraint. When caller_tags carries no information at all (direct SDK Router usage, bypassing the proxy layer), behavior is unchanged: unconditional fall-open to the default pool, exactly as before this fix. * feat(router): add opt-in tag_routing_prefix for collision-proof tag disambiguation router_settings.tag_routing_prefix lets a caller explicitly mark which x-litellm-tags/metadata.tags values are routing directives, exempting them from the known-tag-vocabulary heuristic used to guard fail-open against caller-invented "&"/"!" tags. Unprefixed tags keep going through today's existing handling unchanged (hybrid, no migration required); default "" is a full no-op. Fixes a bug caught during live-proxy verification: the prefix-stripped "confirmed" set kept the "&"/"!" marker character, so it never matched required_set/excluded_set (which _split_tags always strips bare) -- the entire trusted-required/excluded-tag mechanism silently no-opped for its primary use case. Adds regression tests for the bare-value mismatch and updates existing _chain_allows_fail_open/_tag_known_to_group/ _caller_constraint_sets call sites for the new routing_confirmed/ routing_prefix parameters. * fix(router): resolve model_info.enable_tag_filtering override from the full model group, not just healthy deployments Cooldown filtering runs before get_deployments_for_tag, so _chain_tag_filtering_override only saw the survivors of that filter. A model group whose only enable_tag_filtering-carrying deployment goes into cooldown lost the override entirely, silently falling back to the router-wide default and letting any !/&/tag constraint on that chain be bypassed by driving the one overriding deployment into cooldown. Resolve the override from every deployment configured for the model instead, mirroring _tag_known_to_group's existing pattern. Verified live: with a bad-key deployment carrying the override forced into real cooldown via allowed_fails=1, an explicit "!provider:openai" ban on the remaining deployment reproducibly returned 200 via OpenAI before this fix and 401 (tag filtering still enforced) after it. * fix(router): avoid Final-reassignment lint error and a MagicMock router fixture gap from tag_routing_prefix _chain_tag_filtering_override's try/except reassigned a Final-annotated name across branches, which basedpyright flags as illegal; extracted the lookup-with-fallback into its own helper so the binding is assigned once. Also sets tag_routing_prefix on the bare MagicMock router used by test_router_tag_regex_routing.py's fixture, which otherwise returns an auto-generated MagicMock (truthy, non-string) for the new attribute and crashes _strip_routing_prefix's removeprefix() call. * fix(router): key inherited-tag protection off provenance, not value subtraction allow_fail_open's trusted-only pool computed "not caller-attributable" as required_set - caller_required_set. A caller who resubmits the exact value of an inherited "&"/"!" tag (e.g. an inherited "®ion:eu" alongside a caller-supplied "®ion:eu" plus a conflicting "!region:eu") collapses both origins to the same set value, so the subtraction zeroes out the inherited requirement's protection too, letting fail-open route outside a key/team-enforced constraint. Adds metadata.inherited_tags in litellm_pre_call_utils.py: a snapshot of "tags" taken after key/team/project policy is merged in but before this request's own caller-supplied tags are merged on top. A required or excluded tag is now protected from fail-open discard if it has ANY inherited backing (set intersection with inherited_tags), regardless of whether the caller also happens to submit the identical value -- this is what set membership alone could never tell apart under the old subtraction-based approach. caller_tags is kept (documented as the complementary record) but no longer consulted for this decision. Verified live: a virtual key with metadata.tags=["®ion:eu"] hit with header x-litellm-tags: ®ion:eu,!region:eu (the exact value-collision attack) reproducibly routed to the OpenAI/us deployment before this fix and stayed on the Anthropic/eu deployment after it. * fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging Regenerated ruff-strict-budget.json and type-discipline-budget.json via make lint-ruff-budget-update / lint-type-discipline-budget-update against the post-rebase merge-base. * fix(proxy): compute inherited_tags from key/team/project sources directly, not a tags-list snapshot apply_client_tag_policy_pre_auth (run from user_api_key_auth, for _tag_max_budget_check) merges the caller's x-litellm-tags header into the same metadata.tags list before add_litellm_data_to_request ever runs. The previous inherited_tags snapshot ("whatever's in tags before this function's own caller-tag merge") therefore misattributed that caller-controlled value as policy-backed whenever a request arrived with the header set -- Greptile flagged this as a P1 security finding. inherited_tags is now built directly from key_metadata/team_metadata/ project_metadata's own "tags" fields, independent of the shared, pipeline-position-dependent "tags" list's mutation history. Verified with a direct reproduction mirroring the real pipeline (calling apply_client_tag_policy_pre_auth on the same data dict before add_litellm_data_to_request, as user_api_key_auth actually does): the caller's header tag no longer appears in inherited_tags. Added a regression test exercising that same call order; confirmed it fails against the pre-fix snapshot approach and passes against this fix. * fix(router): make tag_routing_prefix configurable through update_settings/get_settings and UpdateRouterConfig router_settings.tag_routing_prefix was only ever applied via the Router() constructor. Router.update_settings's _allowed_settings (used directly by proxy_server.py's _add_router_settings_from_db_config for the DB-backed router_settings path) and get_settings's vars_to_include both omitted it, so an operator relying on that path had the value silently ignored -- flagged by veria-ai. Also adds it to UpdateRouterConfig (the pydantic schema behind POST /config/update), the same bug shape LIT-3152 previously fixed for retry_policy: a field missing from that schema gets silently dropped by model_dump(exclude_none=True) before update_settings is ever called. * chore(ui): regenerate schema.d.ts for UpdateRouterConfig.tag_routing_prefix Adding tag_routing_prefix to UpdateRouterConfig changed the proxy's OpenAPI spec; regenerate the dashboard's generated API types to match. * fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging Regenerated ruff-strict-budget.json and type-discipline-budget.json against the post-rebase merge-base. LIT002/LIT011 ceilings reflect this branch's true current counts (confirmed unchanged across the rebase by diffing against the pre-rebase commit); the base's own counts moved independently. * fix(lint): replace mutable-collection fallbacks with immutable ones in inherited_tags computation key_metadata/team_metadata/project_metadata's "tags" fallbacks used `or {}` / `or []` literals, each a LIT002 mutable-collection-construction violation that pushed the branch 4 over its ratchet ceiling relative to a moved base. Swapped to MappingProxyType({}) / () to match the immutable idiom the rest of tag_based_routing.py already uses; no behavior change, since both are falsy and only ever read via .get()/ unpacking. Tightens type-discipline-budget.json's LIT002 ceiling back down to match, fully closing that gap (LIT011 keeps a genuine 1-count gap from pre-existing, untouched lines in this file, non-gating). * fix(lint): suppress LIT011 on the two new data[...] mutation sites Both new lines follow this file's established data[...] mutation idiom for add_litellm_data_to_request, matching the existing suppression already on the inherited_tags line. * test(router): lock in fallback + tag-filtering interaction Cover the router-level fallbacks mechanism composing with tag-based routing: a plain negation exhausting a group correctly advances to the fallback group, the same exclusion tag exhausting every hop correctly raises, and allow_fail_open resolving locally must not spuriously trigger an unrelated external fallback. * chore: retrigger CI now that litellm-docs#814 is merged --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
…36466) * feat(proxy): per-key prompt caching auto-injection via enable_prompt_caching Adds a key-level enable_prompt_caching toggle that auto-injects Anthropic cache_control breakpoints on requests made with that key, without requiring the gateway-wide enable_anthropic_prompt_caching flag. The flag lives in key metadata, is stamped onto the request root by add_key_level_controls, rides kwargs into both the /chat/completions seeding path and the native /v1/messages path, and reuses every existing gate (anthropic/bedrock only, supports_prompt_caching, client markers win). Client-supplied body values are stripped as an untrusted root control field. Includes the Admin UI switch on key create and key edit plus a read-only settings row, and dedupes the key edit view's drifted initial-values objects. * fix(proxy): drop section comment and suppress LIT011 on key-level prompt caching stamp
…arch_fix fix(bedrock): send tool-search beta header for Haiku 4.5 on Invoke /v1/messages
…itellm_decrease_anys_fable5 # Conflicts: # ruff-strict-budget.json # type-discipline-budget.json
…ing_stream_fix fix(bedrock): preserve adaptive thinking effort through the /v1/messages bridge
ci: retry transient network fetch failures in lint workflow
* fix(alerting): dedupe scheduled Slack spend reports across pods Every pod ran its own weekly/monthly spend report jobs, prometheus fallback stats cron, and daily report loop, so deployments with multiple replicas or uvicorn workers received one copy per pod. Gate each scheduled send behind the shared PodLockManager redis lock. The lock is never released: its TTL (the full reporting window for the weekly interval job, whose per-pod anchors drift by boot time and jitter) doubles as a sent-this-window marker. acquire_lock returning None (no redis wired) proceeds, preserving single-pod behavior. Also generalize the pod lock could-not-acquire log line, which claimed to be about spend tracking for every consumer. Fixes #14809 * fix(alerting): harden spend report locks after adversarial review Weekly lock TTL gets an hour haircut: with ttl equal to the interval, the winner re-fires just before its own key expires, reacquires without a TTL refresh, and the key then lapses in time for a trailing pod to re-send. Job/lock ids move to litellm/constants.py per convention, and spend_report_frequency now rejects non-positive day counts, which previously coerced to an every-second schedule and would now compute a negative lock TTL that silently never sends. Adds the missing test coverage the review flagged: startup_event's pod_lock_manager wiring (identity-asserted), the prometheus closure's positive path, and the ungated immediate prometheus send pinned to exactly one await. * test(alerting): consolidate spend_report_frequency validator coverage Drops a duplicate non-positive-days test and parametrizes the survivor over the suffix half of the validator too * fix(alerting): route the startup prometheus fallback send through the pod lock Greptile caught that the boot-time send still ran once per pod when PROMETHEUS_URL is set, the same duplication class this PR removes * fix(alerting): make report lock acquisition non-reentrant Greptile caught that a pod booting within an hour of the fallback stats cron sent twice: the startup send takes the lock, then the cron fire hits acquire_lock's reacquire branch, which returns True for the holder. Window-marker gates now pass allow_reentrant=False so a live lock blocks everyone including its holder; leader-election consumers keep the reentrant default * test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer _initialize_slack_alerting_jobs now reads it for the pod lock manager, and spec=ProxyLogging blocks instance-only attributes
chore(typing): clear 1.6k basedpyright Any errors across 56 files
…ext_block_fix fix(bedrock): add text block to converse user messages carrying documents
…models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ox (#36568) * fix(deps): ship boto3 with the base SDK so bedrock works out of the box * keep boto3 listed in the proxy extra as well * scrub ambient AWS env vars in the base SDK bedrock smoke check
…y-deprecations fix(model_prices): add provider-announced deprecation dates for Bedrock, Mistral, Cohere and Gemini models
…4 -> 0.4.85, litellm 1.97.0 -> 1.98.0
…63772a chore: bump litellm-enterprise 0.1.54 -> 0.1.55, litellm-proxy-extras 0.4.84 -> 0.4.85, litellm 1.97.0 -> 1.98.0
yuneng-berri
approved these changes
Aug 11, 2026
yucheng-berri
approved these changes
Aug 11, 2026
…pan count (#36582) The otel trace tests asserted that a streamed call produces exactly one gen-AI span. The proxy opens one gen-AI span per upstream attempt, so a call the router retried carries an error span for every failed attempt beside the one that answered, and the assertion fails on a request that succeeded. Select the served attempt instead: drop spans whose otel.status_code is ERROR, require exactly one survivor, and run the TTFT and streaming-flag assertions against it. That keeps what these assertions exist for, a split trace or a stream logged as two served spans, while tolerating a retry. Only the failed attempt lacks TTFT, so the old code also had a second failure mode: when the first span happened to be the error one, the test reported the attribute as missing rather than as belonging to a different attempt. test_span_selection.py covers the selection itself against Jaeger-shaped payloads and carries no e2e marker, since reproducing a first-attempt failure live is not something a test can arrange.
* test(e2e): cover vendor strategy gaps for chat contract, image edits, auth, team activity Resolves the first slice of LIT-4778 (vendor API testing strategy): image edits happy path, chat multi-turn + validation + sanitization, LLM-route auth header matrix, and /team/daily/activity structure * test(e2e): expand vendor API strategy coverage across endpoints Adds validation cases on existing endpoint suites, plus vector stores, search, bedrock native, realtime HTTP secrets/calls, responses retrieve, files/batches contract, and chat stream SSE. Registers coverage cells for LIT-4778 * test(e2e): finish vendor strategy open items Audio transcription negatives, vector-store file attach/poll/search, OpenAI moderation category matrix across chat/messages/responses, and smoke model matrix for chat (LIT-4778) * test(e2e): harden vendor strategy suite against live env edges Fix stream [DONE] tracking, XSS no-crash contract, realtime model routing, vector store list/search models, responses validation, and provider-denied Bedrock paths so the suite is stable against a live proxy * test(e2e): rename suites, drop vendor_contract, fix greptile gaps Move shared status helpers into e2e_http, rename chat auth headers and chat security suites, remove vendor_contract and dev_config files_settings, and tighten transcription validation plus vector-store search assertions * test(e2e): route bedrock stream disconnects through e2e_http Catch mid-stream RequestException in the shared harness so bedrock native tests do not import requests directly
) * test(e2e): add failing reproducers for two open gateway bugs Both tests assert the behavior a customer expects and both are red today. They are reproducers, not regressions: the product is wrong, not the tests. Native passthrough returns almost none of the operational headers the managed route does. A /gemini/ generateContent call comes back with three x-litellm-* headers and no x-ratelimit-* at all, against sixteen and four on /v1beta/models/{m}:generateContent for the same prompt, and critically it omits x-litellm-response-cost. Customers front provider-native traffic through this route and read those headers to reconcile spend and pace themselves, so native traffic is currently invisible to the tooling that covers every other route. /budget/update rejects any model_max_budget with a 500. The reported symptom was model ids containing dots, and that reproduces (prisma raises "Unexpected `-5.2[FloatValue]` Expected `:`" because the key is interpolated into a GraphQL query unquoted, so glm-5.2 lexes as an identifier followed by a float), but the plain name gpt4o fails too, on a separate "model_max_budget should be of any of the following types: Json" type mismatch at budget_management_endpoints.py:173. Omitting the field returns 200. The test drives both names so the failure says whether per-model budgets are broken outright or only for punctuated ids; today it stops on the plain name, which is the wider bug. * test(e2e): add reproducer for unenforced end-user per-model rate limits model_max_budget accepts an rpm_limit alongside the spend cap, and /budget/new stores it: the create response echoes {"gemini-2.5-flash": {"rpm_limit": 1, "max_budget": 100.0, "budget_duration": "1d"}}. Attach that budget to an end user, drive three calls as that user, and all three return 200. The limit is accepted, persisted, and then ignored. The same shape already works when the budget hangs off a key, which is what makes this quietly dangerous: the API gives every indication the cap is in force. A customer using it to hold one end user to a slow rate on a shared key gets no throttling at all. Harness additions this needs: ModelBudgetEntry carries the rpm_limit/tpm_limit the route already accepts, BudgetNewBody and create_budget carry model_max_budget, and create_customer can attach an existing budget_id rather than only an inline max_budget. Red today, for the reason in the assertion message. * test(e2e): tighten model_max_budget reproducers and drop in-loop closure Trim the reproducer docstrings to the contract they assert, keeping the failure messages that document each red-by-design bug. Replace the nested per-model closure in the /budget/update test with a module-level predicate and a per-model helper so nothing closes over a loop variable, and fix the import order the merge left unsorted. * test(e2e): skip the three reproducers while their gateway bugs stay open The passthrough header contract, /budget/update model_max_budget, and end-user per-model rpm enforcement reproducers all still fail against staging by design. Skip each with the product gap named so the combined suite can gate merges on green while the collector keeps reporting the cells as uncovered. * test(e2e): validate model budget response contracts * refactor(e2e): unify model budget schema * refactor(e2e): reuse shared model budget type
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| self.guardrail_name, | ||
| source, | ||
| ) | ||
| return BedrockGuardrailResponse() |
Contributor
There was a problem hiding this comment.
Low: Guardrail bypass for tool-result content
An authenticated caller can send an Anthropic turn whose only payload is nested in tool_result.content; the extractor produces an empty list, and this success return lets the content reach the model without the configured Bedrock guardrail. Extract nested tool-result text before this branch, or fail closed when the original turn contains non-empty but unsupported content; only skip genuinely empty turns.
| rules_obj: Rules, | ||
| start_time: datetime.datetime, | ||
| *args: Any, # positional passthrough to the wrapped LLM call (ANN401 ignored, see ruff-strict.toml) | ||
| is_async_call: bool = True, |
Contributor
There was a problem hiding this comment.
Low: Request-controlled log correlation bypass
ProxyBaseLLMRequestProcessing invokes this function with function_setup(**self.data), so a caller-supplied top-level is_async_call: false binds to this parameter and constructs Logging with correlation stamping disabled. When request correlation is enabled, the attacker can make their request's process logs omit trace and session IDs; keep this flag outside the request-key namespace, such as through a positional-only internal helper, and reject or remove any client-supplied field with this name.
Contributor
PR overviewThis PR promotes the internal staging changes to main, including updates to Bedrock guardrail content handling and proxy request logging setup. Two security issues remain open. An authenticated caller can bypass configured Bedrock guardrail inspection for nested tool-result content, while a request-controlled flag can suppress trace and session correlation in process logs. Both are limited to specific request shapes, but the guardrail bypass enables attacker-supplied content to reach the model without the intended screening. Open issues (2)
Fixed/addressed: 0 · PR risk: 6/10 |
… queue time (#34650) * test(e2e): cover google-native generateContent framing and prometheus queue time Adds live coverage for three shipped regressions that had none, all reached through surfaces a customer drives from Google SDKs and operator dashboards. The managed google-native route (`/v1beta/models/{model}:generateContent`) had no harness support at all, so EndpointsClient gains generate_content and stream_generate_content plus the request body models, and a new suite asserts the two contracts that broke there: the response carries x-litellm-response-cost so SDK traffic reconciles against spend (LIT-4076), and the stream relays single-prefixed SSE frames with no OpenAI [DONE] terminator. A doubled `data:` prefix, a leaked bytes literal, or the [DONE] sentinel each fail the stream test; [DONE] absence is only asserted once real content has arrived, because a first-chunk upstream error legitimately falls back to the OpenAI error shape and does emit it. The prometheus test pins litellm_request_queue_time_seconds to an actual observation on our own key's series rather than to the family merely existing, which is the distinction the original regression turned on: the histogram stayed registered while nothing was ever written to it (LIT-2034). Each assertion was mutation-checked against the live proxy; inverting the [DONE] expectation, the cost-header expectation, or the metric name fails the corresponding test. * refactor(e2e): simplify google native coverage
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Routine promotion of
litellm_internal_stagingintomain.Notable changes in this batch:
Recent promotions: #28709, #28680, #28292
🤖 Generated with Claude Code