Skip to content

refactor(proxy): expose public names for private proxy helpers - #45170

Merged
mateo-berri merged 20 commits into
mainfrom
litellm_private_usage_proxy
Oct 8, 2026
Merged

mateo-berri merged 20 commits into
mainfrom
litellm_private_usage_proxy

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • reportPrivateUsage on main is 800, and 613 of those hits point at litellm/proxy/ symbols
  • Proxy modules, the SDK, and tests reach into each other's _private helpers

How it solves it:

  • 261 proxy functions, classes, and methods get public names, and the old _name stays as an alias to the same object (_should_return_raw_model_name: Final = should_return_raw_model_name, _handle_llm_api_exception = handle_llm_api_exception in the class body), so existing imports and patches keep working
  • The 17 renamed classes keep their original _PROXY_* style definitions and the public name is the alias (PROXY_MaxParallelRequestsHandler_v3 = _PROXY_MaxParallelRequestsHandler_v3), because __name__ drives the no-log callback filter, internal-callback checks, and health and guardrail names
  • Same fix for _PROXY_LiteLLMManagedFiles and _PROXY_LiteLLMManagedVectorStores, which refactor: expose core private helpers under public names #44871 had renamed so their class names lost _PROXY_ on main
  • Every module that imported an old private name at module level still binds it next to the public name (194 bindings in 79 modules, plus the hooks and policy_endpoints packages), so from litellm.proxy.proxy_server import _ProxyDBLogger keeps working
  • _can_object_call_model keeps its private definition with a public alias, since the recursive-function check lists it by name
  • ProxyLogging.convert_mcp_to_llm_format is a public forwarder that calls the old private method, since hooks may override it
  • 9 SDK private-member reads get exact reportPrivateUsage suppressions with a reason
  • 23 symbols were skipped: slotted dataclasses, a Pydantic model, private instance attributes and Prisma locks, _license_check (a rebound global), names whose public twin has a different contract like ProxyConfig._add_deployment, and helpers whose callers live in enterprise/
  • Function bodies are unchanged. Lines the rename touched picked up the : Final (175) or # rebind-ok (65) the type-discipline gate now checks on changed lines, and the 63 newly public functions got the return type basedpyright already inferred, including Never for raise-only helpers
  • reportPrivateUsage goes from 800 to 258 and no basedpyright rule goes up
  • Last of four PRs, after refactor: expose core private helpers under public names #44871, refactor: expose public hidden_params accessors #44668, and refactor(llms): expose public names for private provider helpers #45037

User Flow

Before: a proxy user sends LLM and management requests and gets normal responses

  1. They POST http://localhost:4000/v1/chat/completions, /v1/responses, /v1/messages, or /v1/embeddings, and call /key/generate, /team/new, /user/info, or /spend/logs
  2. Code that imports an old helper like from litellm.proxy.utils import _check_and_merge_model_level_guardrails works

After: the same requests behave the same, and the public names are also available

  1. They make the same calls and get the same status, cost header, model id header, and response shape
  2. The old import still works and is the same object as check_and_merge_model_level_guardrails

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally
  • My PR passes all required CI/CD checks
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Two proxies on the same config, real Postgres, and real provider keys, Before at main 3538e87 and After at this branch, each sent the same 18 requests

curl -si http://127.0.0.1:$PORT/v1/chat/completions \
  -H "Authorization: Bearer $AUTH" -H "Content-Type: application/json" \
  -d '{"model":"bedrock-chat","messages":[{"role":"user","content":"Reply with only OK."}],"temperature":0,"max_tokens":8}'
Case Before status, cost After status, cost
OpenAI chat 200, 2.4e-06 200, 2.4e-06
OpenAI chat stream, usage chunk 200 200
OpenAI Responses 200, 3e-06 200, 3e-06
Anthropic /v1/messages with tools 200, 0.000819 200, 0.000819
Anthropic chat with tools and stop 200, 0.000819 200, 0.000819
OpenAI embeddings 200, 1e-07 200, 1e-07
Invalid key with fallback 200, 2.4e-06 200, 2.4e-06
Bedrock chat 200, 3.52e-05 200, 3.52e-05
Vertex Gemini chat 404, 0 404, 0
/key/generate, then chat with that key 200, 200 200, 200
/key/info 200 200
/team/new, then /team/info 200, 200 200, 200
/user/info, /model/info, /health/readiness 200 200
/spend/logs after the calls 200, 42 rows 200, 42 rows

Model id headers and response shapes matched in every case. The Vertex 404 is the project lacking that model, identical on both sides

tests/unit/test_private_usage_aliases.py checks alias identity for every renamed proxy symbol and that the forwarder reaches a subclass override of the private name with the exact arguments. Swapping two arguments in the forwarder makes it fail

Type

🧹 Refactoring

Caveats (if any)

Low

  • Callers now use the public method names, so a subclass of an internal proxy class such as ProxyBaseLLMRequestProcessing that overrides the old _name is no longer reached through those callers. None of the 216 aliased methods are on documented extension points like CustomLogger or CustomGuardrail
  • Tests that patch a private name production now calls by its public name were retargeted to the public name

Link to Devin session: https://app.devin.ai/sessions/2ca967be4d764e32bf06b907fdafcf4a
Open in Devin Desktop: https://app.devin.ai/desktop/session/2ca967be4d764e32bf06b907fdafcf4a?variant=devin
Requested by: @mateo-berri


Note

Low Risk
Behavior-preserving renames and alias bindings; no auth or billing logic changes in the diff, only which symbol names production code imports.

Overview
This PR continues the proxy public API refactor by switching internal call sites from underscore-prefixed helpers to their public twins, while keeping the old names as module-level aliases (often with Final and legacy F401 re-exports) so existing imports and patches still resolve to the same objects.

Enterprise hooks: PROXY_LiteLLMManagedFiles and PROXY_LiteLLMManagedVectorStores are defined again as _PROXY_* classes with public aliases at module bottom, matching the callback-registry __name__ pattern used for dynamic rate limiters (PROXY_DynamicRateLimitHandler in custom_logger_registry / litellm_logging).

Cross-cutting renames in this diff: file-id helpers (is_base64_encoded_unified_file_id), spend reporting (get_spend_report_for_time_range), auth/budget checks (premium_user_check, team_max_budget_check, can_object_call_model, run_centralized_common_checks), request parsing (read_request_body, safe_get_request_headers), passthrough streaming (route_streaming_logging_to_handler), and Prometheus key listing (list_key_helper).

MCP / OAuth surface (largest footprint): MCPRequestHandler header/body helpers, bridge mint/refresh (prepare_bridge_mint, finish_bridge_mint, bridge_mint_error_response), token TTL (compute_per_user_token_ttl), encryption (get_salt_key, V2_GCM_PREFIX), OpenAPI context vars (request_auth_header, etc.), and MCPServerManager methods (get_tools_from_server, should_strip_caller_authorization, …) are called by public names; private names remain as aliases on the defining modules.

Reviewed by Cursor Bugbot for commit 46b4328. Bugbot is set up for automated code reviews on this repo. Configure here.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Renames private helper functions to public names across proxy modules.

The PR appears safe to merge, with all numbered previous findings fixed and no new actionable issue found

What we checked:

  • Renamed baseline entries still match: The checker takes function names from the source. Both updated entries match the current definitions and preserve the same query details

Summary

This PR exposes public proxy helper names while retaining old imports and class names

  • The three numbered findings from earlier reviews are fixed
  • Since the previous review, only two baseline entries changed. They now match the renamed functions without adding exemptions
  • mateo-berri acknowledges that callers using public methods no longer reach overrides of old private methods on internal classes. The stated reason is that these methods are not documented extension points

Reviews (7) · Last reviewed commit: "test(proxy): update IN-list baseline for..." · Reviewed by Greptile

Comment thread litellm/proxy/hooks/parallel_request_limiter_v3.py Outdated
mateo-berri and others added 4 commits October 7, 2026 21:41
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@codspeed

codspeed Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_private_usage_proxy (46b4328) with main (0e07600)

Open in CodSpeed

@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment on lines 11 to 20
from litellm.proxy.management_endpoints.policy_endpoints.endpoints import ( # noqa: F401
_build_all_names_per_competitor,
_build_comparison_blocked_words,
_build_competitor_guardrail_definitions,
_build_name_blocked_words,
_build_recommendation_blocked_words,
_build_refinement_prompt,
_clean_competitor_line,
_parse_variations_response,
build_all_names_per_competitor,
build_comparison_blocked_words,
build_competitor_guardrail_definitions,
build_name_blocked_words,
build_recommendation_blocked_words,
build_refinement_prompt,
clean_competitor_line,
parse_variations_response,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Old policy imports fail

Existing imports such as from litellm.proxy.management_endpoints.policy_endpoints import _build_refinement_prompt now fail, stopping the caller's code from loading. The wildcard import excludes private names, and this list now exports only public names. Keeping aliases in endpoints.py does not preserve the package imports

Keep both names here. Dropping the old exports violates the repository directive against backwards-incompatible changes without user-controlled flags

Suggested change
from litellm.proxy.management_endpoints.policy_endpoints.endpoints import ( # noqa: F401
_build_all_names_per_competitor,
_build_comparison_blocked_words,
_build_competitor_guardrail_definitions,
_build_name_blocked_words,
_build_recommendation_blocked_words,
_build_refinement_prompt,
_clean_competitor_line,
_parse_variations_response,
build_all_names_per_competitor,
build_comparison_blocked_words,
build_competitor_guardrail_definitions,
build_name_blocked_words,
build_recommendation_blocked_words,
build_refinement_prompt,
clean_competitor_line,
parse_variations_response,
)
from litellm.proxy.management_endpoints.policy_endpoints.endpoints import ( # noqa: F401
_build_all_names_per_competitor,
_build_comparison_blocked_words,
_build_competitor_guardrail_definitions,
_build_name_blocked_words,
_build_recommendation_blocked_words,
_build_refinement_prompt,
_clean_competitor_line,
_parse_variations_response,
build_all_names_per_competitor,
build_comparison_blocked_words,
build_competitor_guardrail_definitions,
build_name_blocked_words,
build_recommendation_blocked_words,
build_refinement_prompt,
clean_competitor_line,
parse_variations_response,
)

Rule Used: What: avoid backwards-incompatible changes without user-controlled flags Why: This breaks current behaviour for users using existing functionality Example of BAD: this PR (#22164) introduced run_post_custom... (source)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Restoring the old private names next to the public ones in this and the hooks package export, with an import test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
mateo-berri and others added 5 commits October 7, 2026 23:31
…_proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	litellm/proxy/management_endpoints/common_utils.py
#	litellm/proxy/management_endpoints/key_management_endpoints.py
#	litellm/proxy/management_endpoints/team_endpoints.py
#	tests/unit/proxy/auth/test_route_checks.py
#	tests/unit/proxy/management_endpoints/test_common_utils.py
…_proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	litellm/proxy/utils.py
#	tests/unit/proxy/_experimental/mcp_server/test_mcp_guardrail_usage_monitor.py
#	tests/unit/proxy/_experimental/mcp_server/test_mcp_server_manager.py
#	tests/unit/proxy/hooks/test_parallel_request_limiter_v3.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
mateo-berri and others added 2 commits October 8, 2026 00:20
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@@ -129,7 +129,7 @@ def test_hash_token_if_needed_handles_sk_prefix() -> None:

def test_hash_token_if_needed_error_on_non_string() -> None:
with pytest.raises(AttributeError):
_hash_token_if_needed(None) # type: ignore[arg-type]
hash_token_if_needed(None) # type: ignore[arg-type]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test suppression does nothing

The updated call retains # type: ignore[arg-type], which pyright ignores. AGENTS.md bans this syntax and requires an exact pyright rule with a reason. Replace it with # pyright: ignore[reportArgumentType] and explain that the invalid input is intentional. This repository requirement must be satisfied before merging

Suggested change
hash_token_if_needed(None) # type: ignore[arg-type]
hash_token_if_needed(None) # pyright: ignore[reportArgumentType] # intentional invalid input checks the error

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Switching it to an exact reportArgumentType suppression with a reason, since the None input is intentional

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 46b4328. Configure here.

@mateo-berri
mateo-berri enabled auto-merge (squash) October 8, 2026 01:52
@mateo-berri
mateo-berri merged commit 5825840 into main Oct 8, 2026
118 of 124 checks passed
@mateo-berri
mateo-berri deleted the litellm_private_usage_proxy branch October 8, 2026 01:59
shudonglin pushed a commit to rayward-external/litellm that referenced this pull request Oct 8, 2026
…ermission's payload

Upstream PR BerriAI#45170 retyped set_object_permission's data_json parameter and
return value as dict[str, object], which silently broke basedpyright's
argument-compatibility checking at its three call sites (key, user, and
team creation), each of which unpacks the dict as **kwargs into
generate_key_helper_fn's dozens of concretely-typed parameters. object
is a real, narrow type that must be checked; the untyped dict the
function had before upstream's change was treated permissively, so the
incompatibility never surfaced. Any is the correct type for this
JSON-like payload destined for **kwargs unpacking, and restores the
previous permissive behavior without widening the function's own
contract.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGP2qHGkEN9qza1tM2AswG
devYRPauli added a commit to devYRPauli/litellm that referenced this pull request Oct 8, 2026
Upstream BerriAI#45170 makes _read_request_body and _get_metadata_variable_name
public. The PR call sites use the new names.

Upstream BerriAI#40193 gates every lint rule at its merge-base count. The PR
lines now carry mutable-ok and rebind-ok markers with reasons.
filter_pinned_deployment is generic over the deployment type. The new
helpers read untyped dicts through a pydantic TypeAdapter.

# Conflicts:
#	litellm/proxy/common_request_processing.py
#	litellm/proxy/video_endpoints/endpoints.py
heyzainali added a commit to heyzainali/litellm that referenced this pull request Oct 10, 2026
…-to-org-config

Resolve the user search check against the user_api_key_has_admin_view
rename from BerriAI#45170

Keep UI_SETTINGS_CACHE_KEY and UI_SETTINGS_CACHE_TTL, which
moyai_endpoints now imports for its own cache write

This branch is waiting to be deployed

1 waiting deployment
e2e-changed — 46b4328c Waiting Oct 8, 2026 by devin-ai-integration[bot] via oauth #3690
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants