[Infra] Merge internal dev branch with main - #25036
Conversation
custom_auth_run_common_checks only runs common_checks (team/user/project model checks). Custom auth now also enforces key-level model restrictions via can_key_call_model. Move the custom-auth key-access regression tests to test_user_api_key_auth.py and keep test_custom_auth_end_user_budget.py focused on end-user budget behavior. Made-with: Cursor
Keep key-level model allowlist enforcement in custom auth behind `custom_auth_run_common_checks` to preserve backwards compatibility, and update tests to verify default non-enforcement and opt-in enforcement behavior. Made-with: Cursor
…tate Patch `proxy_server.general_settings` to an empty dict in the default custom-auth key-access test so it remains deterministic under shared module state. Made-with: Cursor
Tighten custom auth regression tests by asserting exact can_key_call_model args and remove an unused common_checks mock from the default behavior path. Made-with: Cursor
Made-with: Cursor
Add generic docs for running JWT and OAuth2 together, including routing_overrides YAML examples and list-based selector behavior for iss/client_id/aud. Made-with: Cursor
feat(auth): add JWT claim routing overrides for OAuth2 validation
…ey-model-allowlist
…allowlist fix(proxy): enforce key-level model restrictions for custom auth
…ponse-parsing fix(agentcore): parse A2A JSON-RPC responses in AgentCore provider
…g-roles-tool-calls fix(prompt-templates): ensure_alternating_roles handles tool-call chains
|
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 29203053 | Triggered | Generic Password | 6058de4 | .circleci/config.yml | View secret |
| 29375658 | Triggered | JSON Web Token | 6058de4 | tests/test_litellm/proxy/auth/test_handle_jwt.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
Greptile SummaryThis PR merges an internal dev branch into main, landing four distinct improvements: (1) JWT routing overrides for the auth layer so that JWT-shaped machine tokens can be directed to OAuth2 introspection instead of the JWT path based on unverified claims ( Key items to verify before merging:
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| litellm/proxy/auth/user_api_key_auth.py | Adds JWT routing-override helpers (_routing_selector_matches_claim, _matches_routing_override, _should_route_jwt_to_oauth2_override) and wires them into the OAuth2/JWT auth gate; also extracts _enforce_key_and_fallback_model_access and extends _run_post_custom_auth_checks to enforce it when custom_auth_run_common_checks is True. |
| litellm/proxy/auth/handle_jwt.py | Promotes jwt/PyJWK imports to module level, extracts SUPPORTED_JWT_ALGORITHMS as class constant, and adds get_unverified_claims() static method for claim-based routing decisions. |
| litellm/proxy/_types.py | Adds JWTRoutingOverride Pydantic model (iss, client_id, aud, path) and routing_overrides field to LiteLLM_JWTAuth. |
| litellm/litellm_core_utils/prompt_templates/common_utils.py | Refactors _insert_assistant_continue_message to use _counts_for_alternation, now inserting assistant_continue across tool-call chains — a backward-incompatible behavior change without a feature flag. |
| litellm/llms/bedrock/chat/agentcore/transformation.py | New AmazonAgentCoreConfig: handles SigV4/Bearer auth, SSE + JSON streaming, A2A JSON-RPC parsing, and transforms AgentCore responses to LiteLLM ModelResponse format. |
| tests/llm_translation/test_prompt_factory.py | Three existing tests have their expected outputs updated to reflect new assistant_continue injection behavior; several new tests added verifying tool-chain scenarios. |
| tests/test_litellm/proxy/auth/test_user_api_key_auth.py | Adds thorough mock-only tests for JWT routing override: matching, non-matching client_id, list selectors, and backward-compat (OAuth2-only). |
| tests/test_litellm/llms/bedrock/chat/agentcore/test_agentcore_transformation.py | Adds four new unit tests for A2A JSON-RPC response parsing strategies (nested message, direct parts, multi-parts, empty fallback). |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Incoming Request with Bearer Token] --> B{enable_oauth2_auth?}
B -- No --> G{enable_jwt_auth?}
B -- Yes --> C{is_llm_api_route or is_info_route?}
C -- No --> G
C -- Yes --> D{enable_jwt_auth AND token is JWT-shaped?}
D -- No --> E[OAuth2 Introspection\nOauth2Handler.check_oauth2_token]
D -- Yes --> F{_should_route_jwt_to_oauth2_override?\ncheck unverified claims vs routing_overrides}
F -- No match --> G
F -- Match found --> E
G -- No --> H[Virtual Key / DB Auth Path]
G -- Yes --> I{token is JWT-shaped?}
I -- No --> H
I -- Yes --> J[JWT Auth\nJWTAuthManager.auth_builder]
J --> K[UserAPIKeyAuth]
E --> K
H --> K
Comments Outside Diff (1)
-
litellm/llms/bedrock/chat/agentcore/transformation.py, line 134-138 (link)JWT/Bearer token value partially exposed in debug log
The first 50 characters of the bearer token are written to the log at
DEBUGlevel. A typical JWT begins witheyJhbGciOi...(the base64-encoded header), so the first 50 chars can leak the signing algorithm and token type. WhileDEBUGis not normally enabled in production, a truncated token value is still sensitive. Consider logging only a non-sensitive indicator (e.g. whether a token was provided) instead:
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin' in..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| """ | ||
| Add assistant continuation messages between consecutive user messages. | ||
|
|
||
| Only checks directly adjacent messages to preserve backward compatibility. | ||
| Skips tool messages and assistant messages with tool calls in the | ||
| alternation check, matching strict templates like llama.cpp. | ||
| """ | ||
| if not ensure_alternating_roles or len(messages) <= 1: | ||
| return messages | ||
|
|
||
| continue_message = assistant_continue_message or DEFAULT_ASSISTANT_CONTINUE_MESSAGE | ||
|
|
||
| # Find indexes where assistant_continue should be inserted (before that index) | ||
| insert_before_indexes: set = set() | ||
|
|
||
| for i in range(len(messages)): | ||
| curr = messages[i] | ||
| if _counts_for_alternation(curr) and curr["role"] == "user": | ||
| # Look backwards for the previous counted message | ||
| j = i - 1 | ||
| while j >= 0: | ||
| if _counts_for_alternation(messages[j]): | ||
| if messages[j]["role"] == "user": | ||
| insert_before_indexes.add(i) | ||
| break | ||
| j -= 1 | ||
|
|
||
| # Build the result with assistant_continue inserted at the right positions | ||
| modified_messages: List[AllMessageValues] = [] | ||
| for i, message in enumerate(messages): | ||
| if ( | ||
| i < len(messages) - 1 | ||
| and message.get("role") == "user" | ||
| and messages[i + 1].get("role") == "user" | ||
| ): | ||
| modified_messages.append(message) | ||
| if i in insert_before_indexes: | ||
| modified_messages.append(continue_message) | ||
| else: | ||
| modified_messages.append(message) | ||
| modified_messages.append(message) | ||
|
|
||
| return modified_messages |
There was a problem hiding this comment.
The previous implementation only inserted assistant_continue between two directly adjacent user messages. The new implementation uses _counts_for_alternation to skip tool messages and assistant(tool_calls) messages, meaning a pattern like:
[user, assistant(tool_calls), tool, user]
now gets assistant_continue inserted before the second user message — behavior that did not exist before. Three existing tests (test_ensure_alternating_roles_does_not_split_tool_call_chain, test_ensure_alternating_roles_assistant_tool_call_then_assistant, test_ensure_alternating_roles_trailing_tool_call_assistant) were updated to match the new behavior, confirming this is intentional.
However, per the project rule on backwards-incompatible changes, existing users who have ensure_alternating_roles=True and agentic/tool-calling message patterns will silently receive extra injected assistant_continue messages, potentially breaking their existing model calls without any migration path.
The safer approach would be to introduce this under a flag (e.g. ensure_alternating_roles="strict" or a separate skip_tool_messages_in_alternation param) so existing users are not affected.
Rule Used: What: avoid backwards-incompatible changes without... (source)
|
|
||
| def test_ensure_alternating_roles_does_not_split_tool_call_chain(): | ||
| """Tool-call chains [user, assistant(tc), tool, user] are preserved as-is.""" | ||
| def test_ensure_alternating_roles_inserts_assistant_continue_across_tool_chain(): |
There was a problem hiding this comment.
The original test was named test_ensure_alternating_roles_does_not_split_tool_call_chain and its docstring read: "Tool-call chains [user, assistant(tc), tool, user] are preserved as-is." This name and docstring were documenting a deliberate design contract.
The rename to test_ensure_alternating_roles_inserts_assistant_continue_across_tool_chain and the addition of the injected assistant_continue to the expected output reverses that documented contract. While this may be an intentional behavior improvement, it's worth confirming that no downstream callers depended on the old guarantee before treating the changed expected output as correct.
This also applies to test_ensure_alternating_roles_assistant_tool_call_then_assistant and test_ensure_alternating_roles_trailing_tool_call_assistant, both of which add a new {"role": "assistant", "content": "Please continue."} line to their expected message lists.
Rule Used: What: Flag any modifications to existing tests and... (source)
| def _should_route_jwt_to_oauth2_override(token: str, jwt_handler: JWTHandler) -> bool: | ||
| routing_overrides = jwt_handler.litellm_jwtauth.routing_overrides | ||
| if not routing_overrides: | ||
| return False | ||
|
|
||
| token_claims = jwt_handler.get_unverified_claims(token=token) | ||
| if token_claims is None: | ||
| return False | ||
|
|
||
| for override in routing_overrides: | ||
| if override.path == "oauth2" and _matches_routing_override( | ||
| token_claims=token_claims, override=override | ||
| ): | ||
| verbose_proxy_logger.debug( | ||
| "JWT routing override matched. Routing token to OAuth2 introspection." | ||
| ) | ||
| return True | ||
|
|
||
| return False |
There was a problem hiding this comment.
litellm_jwtauth may be unset at first request
The litellm_jwtauth attribute is assigned in JWTHandler.update_environment(), not in __init__, so it may not exist if the handler was constructed but environment setup was skipped. Accessing a missing attribute will raise AttributeError and surface as an opaque 500 to the caller.
A getattr(jwt_handler, "litellm_jwtauth", None) guard at the top of this function would safely return False in that scenario.
* fix(proxy): enforce key-level model allowlist for custom auth custom_auth_run_common_checks only runs common_checks (team/user/project model checks). Custom auth now also enforces key-level model restrictions via can_key_call_model. Move the custom-auth key-access regression tests to test_user_api_key_auth.py and keep test_custom_auth_end_user_budget.py focused on end-user budget behavior. Made-with: Cursor * fix(proxy): gate custom-auth key model checks behind opt-in Keep key-level model allowlist enforcement in custom auth behind `custom_auth_run_common_checks` to preserve backwards compatibility, and update tests to verify default non-enforcement and opt-in enforcement behavior. Made-with: Cursor * test(proxy): isolate custom auth default check from shared settings state Patch `proxy_server.general_settings` to an empty dict in the default custom-auth key-access test so it remains deterministic under shared module state. Made-with: Cursor * test(proxy): strengthen custom auth post-check assertions Tighten custom auth regression tests by asserting exact can_key_call_model args and remove an unused common_checks mock from the default behavior path. Made-with: Cursor * fix(agentcore): parse A2A JSON-RPC responses in AgentCore provider * fix(prompt-templates): ensure_alternating_roles handles tool-call chains * feat(auth): add JWT claim routing overrides for OAuth2 validation Made-with: Cursor * docs(auth): document JWT-to-OAuth2 routing overrides Add generic docs for running JWT and OAuth2 together, including routing_overrides YAML examples and list-based selector behavior for iss/client_id/aud. Made-with: Cursor --------- Co-authored-by: Milan <milan@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
5 participants
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🚄 Infrastructure
Changes