fix(proxy): keep the caller's Google token on credential-less Vertex passthrough under custom auth - #38299
Conversation
…ertex passthrough PR #38114 dropped whichever header user_api_key_auth would read the caller's key from, by precedence. Under custom_auth, JWT auth, or no master key that header is the caller's own Google token, so the bring-your-own-credentials Vertex branch answered 401 to every valid request. A header value is now dropped only when it is the master key or when its hash is the api_key that authenticated the request, so a Google token that auth never consumed keeps flowing while a LiteLLM key still never reaches Google. test_passthrough_post_call_guardrails.py no longer plants a MagicMock proxy_server module in sys.modules at import, which poisoned sibling tests that read module globals at call time.
Greptile SummaryThis PR updates credential-less Vertex passthrough to preserve caller-provided Google credentials while removing the proxy secret that authenticated the request.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py | Refines credential classification and forwarding for credential-less Vertex passthrough; no eligible follow-up defect remains. |
| tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py | Adds focused regression tests for preserving Google credentials and stripping authenticated proxy secrets. |
| tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_post_call_guardrails.py | Removes an obsolete proxy-server import shim without weakening the guardrail assertions. |
| tests/test_litellm/proxy/pass_through_endpoints/test_vertex_passthrough_load_balancing.py | Updates direct route tests to supply the newly required authentication context. |
Reviews (5): Last reviewed commit: "fix(proxy): strip the JWT that authentic..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| authenticated_key: Final = user_api_key_dict.api_key | ||
| if authenticated_key is None: | ||
| return False | ||
| return hmac.compare_digest(hash_token(normalized).encode(), authenticated_key.encode()) |
There was a problem hiding this comment.
Medium: Custom-auth credentials can be forwarded upstream
UserAPIKeyAuth only converts sk- credentials to a plain SHA-256 value; opaque credentials remain unchanged and JWT credentials use a hashed-jwt- prefix. Consequently, this hash-only comparison returns false when a custom auth handler returns UserAPIKeyAuth(api_key=api_key) for either form, and the same credential in Authorization is forwarded to Google. Track whether the auth path actually consumed the header as explicit server-side provenance rather than inferring it from the stored key representation.
There was a problem hiding this comment.
Under custom auth the Authorization value is the caller's own Google token, so marking it consumed strips it and brings back the 401 fixed here
There was a problem hiding this comment.
Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.
There was a problem hiding this comment.
The validator applies the same transform on both sides, so opaque and JWT custom-auth credentials strip; tests custom-auth-echoing-opaque-credential and custom-auth-echoing-jwt prove it
There was a problem hiding this comment.
Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.
PR overviewThis pull request updates credential-less Vertex passthrough behavior under custom authentication so the caller’s Google token is preserved when forwarding requests. One security issue remains open: under certain custom-auth configurations, an opaque or JWT authorization credential consumed by the proxy can also be forwarded to Google. This could disclose caller credentials to the upstream service, though exploitation depends on the custom authentication path and credential format. No issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 5/10 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: JWT auth tokens leak upstream
- Extended
_is_authenticated_caller_secretto also treat a JWT-shaped header value as the authenticating secret whenuser_api_key_dict.jwt_claimsis set, matching production JWT auth whereapi_keyis None or holds a JWT-mapped virtual key hash, so the caller's JWT is stripped from Authorization before the surviving-credential guard runs.
- Extended
Or push these changes by commenting:
@cursor push a2cdaa531e
Preview (a2cdaa531e)
diff --git a/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py b/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
--- a/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
+++ b/litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
@@ -1817,11 +1817,14 @@
def _is_authenticated_caller_secret(value: str, user_api_key_dict: UserAPIKeyAuth) -> bool:
"""Whether a header value is the master key or the key ``user_api_key_auth`` stored as ``api_key``."""
+ from litellm.proxy.auth.handle_jwt import JWTHandler
from litellm.proxy.proxy_server import master_key
normalized: Final = _normalize_credential_value(value)
if master_key is not None and hmac.compare_digest(normalized.encode(), master_key.encode()):
return True
+ if user_api_key_dict.jwt_claims is not None and JWTHandler.is_jwt(token=normalized):
+ return True
authenticated_key: Final = user_api_key_dict.api_key
if authenticated_key is None:
return False
diff --git a/tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py b/tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
--- a/tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
+++ b/tests/test_litellm/proxy/pass_through_endpoints/test_llm_pass_through_endpoints.py
@@ -3823,6 +3823,16 @@
[
pytest.param("modified_key", UserAPIKeyAuth(api_key="modified_key"), id="custom-auth-echoing-opaque-credential"),
pytest.param(LITELLM_JWT, UserAPIKeyAuth(api_key=LITELLM_JWT, user_id="jwt-subject"), id="jwt-auth-consuming-header"),
+ pytest.param(
+ LITELLM_JWT,
+ UserAPIKeyAuth(api_key=None, user_id="jwt-subject", jwt_claims={"sub": "jwt-subject"}),
+ id="jwt-only-auth-with-null-api-key",
+ ),
+ pytest.param(
+ LITELLM_JWT,
+ UserAPIKeyAuth(api_key=VKEY, user_id="jwt-subject", jwt_claims={"sub": "jwt-subject"}),
+ id="jwt-mapped-virtual-key-with-jwt-still-in-authorization",
+ ),
],
)
async def test_non_sk_litellm_credential_that_authenticated_is_rejected_not_forwarded(You can send follow-ups to the cloud agent here.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a3f6547. Configure here.
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer whose proxy runs a custom auth plugin and has no Vertex credential sends Vertex requests with their own Google token and gets 401
custom_authpointing at their own auth plugin and no Vertex credential configured on the proxygcloud auth print-access-tokenand sends POST https://litellm-domain/vertex_ai/v1/projects/{project}/locations/global/publishers/google/models/gemini-3.5-flash-lite:generateContent withAuthorization: Bearer ya29...and acontentsbodyAuthorizationalso gets the 401 and that key is never forwarded to GoogleAfter: the same request is forwarded to Google with the developer's own token and comes back 200
custom_authpointing at their own auth plugin and no Vertex credential configured on the proxygcloud auth print-access-tokenand sends POST https://litellm-domain/vertex_ai/v1/projects/{project}/locations/global/publishers/google/models/gemini-3.5-flash-lite:generateContent withAuthorization: Bearer ya29...and acontentsbodycandidatesresponse and anx-litellm-call-idheaderRelevant issues
Regression introduced by #38114. First red run of the CI job: https://app.circleci.com/workflow/7e7cff9b-5f81-47bc-9377-555351c9a8f1/job/342188e8-152c-42d8-83e6-d354ea58f688
Linear ticket
Resolves LIT-6175
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both legs share the same rig and differ only in the commit the proxies were booted from: Before at the merge base, After at the PR tip. Each leg is its own worktree, its own
.venv, and its own Postgres database, with proxies booted from variants oflitellm/proxy/example_config_yaml/pass_through_config.yaml, every one with--num_workers 2 --timeout_worker_healthcheck 120. Only proxy C holds a Vertex credential. Every 200 below is a real Vertex call against a real GCP project with real spendProxy F (JWT auth) runs on the After leg only: the JWT-stripping branch landed mid-PR, its regression proof lives in the new unit tests, and F proves the tip end-to-end. Its config sets
enable_jwt_auth: truepluslitellm_jwtauth.admin_allowed_routesextended with"/vertex_ai/*", validating against a local JWKS static server, and the caller authenticates with a self-minted RS256 admin JWT (scope: litellm_proxy_admin)Shared shell prelude (per leg;
PORT_Xfrom the table):The 396-byte guard body is shown in full once per side and elided to its first clause afterwards (it is byte-identical each time), and unchanged
usageMetadata/thoughtSignaturefields inside Gemini 200 bodies are elidedBefore (273b01a)
A1 custom auth, caller's own Google token
curl -sS -i "$URL_A" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
The caller's own token was stripped as if it were the virtual key: this is the regression
A2 custom auth, caller's own token, SSE stream
curl -sS -i -m 60 "http://127.0.0.1:${PORT_A}/vertex_ai/v1/projects/vertex-check-481318/locations/global/publishers/google/models/gemini-3.5-flash-lite:streamGenerateContent?alt=sse" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
A3 spend log for A1
x-litellm-call-idheader, so there is no request id to queryA4 custom auth, master key alone
curl -sS -i "$URL_A" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
A5 custom auth, virtual key alone
curl -sS -i "$URL_A" -H "Authorization: Bearer $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
Guard body, not a Google error: the virtual key never reached Google
A6 custom auth, virtual key header plus caller token
curl -sS -i "$URL_A" -H "x-litellm-api-key: $VKEY" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
curl -s "http://127.0.0.1:${PORT_A}/spend/logs?request_id=265e9867-4f98-48fd-ad47-6f0517ec9eab" -H "Authorization: Bearer sk-1234"Observed:
B1 standard auth, virtual key alone
curl -sS -i "$URL_B" -H "Authorization: Bearer $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
B2 standard auth, virtual key in x-goog-api-key
curl -sS -i "$URL_B" -H "x-goog-api-key: $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
Auth-layer 401, not the guard; the key still never reached Google
B3 standard auth, master key alone
curl -sS -i "$URL_B" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
B4 standard auth, virtual key header plus caller token
curl -sS -i "$URL_B" -H "x-litellm-api-key: $VKEY" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
curl -s "http://127.0.0.1:${PORT_B}/spend/logs?request_id=287270b0-0fc4-48c3-9cc9-c937ac2439f8" -H "Authorization: Bearer sk-1234"Observed:
B5 standard auth, Google token alone
curl -sS -i "$URL_B" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
On standard auth the token is rejected by auth itself, so it never reaches the passthrough
C1 proxy credential, master key
curl -sS -i "$URL_C" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
C2 proxy credential, caller token
curl -sS -i "$URL_C" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
D1 strict custom auth, its key alone
curl -sS -i "$URL_D" -H "Authorization: Bearer sk-1234-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
D2 strict custom auth, its key plus caller token
curl -sS -i "$URL_D" -H "x-litellm-api-key: sk-1234-1234" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
curl -s "http://127.0.0.1:${PORT_D}/spend/logs?request_id=5219425c-ea1c-498e-9324-a10b4507bf0d" -H "Authorization: Bearer sk-1234-1234"Observed:
D3 strict custom auth, fake x-goog-api-key
curl -sS -i "$URL_D" -H "Authorization: Bearer sk-1234-1234" -H "x-goog-api-key: not-a-real-google-key" -H "Content-Type: application/json" -d "$BODY"Observed:
A Google error: the bogus
x-goog-api-keywas forwarded and Google itself rejected itE1 no master key, caller's own Google token
curl -sS -i "$URL_E" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
The same regression on a proxy with no master key at all
E2 no master key, sk-shaped value alone
curl -sS -i "$URL_E" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
After (a3f6547)
A1 custom auth, caller's own Google token
curl -sS -i "$URL_A" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
The caller's own token now reaches Google: the regression is fixed
A2 custom auth, caller's own token, SSE stream
curl -sS -i -m 90 "http://127.0.0.1:${PORT_A}/vertex_ai/v1/projects/vertex-check-481318/locations/global/publishers/google/models/gemini-3.5-flash-lite:streamGenerateContent?alt=sse" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
A3 spend log for A1
curl -s "http://127.0.0.1:${PORT_A}/spend/logs?request_id=ffb0379b-501c-4262-88ca-53bfcba9466f" -H "Authorization: Bearer sk-1234"(found on first poll)Observed:
A4 custom auth, master key alone
curl -sS -i "$URL_A" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
The master key is still stripped: the fix(passthrough): stop leaking the caller's virtual key on credential-less Vertex passthrough #38114 leak protection holds
A5 custom auth, virtual key alone
curl -sS -i "$URL_A" -H "Authorization: Bearer $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
Under custom auth this key did not authenticate the request, so it is forwarded and Google rejects it: the pre-fix(passthrough): stop leaking the caller's virtual key on credential-less Vertex passthrough #38114 behavior (see Caveats)
A6 custom auth, virtual key header plus caller token
curl -sS -i "$URL_A" -H "x-litellm-api-key: $VKEY" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
B1 standard auth, virtual key alone
curl -sS -i "$URL_B" -H "Authorization: Bearer $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
Here the virtual key did authenticate, so it is stripped and never forwarded
B2 standard auth, virtual key in x-goog-api-key
curl -sS -i "$URL_B" -H "x-goog-api-key: $VKEY" -H "Content-Type: application/json" -d "$BODY"Observed:
B3 standard auth, master key alone
curl -sS -i "$URL_B" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
B4 standard auth, virtual key header plus caller token
curl -sS -i "$URL_B" -H "x-litellm-api-key: $VKEY" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
curl -s "http://127.0.0.1:${PORT_B}/spend/logs?request_id=87703f23-8e11-41a0-b5df-9e27d2de3705" -H "Authorization: Bearer sk-1234"Observed:
B5 standard auth, Google token alone
curl -sS -i "$URL_B" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
C1 proxy credential, master key
curl -sS -i "$URL_C" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
Master key stripped, the proxy's own credential used
C2 proxy credential, caller token
curl -sS -i "$URL_C" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
D1 strict custom auth, its key alone
curl -sS -i "$URL_D" -H "Authorization: Bearer sk-1234-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
The sk-shaped custom-auth key that authenticated is stripped, never forwarded
D2 strict custom auth, its key plus caller token
curl -sS -i "$URL_D" -H "x-litellm-api-key: sk-1234-1234" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
curl -s "http://127.0.0.1:${PORT_D}/spend/logs?request_id=75a4eaf2-a9ea-4dcb-8b8c-6dc15b3229d2" -H "Authorization: Bearer sk-1234-1234"Observed:
D3 strict custom auth, fake x-goog-api-key
curl -sS -i "$URL_D" -H "Authorization: Bearer sk-1234-1234" -H "x-goog-api-key: not-a-real-google-key" -H "Content-Type: application/json" -d "$BODY"Observed:
E1 no master key, caller's own Google token
curl -sS -i "$URL_E" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
The regression is fixed on the no-master-key branch too
E2 no master key, sk-shaped value alone
curl -sS -i "$URL_E" -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d "$BODY"Observed:
sk-shaped secrets stay stripped even with no master key configured
F1 JWT auth, the authenticating LiteLLM JWT alone
curl -sS -i "$URL_F" -H "Authorization: Bearer $LJWT" -H "Content-Type: application/json" -d "$BODY"Observed:
JWT auth accepted the request, then the JWT that authenticated was stripped rather than forwarded to Google
F2 JWT auth, LiteLLM JWT in header plus caller token
curl -sS -i "$URL_F" -H "x-litellm-api-key: $LJWT" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d "$BODY"Observed:
The JWT-authed caller's own Google token still flows through
Observations from the run, all pre-existing and left alone by this PR:
Type
🐛 Bug Fix
Caveats (if any)
Low
Final Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
be0cac1 passes /live-pr-risk
a3f6547 passes /live-pr-risk
Note
Medium Risk
Changes auth-secret stripping on the Vertex BYO-credential path—a security-sensitive area where a mistake could leak keys upstream or break legitimate Google tokens; scope is limited to credential-less passthrough with expanded unit coverage.
Overview
Fixes a #38114 regression where credential-less Vertex passthrough dropped
Authorization(or other credential headers) using header precedence, so valid bring-your-own Google tokens were treated as LiteLLM keys and requests 401’d under custom auth or JWT auth.Credential-less forwarding now removes headers by what actually authenticated (
user_api_key_dict), not by which header won precedence: master key (hmac), JWT matching storedjwt_claims, or virtual keys compared via the same hash auth uses forapi_key. Proxy-only key headers are still dropped by name;sk-*values stay stripped when no master key is configured._prepare_vertex_auth_headerstakesuser_api_key_dictand passes it into_forwarded_headers_for_credentialless_vertex_passthrough. Tests add JWT, master-key, and custom-auth regression cases; a guardrails test drops a fakeproxy_servermodule shim.Reviewed by Cursor Bugbot for commit a3f6547. Bugbot is set up for automated code reviews on this repo. Configure here.