Skip to content

fix(proxy): treat omitted auth on config pass-through routes as enforced at registration (stable/1.103.x backport of #44253) - #44256

Closed
yuneng-berri wants to merge 3 commits into
stable/1.103.xfrom
litellm_fix_c3c2bd
Closed

yuneng-berri wants to merge 3 commits into
stable/1.103.xfrom
litellm_fix_c3c2bd

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

How it solves it:

Depends on #44253 landing on main first

User Flow

Before: an admin's config pass-through route with no auth key hides upstream failures from spend logs

  1. The admin adds /audit-pt to general_settings.pass_through_endpoints with include_subpath: true and no auth
  2. A developer sends POST https://litellm-domain/audit-pt with a valid virtual key
  3. The upstream rejects it and the developer gets back a 403 with the upstream error body
  4. The admin queries GET https://litellm-domain/spend/logs and finds no row for that request

After: the same failed request shows up in spend logs as a failure row

  1. Same config, no auth key
  2. Same POST https://litellm-domain/audit-pt with a valid virtual key
  3. Same 403 with the upstream error body
  4. GET https://litellm-domain/spend/logs now returns a failure row for that request id with error code 403

Relevant issues

Backport of #44253 to stable/1.103.x. Regression from #43962

Affected release

Regression in v1.103.2, which carries the #43962 backport (#43984)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy with a real Postgres and a local upstream that always answers 403 {"error": {"message": "max budget reached for this deployment"}}, same config on both sides. MK is the master key and KEY the virtual key from step 1. The After tip 74896fd5c6 only adds a test formatting commit on top of 653910825c, so the production code is identical

model_list: []
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 1
  pass_through_endpoints:
    - path: /audit-pt
      target: http://127.0.0.1:8291/upstream
      include_subpath: true

The regression unit test test_pass_through_registration_auth_enforcement fails its two config-omitted cases on stable/1.103.x and passes with this PR

Before (3e32c0c)

  1. Create a virtual key
$ curl -s -X POST localhost:4711/key/generate -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' -d '{"key_alias": "audit-proof"}' | jq -r .key
sk-...K1Vg
  1. Call the config route with that key, the upstream answers 403
$ curl -s -i -X POST localhost:4711/audit-pt -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"contents":[{"parts":[{"text":"hi"}]}]}' | grep -iE '^HTTP|^x-litellm-call-id|^\{'
HTTP/1.1 403 Forbidden
x-litellm-call-id: f0ee668c-e08e-4a6e-bc7d-5b11737c35d0
{"error": {"message": "max budget reached for this deployment"}}
  1. Read spend logs for pass-through rows
$ curl -s 'localhost:4711/spend/logs?start_date=2026-01-01&end_date=2027-01-01&summarize=false' -H "Authorization: Bearer $MK" | jq '[.[] | select(.call_type=="pass_through_endpoint") | {request_id, status, call_type, api_key_alias: .metadata.user_api_key_alias, error_code: .metadata.error_information.error_code}]'
[]

After (6539108)

  1. Create a virtual key
$ curl -s -X POST localhost:4712/key/generate -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' -d '{"key_alias": "audit-proof"}' | jq -r .key
sk-...wPkQ
  1. Call the config route with that key, the upstream answers 403
$ curl -s -i -X POST localhost:4712/audit-pt -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"contents":[{"parts":[{"text":"hi"}]}]}' | grep -iE '^HTTP|^x-litellm-call-id|^\{'
HTTP/1.1 403 Forbidden
x-litellm-call-id: f1162b40-6276-443e-8640-1509c4b93fd3
{"error": {"message": "max budget reached for this deployment"}}
  1. Read spend logs for pass-through rows
$ curl -s 'localhost:4712/spend/logs?start_date=2026-01-01&end_date=2027-01-01&summarize=false' -H "Authorization: Bearer $MK" | jq '[.[] | select(.call_type=="pass_through_endpoint") | {request_id, status, call_type, api_key_alias: .metadata.user_api_key_alias, error_code: .metadata.error_information.error_code}]'
[
  {
    "request_id": "f1162b40-6276-443e-8640-1509c4b93fd3",
    "status": "failure",
    "call_type": "pass_through_endpoint",
    "api_key_alias": "audit-proof",
    "error_code": "403"
  }
]

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Severe

  • Config routes without auth get the same key checks as auth: true again
  • Proxies without a database now enforce this too, which they never did
    • Set auth: false on the entry to keep the old behavior

Low

@yuneng-berri
yuneng-berri requested a review from a team October 2, 2026 22:28
@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical risk] Changes default authentication enforcement for proxy pass-through routes.

The PR appears safe to merge, with no actionable issues found

What we checked:

  • Explicitly public routes stay public: The default applies only when the key is missing. Explicit False still skips registration-level enforcement, and the existing request-time check returns an empty UserAPIKeyAuth

Summary

Config pass-through routes with omitted auth now register as authenticated routes. This matches the existing request-time default and lets upstream failures reach spend logging

  • Adds eight regression cases covering config and database endpoints, exact paths, subpaths, and explicit auth: false
  • No actionable issues found
  • yuneng-berri explicitly acknowledged the changed key restrictions and enforcement on proxies without a database as intentional. The stated opt-out is auth: false

Reviews (1) · Last reviewed commit: "test(proxy): wrap long pass-through auth..."

@yuneng-berri

Copy link
Copy Markdown
Contributor Author

Closing with #44253: no product change needed, the red test is fixed test-side in #44265 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant