Skip to content

fix(proxy): treat omitted auth on config pass-through routes as enforced at registration (rc/1.105.0 backport of #44253) - #44258

Closed
yuneng-berri wants to merge 3 commits into
rc/1.105.0from
litellm_fix_6d1634
Closed

yuneng-berri wants to merge 3 commits into
rc/1.105.0from
litellm_fix_6d1634

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

How it solves it:

Depends on #44253 landing on main first

User Flow

Before: an admin's config pass-through route with no auth key hides upstream failures from spend logs

  1. The admin adds /audit-pt to general_settings.pass_through_endpoints with include_subpath: true and no auth
  2. A developer sends POST https://litellm-domain/audit-pt with a valid virtual key
  3. The upstream rejects it and the developer gets back a 403 with the upstream error body
  4. The admin queries GET https://litellm-domain/spend/logs and finds no row for that request

After: the same failed request shows up in spend logs as a failure row

  1. Same config, no auth key
  2. Same POST https://litellm-domain/audit-pt with a valid virtual key
  3. Same 403 with the upstream error body
  4. GET https://litellm-domain/spend/logs now returns a failure row for that request id with error code 403

Relevant issues

Backport of #44253 to rc/1.105.0. Regression from #43962

Affected release

Regression in v1.105.0-dev.2

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy with a real Postgres and a local upstream that always answers 403 {"error": {"message": "max budget reached for this deployment"}}, same config on both sides. MK is the master key and KEY the virtual key from step 1. The After tip 84b70d58f5 only adds a test formatting commit on top of eeb88475c3, so the production code is identical

model_list: []
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 1
  pass_through_endpoints:
    - path: /audit-pt
      target: http://127.0.0.1:8291/upstream
      include_subpath: true

The regression unit test test_pass_through_registration_auth_enforcement fails its two config-omitted cases on rc/1.105.0 and passes with this PR

Before (bb266ad)

  1. Create a virtual key
$ curl -s -X POST localhost:4731/key/generate -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' -d '{"key_alias": "audit-proof"}' | jq -r .key
sk-...U3tQ
  1. Call the config route with that key, the upstream answers 403
$ curl -s -i -X POST localhost:4731/audit-pt -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"contents":[{"parts":[{"text":"hi"}]}]}' | grep -iE '^HTTP|^x-litellm-call-id|^\{'
HTTP/1.1 403 Forbidden
x-litellm-call-id: e9d652ec-3a4d-4eca-a3ef-d97dea136b6a
{"error": {"message": "max budget reached for this deployment"}}
  1. Read spend logs for pass-through rows
$ curl -s 'localhost:4731/spend/logs?start_date=2026-01-01&end_date=2027-01-01&summarize=false' -H "Authorization: Bearer $MK" | jq '[.[] | select(.call_type=="pass_through_endpoint") | {request_id, status, call_type, api_key_alias: .metadata.user_api_key_alias, error_code: .metadata.error_information.error_code}]'
[]

After (eeb8847)

  1. Create a virtual key
$ curl -s -X POST localhost:4732/key/generate -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' -d '{"key_alias": "audit-proof"}' | jq -r .key
sk-...mDiQ
  1. Call the config route with that key, the upstream answers 403
$ curl -s -i -X POST localhost:4732/audit-pt -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"contents":[{"parts":[{"text":"hi"}]}]}' | grep -iE '^HTTP|^x-litellm-call-id|^\{'
HTTP/1.1 403 Forbidden
x-litellm-call-id: 76a9a939-c21c-4b96-8bcc-85a86483582e
{"error": {"message": "max budget reached for this deployment"}}
  1. Read spend logs for pass-through rows
$ curl -s 'localhost:4732/spend/logs?start_date=2026-01-01&end_date=2027-01-01&summarize=false' -H "Authorization: Bearer $MK" | jq '[.[] | select(.call_type=="pass_through_endpoint") | {request_id, status, call_type, api_key_alias: .metadata.user_api_key_alias, error_code: .metadata.error_information.error_code}]'
[
  {
    "request_id": "76a9a939-c21c-4b96-8bcc-85a86483582e",
    "status": "failure",
    "call_type": "pass_through_endpoint",
    "api_key_alias": "audit-proof",
    "error_code": "403"
  }
]

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Severe

  • Config routes without auth get the same key checks as auth: true again
  • Proxies without a database now enforce this too, which they never did
    • Set auth: false on the entry to keep the old behavior

Low

  • The integration node test_config_pass_through_route_logs_body_and_strips_query was not run on this line, the live run above covers the same path

@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical risk] Changes default authentication enforcement for proxy pass-through endpoints.

The PR appears safe to merge, with no actionable issues found

What we checked:

  • Tests clean up registered routes: Each case removes its registered routes in finally. The removal helper clears exact paths, wildcard paths, and matching registry entries

Summary

Config pass-through routes now default to enforced auth during registration when auth is omitted. This matches the existing request-time default and lets upstream failures reach the spend-log check

  • Tests cover exact paths and subpaths for config and database entries, including explicit auth: false
  • No actionable issues found
  • yuneng-berri explicitly acknowledged stricter checks for restricted and over-budget keys, plus newly enforced auth on proxies without a database. These changes are intentional; auth: false preserves unauthenticated forwarding

Reviews (1) · Last reviewed commit: "test(proxy): wrap long pass-through auth..."

@yuneng-berri

Copy link
Copy Markdown
Contributor Author

Closing with #44253: no product change needed, the red test is fixed test-side in #44265 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant