Skip to content

fix(proxy): downgrade routine auth failures from ERROR+traceback to WARN - #26

Merged
songkuan-zheng merged 1 commit into
ship/v1.83.10from
fix/proxy-auth-log-level-down
May 28, 2026
Merged

fix(proxy): downgrade routine auth failures from ERROR+traceback to WARN#26
songkuan-zheng merged 1 commit into
ship/v1.83.10from
fix/proxy-auth-log-level-down

Conversation

@songkuan-zheng

Copy link
Copy Markdown
Collaborator

Summary

verbose_proxy_logger.exception() in auth_exception_handler.py:78 unconditionally logged every auth failure at ERROR with a full Python traceback. Routine 401/403 outcomes (missing key, expired key, invalid key, role mismatch, budget exceeded) hit this path constantly from probes, scanners, and fat-fingered keys — flooding the error stream with noise and burying real bugs.

The same log line was also lossy in three ways:

  1. str(e) is often empty for ProxyException, so the formatted message collapsed to a bare exception-type name with no signal why the request was rejected.
  2. IP was double-encoded — written into the format string AND into extra={...}. Structured log consumers saw it twice.
  3. No correlation idx-litellm-call-id exists per-request but was never carried into the log entry. Hard to grep "the log line for that 401 the user reported".

Change

litellm/proxy/auth/auth_exception_handler.py:

  • Demote known auth-failure types (ProxyException, HTTPException) to WARNING without exc_info.
  • Keep ERROR + full traceback for unexpected exceptions so real bugs stay visible.
  • Fall back to type(e).__name__ when str(e) is empty.
  • Drop IP from the format string; keep it in extra only.
  • Add request_id, route, exception_type, http_status to extra for structured-log aggregation.

Before / After

Before (every 401 emitted ~10 lines):

ERROR auth_exception_handler.py:78 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured -
Requester IP Address:172.20.0.1
Traceback (most recent call last):
  ...10 lines of stack...
litellm.proxy._types.ProxyException
INFO: 172.20.0.1:59948 - "GET /model/cost_map/source HTTP/1.1" 401 Unauthorized

After:

WARNING auth_exception_handler.py:100 - user_api_key_auth failed: Authentication Error - Expired Key
INFO: 172.20.0.1:59948 - "GET /model/cost_map/source HTTP/1.1" 401 Unauthorized

Structured extras (requester_ip, request_id, route, exception_type, http_status) attached to the WARNING record for log aggregators.

Test plan

  • python3 -m pytest tests/test_litellm/proxy/auth/test_auth_exception_handler.py -v → 15 passed
  • New tests (3):
    • test_known_auth_failure_logs_at_warning_without_traceback — ProxyException → WARN level, no traceback, extras populated (request_id, route, exception_type=ProxyException, http_status=401)
    • test_unknown_exception_logs_at_error_with_traceback — ValueError → ERROR + traceback (preserves debuggability of real bugs)
    • test_empty_exception_message_falls_back_to_type_name — ProxyException(message="") → log message contains "ProxyException" (no signal loss)
  • Existing 12 tests still pass.

Followup

PR for the frontend half (status-aware redirect on 401) coming next — that one fixes the user-visible side: dashboard currently only redirects to login on string-match of "Expired Key", missing every other auth-failure variant. Together they make auth failures both quiet in logs and correctly handled in UI.

verbose_proxy_logger.exception() unconditionally logged every auth
failure at ERROR with a full traceback. Routine 401/403 outcomes (no
key, expired key, invalid key, role mismatch, budget exceeded) hit
this path constantly from probes / scanners / fat-fingered keys and
buried genuine errors under noise. The log line itself also lost
signal in two ways:

1. When ProxyException.message was empty, str(e) was empty, so the
   formatted message collapsed to a bare exception type with no hint
   why the request was rejected.
2. The requester IP was double-encoded (string concat into message +
   extra dict), bloating structured-log size.
3. No correlation id — x-litellm-call-id existed but wasn't carried
   into the log, making it hard to grep "the log entry for that one
   401 the user reported".

Fix:
- Demote known auth-failure exception types (ProxyException,
  HTTPException) to WARNING without exc_info.
- Keep ERROR + full traceback for truly unexpected exceptions so real
  bugs are still visible.
- Fall back to type(e).__name__ when str(e) is empty so there's
  always *something* to grep on.
- Drop the IP from the format string (kept in extra only).
- Add request_id, route, exception_type, http_status to extra for
  structured-log aggregation.

Test plan:
- python3 -m pytest tests/test_litellm/proxy/auth/test_auth_exception_handler.py -v
  → 15 passed (3 new + 12 existing)
- New cases:
  - test_known_auth_failure_logs_at_warning_without_traceback —
    ProxyException → WARN, no traceback, extras populated
  - test_unknown_exception_logs_at_error_with_traceback —
    ValueError → ERROR + traceback
  - test_empty_exception_message_falls_back_to_type_name —
    ProxyException(message="") → log message contains "ProxyException"
@songkuan-zheng
songkuan-zheng merged commit 71ac81f into ship/v1.83.10 May 28, 2026
1 check passed
@songkuan-zheng
songkuan-zheng deleted the fix/proxy-auth-log-level-down branch May 28, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant