fix(proxy): downgrade routine auth failures from ERROR+traceback to WARN - #26
Merged
Merged
Conversation
verbose_proxy_logger.exception() unconditionally logged every auth
failure at ERROR with a full traceback. Routine 401/403 outcomes (no
key, expired key, invalid key, role mismatch, budget exceeded) hit
this path constantly from probes / scanners / fat-fingered keys and
buried genuine errors under noise. The log line itself also lost
signal in two ways:
1. When ProxyException.message was empty, str(e) was empty, so the
formatted message collapsed to a bare exception type with no hint
why the request was rejected.
2. The requester IP was double-encoded (string concat into message +
extra dict), bloating structured-log size.
3. No correlation id — x-litellm-call-id existed but wasn't carried
into the log, making it hard to grep "the log entry for that one
401 the user reported".
Fix:
- Demote known auth-failure exception types (ProxyException,
HTTPException) to WARNING without exc_info.
- Keep ERROR + full traceback for truly unexpected exceptions so real
bugs are still visible.
- Fall back to type(e).__name__ when str(e) is empty so there's
always *something* to grep on.
- Drop the IP from the format string (kept in extra only).
- Add request_id, route, exception_type, http_status to extra for
structured-log aggregation.
Test plan:
- python3 -m pytest tests/test_litellm/proxy/auth/test_auth_exception_handler.py -v
→ 15 passed (3 new + 12 existing)
- New cases:
- test_known_auth_failure_logs_at_warning_without_traceback —
ProxyException → WARN, no traceback, extras populated
- test_unknown_exception_logs_at_error_with_traceback —
ValueError → ERROR + traceback
- test_empty_exception_message_falls_back_to_type_name —
ProxyException(message="") → log message contains "ProxyException"
1 task
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
verbose_proxy_logger.exception()inauth_exception_handler.py:78unconditionally logged every auth failure at ERROR with a full Python traceback. Routine 401/403 outcomes (missing key, expired key, invalid key, role mismatch, budget exceeded) hit this path constantly from probes, scanners, and fat-fingered keys — flooding the error stream with noise and burying real bugs.The same log line was also lossy in three ways:
str(e)is often empty for ProxyException, so the formatted message collapsed to a bare exception-type name with no signal why the request was rejected.extra={...}. Structured log consumers saw it twice.x-litellm-call-idexists per-request but was never carried into the log entry. Hard to grep "the log line for that 401 the user reported".Change
litellm/proxy/auth/auth_exception_handler.py:ProxyException,HTTPException) to WARNING withoutexc_info.type(e).__name__whenstr(e)is empty.extraonly.request_id,route,exception_type,http_statustoextrafor structured-log aggregation.Before / After
Before (every 401 emitted ~10 lines):
After:
Structured extras (
requester_ip,request_id,route,exception_type,http_status) attached to the WARNING record for log aggregators.Test plan
python3 -m pytest tests/test_litellm/proxy/auth/test_auth_exception_handler.py -v→ 15 passedtest_known_auth_failure_logs_at_warning_without_traceback— ProxyException → WARN level, no traceback, extras populated (request_id,route,exception_type=ProxyException,http_status=401)test_unknown_exception_logs_at_error_with_traceback— ValueError → ERROR + traceback (preserves debuggability of real bugs)test_empty_exception_message_falls_back_to_type_name— ProxyException(message="") → log message contains "ProxyException" (no signal loss)Followup
PR for the frontend half (status-aware redirect on 401) coming next — that one fixes the user-visible side: dashboard currently only redirects to login on string-match of "Expired Key", missing every other auth-failure variant. Together they make auth failures both quiet in logs and correctly handled in UI.