fix(proxy): extend WARN/ERROR exception routing to LLM hot path - #40
Merged
songkuan-zheng merged 1 commit intoJun 3, 2026
Conversation
PR #39 covered management endpoints; this finishes the job for the LLM request path and shared utilities that wrap it. Same policy: known business exceptions (4xx with stable type, upstream 502/503/504) log at WARN with a single structured line; truly unexpected exceptions (bare RuntimeError, KeyError, real 500) keep ERROR + traceback. Scope: - proxy_server.py: 20 route handlers — completion, moderations, audio_transcriptions, realtime websocket (pre-call / upstream-status / internal-error), token_counter, login_v2/v3/exchange, async_data_generator + async_assistants_data_generator streaming wrappers, plus 9 reload/schedule/cancel/status admin endpoints for model_cost_map and anthropic_beta_headers. Deliberately NOT migrated: 25 startup / background-health-check / config-reload / model-list-filter helpers. Those run outside the request path, and failures there ARE the bug — they should keep ERROR + traceback. - common_request_processing.py: stream first-chunk consumer, deferred-logging hooks, post-call streaming guardrail, orphaned streaming logging, async_data_generator wrapper, and the central _handle_llm_api_exception classifier. - route_llm_request.py: shared aiohttp session recreation fallback — failure is non-fatal (we fall back to None and continue), shouldn't flood ERROR. - litellm_pre_call_utils.py: api-version query-param parse. - utils.py: post_call_failure_hook callback failures (both inner and setup), post_call_response_headers_hook, send_email. - auth/user_api_key_auth.py: websocket auth wrapper (was logging via the noisy code path). Also fixes a misuse of verbose_proxy_logger .exception() outside an except block (custom_litellm_key_header branch) — there's no current exception, so demoted to a single WARNING line that no longer prints "(NoneType: None)". - auth/auth_checks.py: get_team_membership, access_group lookup, team alias lookup, organization alias lookup. All four were ERROR + traceback for routine 404 outcomes. - common_utils/http_parsing_utils.py: invalid-JSON 400 was logged at ERROR (no traceback, but still polluted the ERROR stream on every malformed client payload); demoted to WARN. The unexpected-exception catch-all stays ERROR + traceback — that's the path that catches real server faults. E2E verification on this branch: - 7 distinct 4xx scenarios (unknown model / bad provider key / unauthenticated /schedule/* / bogus bearer / malformed JSON body / bogus login) → all WARN single line, 0 traceback. - 50 successive 401s from rotated bogus bearer tokens → 50 WARN, 0 ERROR, 0 traceback. - Pre-existing 79 unit tests in test_auth_exception_handler + test_exception_logging still pass (RuntimeError/ValueError still route to ERROR + traceback).
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Files
E2E verification (this branch built and deployed to local e2e proxy)
7 distinct 4xx scenarios — all WARN single line, 0 traceback:
Stress test — 50 successive 401s from rotated bogus bearer tokens:
```
ERROR=0 WARNING=57 Traceback=0
```
(57 = 7 above + 50 stress.)
Pre-existing 79 tests in `test_auth_exception_handler.py` + `test_exception_logging.py` still pass — `RuntimeError` / `ValueError` / no-status-code exceptions still route to ERROR + traceback.
Test plan