diff --git a/e2e/cases/25_backend_auth_error_type_contract.md b/e2e/cases/25_backend_auth_error_type_contract.md new file mode 100644 index 000000000000..7170cdd30164 --- /dev/null +++ b/e2e/cases/25_backend_auth_error_type_contract.md @@ -0,0 +1,93 @@ +# Case 25 — Backend `error.type` contract on 401 / 403 + +## Goal + +Lock in the wire contract that Wave 6a's `_classify_auth_failure` +established: every 401 / 403 response from the proxy carries a +structured `error.type` value drawn from `ProxyErrorTypes`. The +frontend SPA (PR #68's `networking.tsx` 401 redirect taxonomy) reads +this field to decide between "session expired → redirect to login" vs +"permission denied → inline error". If the backend silently drops the +field (e.g. a future upstream refactor consolidates auth-error paths +through a code path that doesn't call `_classify_auth_failure`), the +UI falls back to substring heuristics and the redirect decisions +silently drift. + +## Background + +Wave 6a (PR #56) added two pieces to +`litellm/proxy/auth/auth_exception_handler.py`: + +- `_classify_auth_failure(e: Exception) -> ProxyErrorTypes` — inspects + status code + message text and returns one of the four structured + values listed below. +- `UserAPIKeyAuthExceptionHandler._handle_authentication_error` is the + call site that wraps the classifier into `ProxyException.type`. + +Wave 7 + PR #68 made the frontend honor these: + +| `error.type` value | UI action | +|---|---| +| `auth_session_expired` | clear cookies, redirect to `/login` | +| `auth_invalid_credentials` | clear cookies, redirect to `/login` | +| `token_not_found_in_db` | clear cookies, redirect to `/login` | +| `auth_permission_denied` | inline toast, stay on page | +| `auth_error` (fallback) | substring heuristic on the `message` field | + +Without this case, neither side of that contract is locked end-to-end. + +## What the fixture does + +`e2e/cases/data/25_backend_auth_error_type_contract.sh` runs four +probes against the running proxy and asserts each `error.type` value: + +| Probe | Request | Expected `error.type` | +|---|---|---| +| A1 | `POST /v1/chat/completions` with no `Authorization` header | `auth_invalid_credentials` | +| A2 | `POST /v1/chat/completions` with `Authorization: Bearer notavalidkey` (no `sk-` prefix) | `auth_invalid_credentials` | +| A3 | `POST /v1/chat/completions` with `Authorization: Bearer sk-doesnotexist-case25` (well-formed but absent from `LiteLLM_VerificationTokenTable`) | `token_not_found_in_db` | +| A4 | Provision an `internal_user`-role virtual key via the master key, then `POST /key/generate` with that key (admin-only route) | `auth_permission_denied` | + +`auth_session_expired` is intentionally not exercised here — driving it +deterministically requires a `duration: "1s"` key + a 2-second sleep, +which is fragile under load. The classifier's "expired" / "revoked" / +"key has been deleted" / "key has expired" markers are covered by +`tests/test_litellm/proxy/auth/test_auth_exception_handler.py` instead. + +All four probes return HTTP 401 (not 403, even for permission denial — +that's an upstream quirk of how `auth_pipeline_failure` is raised). +The case asserts on `error.type`, not on the status code. + +## Steps + +```bash +e2e/tools/proxy start --with-mock # if not already running +e2e/tools/run-all-cases --mock-only # case 25 is Tier=mock +# Or run case 25 alone: +bash e2e/cases/data/25_backend_auth_error_type_contract.sh +``` + +## Expected outcome + +``` +PASS: A1 no-auth-header → error.type=auth_invalid_credentials +PASS: A2 malformed-key → error.type=auth_invalid_credentials +PASS: A3 bogus-sk-key → error.type=token_not_found_in_db +PASS: A4 internal-user-on-admin-route → error.type=auth_permission_denied +PASS: all 4 auth error.type contract probes hit expected values +``` + +## When this case will fail + +- A future upstream PR routes some auth-pipeline exceptions around + `UserAPIKeyAuthExceptionHandler._handle_authentication_error`, so + `_classify_auth_failure` never runs and `error.type` defaults to a + raw `auth_error` string. **Action**: re-route the new path through + the classifier. +- The classifier's marker lists in + `litellm/proxy/auth/auth_exception_handler.py` are pruned to match a + message-text refactor and the four probe messages no longer match. + **Action**: add the new markers or update the probes. +- Upstream removes the `error.type` field from `ProxyException`'s JSON + serialization entirely. **Action**: restore the field; the UI's + redirect taxonomy depends on it. diff --git a/e2e/cases/README.md b/e2e/cases/README.md index c38068fa9929..37437fcbf0ce 100644 --- a/e2e/cases/README.md +++ b/e2e/cases/README.md @@ -38,6 +38,8 @@ verdict. See "Mock-only mode" section below. | 21 | `21_anthropic_error_shape.md` | mock | (none — malformed body) | `/v1/messages` 4xx responses use Anthropic envelope (`{type:"error",error:{type,message}}`), no `{"detail":...}` wrapper, no OpenAI-only `param`/`code`; streaming-pre-SSE error path same shape; `/v1/chat/completions` stays OpenAI-shaped (scope guard) | — | | 22 | `22_gemini_credential_custom_api_base.md` | mock | Gemini | `gemini/` provider with custom `api_base` (UI PR #24) — exercises `/v1beta/models/:generateContent` against a non-Google host with `x-goog-api-key` | — | | 23 | `23_mock_memory_pressure.md` | mock | mock (no real provider) | Memory amplification under streaming + large bodies + retries + slow callbacks. Reproduces the prod 12 GB OOM math (peak Δ +900 MB for 5×40MB concurrent; +1.5 GB with `num_retries:2` + 30% 503). Provider-cost-free | ✓ | +| 24 | `24_anthropic_beta_overrides_bedrock_gateway.md` | real | Bedrock-Anthropic gateway | `anthropic_beta_overrides` config rewrites `advanced-tool-use-2025-11-20` → `tool-search-tool-2025-10-19` so a Bedrock-backed Anthropic-spec gateway accepts the request. Manual runbook (Tier=real — needs Bedrock); `case_24` in the runner SKIPs to keep `--mock-only` honest | — | +| 25 | `25_backend_auth_error_type_contract.md` | mock | (none — auth failures) | 401/403 responses carry structured `error.type` (auth_session_expired / auth_invalid_credentials / token_not_found_in_db / auth_permission_denied). Locks the contract that PR #68's UI 401 redirect taxonomy depends on | ✓ | ## How to invoke diff --git a/e2e/cases/data/25_backend_auth_error_type_contract.sh b/e2e/cases/data/25_backend_auth_error_type_contract.sh new file mode 100755 index 000000000000..dcf6ab6405bf --- /dev/null +++ b/e2e/cases/data/25_backend_auth_error_type_contract.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Regression fixture for Case 25 — backend auth error.type contract. +# +# Wave 6a added `_classify_auth_failure` to +# litellm/proxy/auth/auth_exception_handler.py — it emits a structured +# `error.type` field in 401/403 response bodies so the UI can route on +# the failure category (PR #68 layered the frontend to read it). +# +# Without an e2e lock, a future upstream refactor of the auth pipeline +# could silently stop emitting these structured types — unit tests on +# the classifier itself would still pass, but end-to-end the UI would +# fall back to substring heuristics for every 401/403. This case asserts +# the wire contract directly so that regression is caught. +# +# 4 probes, one per ProxyErrorTypes: +# A1 no Authorization header → auth_invalid_credentials +# A2 malformed (no sk- prefix) → auth_invalid_credentials +# A3 sk- key absent from DB → token_not_found_in_db +# A4 non-admin role + admin route → auth_permission_denied +# +# All four currently return HTTP 401 (Bedrock-style 403 is recognized by +# the classifier but the proxy's auth pipeline raises 401 even for +# permission denials — that's an upstream quirk we don't try to +# rationalize). The assertion is on `error.type`, not the status code. + +set -eu + +PROXY_URL="${PROXY_URL:-http://localhost:4011}" +MASTER_KEY="${MASTER_KEY:-sk-e2e-test}" + +BODY='{"model":"mock-openai","messages":[{"role":"user","content":"hi"}]}' + +# Helper: POST $BODY with given Authorization header (or none), echo body. +probe() { + local auth_arg="$1" route="$2" + if [ -n "$auth_arg" ]; then + curl -sSL -X POST "$PROXY_URL$route" \ + -H "Authorization: Bearer $auth_arg" \ + -H "Content-Type: application/json" \ + -d "$BODY" + else + curl -sSL -X POST "$PROXY_URL$route" \ + -H "Content-Type: application/json" \ + -d "$BODY" + fi +} + +assert_error_type() { + local label="$1" expected="$2" payload="$3" + local actual + actual=$(printf '%s' "$payload" | python3 -c "import json,sys; print((json.load(sys.stdin).get('error') or {}).get('type',''))" 2>/dev/null || echo "") + if [ "$actual" = "$expected" ]; then + echo "PASS: $label → error.type=$actual" + return 0 + else + echo "FAIL: $label expected error.type=$expected, got '$actual'. Full body:" + printf '%s\n' "$payload" | head -c 500 + echo "" + return 1 + fi +} + +fails=0 + +# A1: no Authorization header at all. +r=$(probe "" /v1/chat/completions) +assert_error_type "A1 no-auth-header" "auth_invalid_credentials" "$r" || fails=$((fails+1)) + +# A2: malformed key (doesn't start with sk-). +r=$(probe "notavalidkey" /v1/chat/completions) +assert_error_type "A2 malformed-key" "auth_invalid_credentials" "$r" || fails=$((fails+1)) + +# A3: well-formed sk- key but not in the DB cache or VerificationTokenTable. +r=$(probe "sk-doesnotexist-case25" /v1/chat/completions) +assert_error_type "A3 bogus-sk-key" "token_not_found_in_db" "$r" || fails=$((fails+1)) + +# A4: authenticated user without admin role hits an admin-only route. +# Provision an internal_user-role key via the master key, then have it +# call /key/generate (admin-only). +internal_key=$(curl -sSL -X POST "$PROXY_URL/key/generate" \ + -H "Authorization: Bearer $MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"user_role":"internal_user"}' \ + | python3 -c "import json,sys; print(json.load(sys.stdin).get('key',''))" 2>/dev/null) +if [ -z "$internal_key" ]; then + echo "FAIL: A4 setup — could not provision internal_user key" + fails=$((fails+1)) +else + r=$(curl -sSL -X POST "$PROXY_URL/key/generate" \ + -H "Authorization: Bearer $internal_key" \ + -H "Content-Type: application/json" \ + -d '{}') + assert_error_type "A4 internal-user-on-admin-route" "auth_permission_denied" "$r" || fails=$((fails+1)) +fi + +if [ "$fails" -eq 0 ]; then + echo "PASS: all 4 auth error.type contract probes hit expected values" + exit 0 +fi +echo "FAIL: $fails of 4 probes did not match contract" +exit 1 diff --git a/e2e/tools/run-all-cases b/e2e/tools/run-all-cases index ce0d5e4597be..94228cb7929f 100755 --- a/e2e/tools/run-all-cases +++ b/e2e/tools/run-all-cases @@ -478,6 +478,33 @@ case_23() { fi } +# Case 24 is a manual runbook against a real Bedrock-backed Anthropic-spec +# gateway. The behaviour it verifies (anthropic_beta_overrides rewrite of +# the auto-injected advanced-tool-use header to the older +# tool-search-tool flag Bedrock accepts) cannot be driven by the mock +# provider — Bedrock's beta-flag validation is the whole point of the +# test. SKIP unconditionally so the e2e summary records its existence +# without driving it. +case_24() { + skip "24 anthropic-beta-overrides-bedrock-gateway" \ + "Tier=real — requires Bedrock; runbook is e2e/cases/24_*.md" +} + +# ------------------------------------------------------------------ 25 +case_25() { + echo "[25] backend auth error.type contract..." + local out=/tmp/e2e_case25.out + bash e2e/cases/data/25_backend_auth_error_type_contract.sh > "$out" 2>&1 + local rc=$? + if [ "$rc" -eq 0 ]; then + ok "25 auth-error-type-contract: $(grep -m1 '^PASS: all' "$out")" + elif [ "$rc" -eq 77 ]; then + skip "25 auth-error-type-contract" "$(grep -m1 SKIP "$out")" + else + fail "25 auth-error-type-contract" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")" + fi +} + # Pre-flight: proxy must be ready if ! curl -sSL --max-time 3 -o /dev/null -w '%{http_code}' \ "$PROXY/health/readiness" 2>/dev/null | grep -q '^2'; then @@ -604,6 +631,15 @@ fi # call; safe under --skip-paid. case_21 +# Case 24 always SKIPs (Tier=real, Bedrock-only); recorded so the e2e +# summary reflects that the runbook exists and hasn't been forgotten. +case_24 + +# Case 25 (backend auth error.type contract) is mock-friendly — no +# provider calls, all curls go to /v1/chat/completions auth-failure +# paths or /key/generate. Safe outside --mock-only. +case_25 + echo echo "============ SUMMARY ============" printf '%s\n' "${RESULTS[@]}"