test(e2e): register case 24 (Tier=real SKIP) + add case 25 backend error.type contract

e2e/cases/25_backend_auth_error_type_contract.md

@@ -0,0 +1,93 @@
+# Case 25 — Backend `error.type` contract on 401 / 403
+
+## Goal
+
+Lock in the wire contract that Wave 6a's `_classify_auth_failure`
+established: every 401 / 403 response from the proxy carries a
+structured `error.type` value drawn from `ProxyErrorTypes`. The
+frontend SPA (PR #68's `networking.tsx` 401 redirect taxonomy) reads
+this field to decide between "session expired → redirect to login" vs
+"permission denied → inline error". If the backend silently drops the
+field (e.g. a future upstream refactor consolidates auth-error paths
+through a code path that doesn't call `_classify_auth_failure`), the
+UI falls back to substring heuristics and the redirect decisions
+silently drift.
+
+## Background
+
+Wave 6a (PR #56) added two pieces to
+`litellm/proxy/auth/auth_exception_handler.py`:
+
+- `_classify_auth_failure(e: Exception) -> ProxyErrorTypes` — inspects
+  status code + message text and returns one of the four structured
+  values listed below.
+- `UserAPIKeyAuthExceptionHandler._handle_authentication_error` is the
+  call site that wraps the classifier into `ProxyException.type`.
+
+Wave 7 + PR #68 made the frontend honor these:
+
+| `error.type` value | UI action |
+|---|---|
+| `auth_session_expired` | clear cookies, redirect to `/login` |
+| `auth_invalid_credentials` | clear cookies, redirect to `/login` |
+| `token_not_found_in_db` | clear cookies, redirect to `/login` |
+| `auth_permission_denied` | inline toast, stay on page |
+| `auth_error` (fallback) | substring heuristic on the `message` field |
+
+Without this case, neither side of that contract is locked end-to-end.
+
+## What the fixture does
+
+`e2e/cases/data/25_backend_auth_error_type_contract.sh` runs four
+probes against the running proxy and asserts each `error.type` value:
+
+| Probe | Request | Expected `error.type` |
+|---|---|---|
+| A1 | `POST /v1/chat/completions` with no `Authorization` header | `auth_invalid_credentials` |
+| A2 | `POST /v1/chat/completions` with `Authorization: Bearer notavalidkey` (no `sk-` prefix) | `auth_invalid_credentials` |
+| A3 | `POST /v1/chat/completions` with `Authorization: Bearer sk-doesnotexist-case25` (well-formed but absent from `LiteLLM_VerificationTokenTable`) | `token_not_found_in_db` |
+| A4 | Provision an `internal_user`-role virtual key via the master key, then `POST /key/generate` with that key (admin-only route) | `auth_permission_denied` |
+
+`auth_session_expired` is intentionally not exercised here — driving it
+deterministically requires a `duration: "1s"` key + a 2-second sleep,
+which is fragile under load. The classifier's "expired" / "revoked" /
+"key has been deleted" / "key has expired" markers are covered by
+`tests/test_litellm/proxy/auth/test_auth_exception_handler.py` instead.
+
+All four probes return HTTP 401 (not 403, even for permission denial —
+that's an upstream quirk of how `auth_pipeline_failure` is raised).
+The case asserts on `error.type`, not on the status code.
+
+## Steps
+
+```bash
+e2e/tools/proxy start --with-mock     # if not already running
+e2e/tools/run-all-cases --mock-only   # case 25 is Tier=mock
+# Or run case 25 alone:
+bash e2e/cases/data/25_backend_auth_error_type_contract.sh
+```
+
+## Expected outcome
+
+```
+PASS: A1 no-auth-header → error.type=auth_invalid_credentials
+PASS: A2 malformed-key → error.type=auth_invalid_credentials
+PASS: A3 bogus-sk-key → error.type=token_not_found_in_db
+PASS: A4 internal-user-on-admin-route → error.type=auth_permission_denied
+PASS: all 4 auth error.type contract probes hit expected values
+```
+
+## When this case will fail
+
+- A future upstream PR routes some auth-pipeline exceptions around
+  `UserAPIKeyAuthExceptionHandler._handle_authentication_error`, so
+  `_classify_auth_failure` never runs and `error.type` defaults to a
+  raw `auth_error` string. **Action**: re-route the new path through
+  the classifier.
+- The classifier's marker lists in
+  `litellm/proxy/auth/auth_exception_handler.py` are pruned to match a
+  message-text refactor and the four probe messages no longer match.
+  **Action**: add the new markers or update the probes.
+- Upstream removes the `error.type` field from `ProxyException`'s JSON
+  serialization entirely. **Action**: restore the field; the UI's
+  redirect taxonomy depends on it.

e2e/cases/README.md

@@ -38,6 +38,8 @@ verdict. See "Mock-only mode" section below.
 | 21 | `21_anthropic_error_shape.md` | mock | (none — malformed body) | `/v1/messages` 4xx responses use Anthropic envelope (`{type:"error",error:{type,message}}`), no `{"detail":...}` wrapper, no OpenAI-only `param`/`code`; streaming-pre-SSE error path same shape; `/v1/chat/completions` stays OpenAI-shaped (scope guard) | — |
 | 22 | `22_gemini_credential_custom_api_base.md` | mock | Gemini | `gemini/` provider with custom `api_base` (UI PR #24) — exercises `/v1beta/models/<m>:generateContent` against a non-Google host with `x-goog-api-key` | — |
 | 23 | `23_mock_memory_pressure.md` | mock | mock (no real provider) | Memory amplification under streaming + large bodies + retries + slow callbacks. Reproduces the prod 12 GB OOM math (peak Δ +900 MB for 5×40MB concurrent; +1.5 GB with `num_retries:2` + 30% 503). Provider-cost-free | ✓ |
+| 24 | `24_anthropic_beta_overrides_bedrock_gateway.md` | real | Bedrock-Anthropic gateway | `anthropic_beta_overrides` config rewrites `advanced-tool-use-2025-11-20` → `tool-search-tool-2025-10-19` so a Bedrock-backed Anthropic-spec gateway accepts the request. Manual runbook (Tier=real — needs Bedrock); `case_24` in the runner SKIPs to keep `--mock-only` honest | — |
+| 25 | `25_backend_auth_error_type_contract.md` | mock | (none — auth failures) | 401/403 responses carry structured `error.type` (auth_session_expired / auth_invalid_credentials / token_not_found_in_db / auth_permission_denied). Locks the contract that PR #68's UI 401 redirect taxonomy depends on | ✓ |
 
 ## How to invoke
 

e2e/cases/data/25_backend_auth_error_type_contract.sh

@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+# Regression fixture for Case 25 — backend auth error.type contract.
+#
+# Wave 6a added `_classify_auth_failure` to
+# litellm/proxy/auth/auth_exception_handler.py — it emits a structured
+# `error.type` field in 401/403 response bodies so the UI can route on
+# the failure category (PR #68 layered the frontend to read it).
+#
+# Without an e2e lock, a future upstream refactor of the auth pipeline
+# could silently stop emitting these structured types — unit tests on
+# the classifier itself would still pass, but end-to-end the UI would
+# fall back to substring heuristics for every 401/403. This case asserts
+# the wire contract directly so that regression is caught.
+#
+# 4 probes, one per ProxyErrorTypes:
+#   A1 no Authorization header        → auth_invalid_credentials
+#   A2 malformed (no sk- prefix)      → auth_invalid_credentials
+#   A3 sk- key absent from DB         → token_not_found_in_db
+#   A4 non-admin role + admin route   → auth_permission_denied
+#
+# All four currently return HTTP 401 (Bedrock-style 403 is recognized by
+# the classifier but the proxy's auth pipeline raises 401 even for
+# permission denials — that's an upstream quirk we don't try to
+# rationalize). The assertion is on `error.type`, not the status code.
+
+set -eu
+
+PROXY_URL="${PROXY_URL:-http://localhost:4011}"
+MASTER_KEY="${MASTER_KEY:-sk-e2e-test}"
+
+BODY='{"model":"mock-openai","messages":[{"role":"user","content":"hi"}]}'
+
+# Helper: POST $BODY with given Authorization header (or none), echo body.
+probe() {
+    local auth_arg="$1" route="$2"
+    if [ -n "$auth_arg" ]; then
+        curl -sSL -X POST "$PROXY_URL$route" \
+            -H "Authorization: Bearer $auth_arg" \
+            -H "Content-Type: application/json" \
+            -d "$BODY"
+    else
+        curl -sSL -X POST "$PROXY_URL$route" \
+            -H "Content-Type: application/json" \
+            -d "$BODY"
+    fi
+}
+
+assert_error_type() {
+    local label="$1" expected="$2" payload="$3"
+    local actual
+    actual=$(printf '%s' "$payload" | python3 -c "import json,sys; print((json.load(sys.stdin).get('error') or {}).get('type',''))" 2>/dev/null || echo "")
+    if [ "$actual" = "$expected" ]; then
+        echo "PASS: $label → error.type=$actual"
+        return 0
+    else
+        echo "FAIL: $label expected error.type=$expected, got '$actual'. Full body:"
+        printf '%s\n' "$payload" | head -c 500
+        echo ""
+        return 1
+    fi
+}
+
+fails=0
+
+# A1: no Authorization header at all.
+r=$(probe "" /v1/chat/completions)
+assert_error_type "A1 no-auth-header" "auth_invalid_credentials" "$r" || fails=$((fails+1))
+
+# A2: malformed key (doesn't start with sk-).
+r=$(probe "notavalidkey" /v1/chat/completions)
+assert_error_type "A2 malformed-key" "auth_invalid_credentials" "$r" || fails=$((fails+1))
+
+# A3: well-formed sk- key but not in the DB cache or VerificationTokenTable.
+r=$(probe "sk-doesnotexist-case25" /v1/chat/completions)
+assert_error_type "A3 bogus-sk-key" "token_not_found_in_db" "$r" || fails=$((fails+1))
+
+# A4: authenticated user without admin role hits an admin-only route.
+# Provision an internal_user-role key via the master key, then have it
+# call /key/generate (admin-only).
+internal_key=$(curl -sSL -X POST "$PROXY_URL/key/generate" \
+    -H "Authorization: Bearer $MASTER_KEY" \
+    -H "Content-Type: application/json" \
+    -d '{"user_role":"internal_user"}' \
+    | python3 -c "import json,sys; print(json.load(sys.stdin).get('key',''))" 2>/dev/null)
+if [ -z "$internal_key" ]; then
+    echo "FAIL: A4 setup — could not provision internal_user key"
+    fails=$((fails+1))
+else
+    r=$(curl -sSL -X POST "$PROXY_URL/key/generate" \
+        -H "Authorization: Bearer $internal_key" \
+        -H "Content-Type: application/json" \
+        -d '{}')
+    assert_error_type "A4 internal-user-on-admin-route" "auth_permission_denied" "$r" || fails=$((fails+1))
+fi
+
+if [ "$fails" -eq 0 ]; then
+    echo "PASS: all 4 auth error.type contract probes hit expected values"
+    exit 0
+fi
+echo "FAIL: $fails of 4 probes did not match contract"
+exit 1

e2e/tools/run-all-cases

@@ -478,6 +478,33 @@ case_23() {
     fi
 }
 
+# Case 24 is a manual runbook against a real Bedrock-backed Anthropic-spec
+# gateway. The behaviour it verifies (anthropic_beta_overrides rewrite of
+# the auto-injected advanced-tool-use header to the older
+# tool-search-tool flag Bedrock accepts) cannot be driven by the mock
+# provider — Bedrock's beta-flag validation is the whole point of the
+# test. SKIP unconditionally so the e2e summary records its existence
+# without driving it.
+case_24() {
+    skip "24 anthropic-beta-overrides-bedrock-gateway" \
+        "Tier=real — requires Bedrock; runbook is e2e/cases/24_*.md"
+}
+
+# ------------------------------------------------------------------ 25
+case_25() {
+    echo "[25] backend auth error.type contract..."
+    local out=/tmp/e2e_case25.out
+    bash e2e/cases/data/25_backend_auth_error_type_contract.sh > "$out" 2>&1
+    local rc=$?
+    if [ "$rc" -eq 0 ]; then
+        ok "25 auth-error-type-contract: $(grep -m1 '^PASS: all' "$out")"
+    elif [ "$rc" -eq 77 ]; then
+        skip "25 auth-error-type-contract" "$(grep -m1 SKIP "$out")"
+    else
+        fail "25 auth-error-type-contract" "$(grep -m1 '^FAIL' "$out" || tail -2 "$out")"
+    fi
+}
+
 # Pre-flight: proxy must be ready
 if ! curl -sSL --max-time 3 -o /dev/null -w '%{http_code}' \
         "$PROXY/health/readiness" 2>/dev/null | grep -q '^2'; then
@@ -604,6 +631,15 @@ fi
 # call; safe under --skip-paid.
 case_21
 
+# Case 24 always SKIPs (Tier=real, Bedrock-only); recorded so the e2e
+# summary reflects that the runbook exists and hasn't been forgotten.
+case_24
+
+# Case 25 (backend auth error.type contract) is mock-friendly — no
+# provider calls, all curls go to /v1/chat/completions auth-failure
+# paths or /key/generate. Safe outside --mock-only.
+case_25
+
 echo
 echo "============ SUMMARY ============"
 printf '%s\n' "${RESULTS[@]}"