chore(e2e): add Claude-driven real-provider test harness

.gitignore

@@ -3,6 +3,12 @@
 .venv_policy_test
 .env
 .claude
+
+# e2e: rendered config produced at proxy-start (contains derived model
+# names + env-var references; never edit by hand).
+e2e/_config/.litellm.rendered.yaml
+e2e/tools/__pycache__/
+e2e/_config/.*.swp
 .newenv
 newenv/*
 litellm/proxy/myenv/*

CLAUDE.md

@@ -45,6 +45,38 @@ When contributing to the project, use the appropriate templates:
 - Add at least 1 test in `tests/litellm/`
 - Ensure `make test-unit` passes
 
+### Branching strategy (internal fork)
+
+This fork pins to the upstream `v1.83.10-stable` tag and ships internal
+fixes on top of it.
+
+**Branches:**
+
+| Branch | Purpose | Stays clean? |
+|---|---|---|
+| `v1.83.10-stable` (tag) | Immutable upstream pin | yes — never moves |
+| `ship/v1.83.10` | Long-term ship branch — starts at the tag, only advances via merges of internal `fix/*` PRs | yes |
+| `internal/v1.83.10-stable` | Upstream-sync working branch — may collect upstream commits via teammate / CI sync | **no** — can have hundreds of upstream commits |
+| `litellm_internal_staging` | Pure upstream tracker for `BerriAI/litellm` | tracks upstream |
+| `fix/<short-description>` | Per-bug feature branch | yes — merged into `ship/v1.83.10` via PR merge commit |
+
+**PR target:** every internal fix PR **must target `ship/v1.83.10`**, not
+`internal/v1.83.10-stable` (which has 1700+ upstream-sync commits on top
+of the tag) and not `litellm_internal_staging` (pure upstream).
+
+```bash
+# Default new fix branch from the latest ship state
+git checkout -b fix/<name> ship/v1.83.10
+
+# Open PR
+gh pr create --base ship/v1.83.10 --head fix/<name>
+```
+
+**Conflicts:** `ship/v1.83.10` only moves when a `fix/*` PR merges, so it
+stays exactly TAG + (merged fixes). Fixes never have to rebase against
+moving upstream; the upstream-sync churn lives entirely on
+`internal/v1.83.10-stable`.
+
 ## Architecture Overview
 
 LiteLLM is a unified interface for 100+ LLM providers with two main components:

e2e/.env.example

@@ -0,0 +1,35 @@
+# Copy to e2e/.env and fill in real keys. e2e/.env is gitignored.
+#
+# Only the keys you have are required; cases for missing providers will
+# fail fast with a clear "API key missing" error from `e2e/tools/call`.
+
+# ─── credentials ──────────────────────────────────────────────────────
+ANTHROPIC_API_KEY=
+OPENAI_API_KEY=
+
+# ─── provider base URLs (optional) ────────────────────────────────────
+# Gateway, Azure OpenAI, region-pinned host, self-hosted proxy, etc.
+# Leave blank to use each provider's default.
+# Examples:
+#   ANTHROPIC_API_BASE=https://api.anthropic.com
+#   OPENAI_API_BASE=https://api.openai.com/v1
+#   OPENAI_API_BASE=https://your-resource.openai.azure.com/openai/deployments/<dep>
+ANTHROPIC_API_BASE=
+OPENAI_API_BASE=
+
+# ─── upstream models (optional) ───────────────────────────────────────
+# The full litellm model identifier including provider prefix (so you can
+# route to bedrock/vertex/openrouter without code changes).
+# Leave blank to use the defaults shown.
+#
+# Defaults:
+#   MODEL_ANTHROPIC_SONNET=anthropic/claude-3-5-sonnet-20241022
+#   MODEL_ANTHROPIC_HAIKU=anthropic/claude-3-5-haiku-20241022
+#   MODEL_OPENAI=openai/gpt-4o-mini
+MODEL_ANTHROPIC_SONNET=
+MODEL_ANTHROPIC_HAIKU=
+MODEL_OPENAI=
+
+# ─── misc ─────────────────────────────────────────────────────────────
+# Override default proxy port (4011) if you have a conflict.
+# E2E_PROXY_PORT=4011

e2e/README.md

@@ -0,0 +1,169 @@
+# LiteLLM E2E Test Harness — Claude-driven
+
+This directory is a **toolkit + runbook library** for end-to-end testing
+of the litellm proxy against real provider APIs.
+
+**Philosophy**: Claude Code drives the test sessions. Scripts are
+single-purpose Unix tools; runbooks (`cases/*.md`) describe scenarios.
+No pytest, no framework lock-in. Tools also work fine when invoked by
+a human.
+
+```
+e2e/
+├── README.md                   ← you are here
+├── .env.example                ← copy to .env, fill in keys
+├── _config/
+│   └── docker-compose.yml      ← builds litellm from local source + Postgres
+├── tools/
+│   ├── proxy                   ← lifecycle: start | stop | status | logs | rebuild | url
+│   ├── call                    ← issue one chat-completions request, output JSON
+│   ├── metrics                 ← /metrics: snapshot | diff | get
+│   ├── keys                    ← virtual key lifecycle: new | info | delete | hash
+│   └── teams                   ← team lifecycle: new | info | delete
+└── cases/
+    ├── README.md               ← index of test scenarios
+    └── 01..07_*.md             ← runbooks Claude executes
+```
+
+## One-time setup
+
+```bash
+# 1. Provide API keys (and optionally base URLs / model overrides)
+cp e2e/.env.example e2e/.env
+$EDITOR e2e/.env
+```
+
+`e2e/.env` supports the following keys (all optional except API keys for
+providers you intend to exercise):
+
+| Key | Purpose | Default |
+|---|---|---|
+| `ANTHROPIC_API_KEY` | Anthropic credential | — (required for case 01-04) |
+| `OPENAI_API_KEY` | OpenAI credential | — (required for case 05-06) |
+| `ANTHROPIC_API_BASE` | Gateway / region-pinned endpoint | `https://api.anthropic.com` |
+| `OPENAI_API_BASE` | Gateway / Azure / self-hosted | `https://api.openai.com/v1` |
+| `MODEL_ANTHROPIC_SONNET` | Full litellm model id | `anthropic/claude-3-5-sonnet-20241022` |
+| `MODEL_ANTHROPIC_HAIKU` | Full litellm model id | `anthropic/claude-3-5-haiku-20241022` |
+| `MODEL_OPENAI` | Full litellm model id | `openai/gpt-4o-mini` |
+| `E2E_PROXY_PORT` | Host port for the proxy | `4011` |
+
+Each `MODEL_*` value must include the provider prefix
+(`anthropic/`, `openai/`, `bedrock/`, `vertex_ai/`, `openrouter/`, ...).
+This lets you point the same `model_name` slot at a non-default routing
+path without touching code.
+
+```bash
+# 2. Pre-build the image (subsequent starts are instant)
+e2e/tools/proxy rebuild             # ~3-5 min first time
+```
+
+The Python interpreter that runs the tools must have
+`prometheus_client` available. Easiest: use the litellm dev venv
+(`make install-dev` or `uv run python e2e/tools/metrics ...`).
+
+The proxy config is **generated** from `.env` at `proxy start` time and
+written to `e2e/_config/.litellm.rendered.yaml` (gitignored). Edit `.env`
++ rerun `e2e/tools/proxy restart` to pick up changes — never edit the
+rendered file by hand.
+
+### Postgres (always-on, ephemeral)
+
+`proxy start` brings up a Postgres 16 container alongside litellm so DB-backed
+features (virtual keys, teams, spend logs) work out of the box. The DB is
+**ephemeral** — every `proxy stop` (or `restart`) wipes data. This keeps test
+runs reproducible and prevents stale virtual keys from poisoning later cases.
+
+If you need persistence (e.g. to attach a debugger to spend logs), edit
+`e2e/_config/docker-compose.yml` and add a `volumes:` block under the `db`
+service.
+
+## Typical session
+
+```bash
+# 1. Boot proxy
+e2e/tools/proxy start
+
+# 2. Sanity smoke test (no API key needed)
+# → see e2e/cases/07_prometheus_endpoint_smoke.md
+
+# 3. Drive a real test (Claude reads the runbook and executes)
+# → see e2e/cases/01_prometheus_anthropic_creation_5m.md
+
+# 4. When done
+e2e/tools/proxy stop
+```
+
+## How Claude uses this
+
+Tell Claude:
+
+> "Run case 01 against the running proxy and report what you see."
+
+Claude will:
+1. `cat e2e/cases/01_prometheus_anthropic_creation_5m.md`
+2. Execute the Steps via the Bash tool
+3. Compare actual against Expected
+4. Surface diffs, judge pass/fail, debug if needed
+
+Because Claude is the orchestrator, it can:
+- Adapt mid-test (e.g. retry with a longer prompt if `cache_creation=0`)
+- Cross-check provider responses against metric deltas
+- Open a logs tail when something looks off
+- Decide to skip cases that don't apply to your account
+
+## Adding new cases
+
+1. Drop a new markdown file under `cases/` following the existing
+   Goal / Preconditions / Steps / Expected shape.
+2. If the case uses a new metric / endpoint, the existing 3 tools may
+   already cover it. Only add a new tool when the same logic is needed
+   in ≥ 2 cases.
+3. Update `cases/README.md` index.
+
+## Tools reference (cheat sheet)
+
+```bash
+# Proxy
+e2e/tools/proxy start                # boot (idempotent) — brings up db + litellm
+e2e/tools/proxy stop                 # tear down (wipes db)
+e2e/tools/proxy status               # exit 0 if ready
+e2e/tools/proxy logs --tail 100 -f   # follow logs
+e2e/tools/proxy rebuild              # force image rebuild after source change
+e2e/tools/proxy url                  # prints e.g. http://localhost:4011
+
+# Make a call (full response JSON on stdout)
+e2e/tools/call --provider anthropic --cache ephemeral --ttl 5m
+e2e/tools/call --provider anthropic --cache none
+e2e/tools/call --provider openai --prompt-tokens 1800 --seed run42
+e2e/tools/call --provider anthropic --api-key sk-...  # use virtual key
+
+# Metrics
+e2e/tools/metrics snapshot                              # → JSON
+e2e/tools/metrics get litellm_prompt_cache_read_tokens_metric
+e2e/tools/metrics get litellm_prompt_cache_read_tokens_metric \
+    --label api_provider=anthropic
+e2e/tools/metrics diff before.json after.json \
+    --metric litellm_prompt_cache_creation_tokens_metric \
+    --label cache_ttl=5m
+
+# Virtual keys (needs DB)
+e2e/tools/keys new --alias my-key --models claude-sonnet-cache --duration 30m
+e2e/tools/keys hash sk-...           # print sha256 → matches `hashed_api_key` label
+e2e/tools/keys delete --key sk-...
+
+# Teams (needs DB)
+e2e/tools/teams new --alias team-foo --max-budget 10
+e2e/tools/teams delete --team-id <uuid>
+```
+
+## Cost discipline
+
+These tests call real provider APIs. Per-case cost is < $0.01 with
+default prompt sizes, but adds up if you `loop` recklessly. The case
+runbooks are intentionally short — one or two calls each.
+
+## Out of scope
+
+- Load / concurrency testing (see `tests/load_tests/`)
+- Per-virtual-key isolation (needs Postgres; add when needed)
+- CI automation (these cost money; run on-demand only)

e2e/_config/docker-compose.yml

@@ -0,0 +1,84 @@
+# E2E docker-compose. Builds litellm proxy from the local source tree
+# (so unreleased fixes on the current branch are exercised), exposes /metrics
+# on port 4011 to avoid colliding with a developer's regular proxy on 4000,
+# forwards real provider API keys from `e2e/.env`, and runs an ephemeral
+# Postgres so virtual-key / team / spend-log features can be exercised.
+#
+# Postgres is INTENTIONALLY ephemeral — no named volume — so each
+# `e2e/tools/proxy restart` wipes accumulated test data. To persist data
+# across restarts, add a `volumes:` declaration manually (not recommended
+# for test runs: stale virtual keys + spend logs poison later cases).
+#
+# Run via `e2e/tools/proxy {start|stop|status|logs|rebuild}` — not `docker
+# compose` directly — so behavior stays consistent across machines.
+
+services:
+  db:
+    image: postgres:16-alpine
+    container_name: litellm-e2e-db
+    environment:
+      POSTGRES_DB: litellm
+      POSTGRES_USER: litellm
+      POSTGRES_PASSWORD: e2e-test-only-not-a-secret
+    # No `ports:` — Postgres is only reachable inside the compose network.
+    # No `volumes:` — ephemeral by design.
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"]
+      interval: 3s
+      timeout: 2s
+      retries: 10
+      start_period: 5s
+    restart: "no"
+
+  litellm:
+    build:
+      context: ../..              # repo root — picks up local source changes
+      dockerfile: Dockerfile
+    image: litellm-e2e:local
+    container_name: litellm-e2e
+    ports:
+      - "127.0.0.1:4011:4000"
+    environment:
+      LITELLM_MASTER_KEY: "sk-e2e-test"
+      LITELLM_LOG: "INFO"
+      # Match the user's production env: force using the locally-bundled
+      # model_prices JSON instead of fetching the latest from GitHub.
+      LITELLM_LOCAL_MODEL_COST_MAP: "True"
+      # DB wiring — litellm runs `prisma migrate deploy` on startup against
+      # this URL. Compose-network DNS resolves `db` to the postgres container.
+      DATABASE_URL: "postgresql://litellm:e2e-test-only-not-a-secret@db:5432/litellm"
+      STORE_MODEL_IN_DB: "True"
+      # Forwarded from e2e/.env via `env_file`
+      ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY:-}"
+      OPENAI_API_KEY: "${OPENAI_API_KEY:-}"
+      # Optional provider base URLs (gateway / Azure / self-hosted endpoints).
+      # Blank = use provider default (Anthropic / OpenAI fallback chains
+      # treat empty string as "use default", so forwarding "" is safe here.)
+      ANTHROPIC_API_BASE: "${ANTHROPIC_API_BASE:-}"
+      OPENAI_API_BASE: "${OPENAI_API_BASE:-}"
+    env_file:
+      - path: ../.env
+        required: false
+    volumes:
+      # Rendered at proxy-start time by `e2e/tools/proxy` from .env values.
+      # See ../tools/proxy `render_config()`. The file is gitignored.
+      - ./.litellm.rendered.yaml:/app/config.yaml:ro
+    command:
+      - --config=/app/config.yaml
+      - --port=4000
+    depends_on:
+      db:
+        condition: service_healthy
+    healthcheck:
+      # /health/liveliness comes for free without DB / provider connectivity,
+      # but readiness needs migrations to finish — give it more time.
+      test:
+        - CMD-SHELL
+        - >-
+          python3 -c "import urllib.request;
+          urllib.request.urlopen('http://localhost:4000/health/liveliness')"
+      interval: 5s
+      timeout: 3s
+      retries: 24                # 24*5s = 2 min; covers prisma migrate cold start
+      start_period: 30s
+    restart: "no"