From 0b01d9e70b161018f40e3cff5d8995129cea76ea Mon Sep 17 00:00:00 2001 From: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Date: Fri, 22 May 2026 08:49:47 +0000 Subject: [PATCH] docs(claude): refactor CLAUDE.md for higher signal-to-noise MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure the project guidance file along three axes without dropping any rule: - Hoist the five most safety-critical rules into a "Hot path" section at the top so Claude grep-anchors there first when reading the file. - Add an "Agent constraints (this fork)" section that names the actions Claude must not take autonomously (push to ship/internal, run scripts/release-tag.sh, edit upstream UI build artifacts). - Add a "Test discipline" section covering spec-first assertions, pytest.mark.xfail(strict=True) for known doc-vs-impl gaps (not skip / TODO), and the "test exposed a bug != bug is fixed" PR-summary norm. Mechanical cleanups: - Collapse five command-list sections (Installation / Testing / Code Quality / Single Test Files / Running Scripts) into one "Development workflow" block with a single shell example. - Compress the "Architecture Overview" enumeration of folders — what litellm/main.py, litellm/llms/, litellm/proxy/auth/ etc. contain is inferable from `ls`. Keep only non-obvious design decisions (Router sync vs proxy async; litellm_extras/ isolation; UI build-artifact format churn). - Drop emphasis (**bold** / NEVER / MUST) on style preferences that don't cause production incidents (dict spread, comprehension extraction, FastAPI Query declarations). Emphasis remaining is reserved for rules whose violation has caused or could cause data loss, security holes, or production outages. Numbers: - words 2538 → 2000 (-21%) - estimated tokens 3300 → 2600 (-21%) - emphasis markers 38 → 9 (-76%) All 13 spot-checked critical rules verified preserved: Never-raw-SQL, Prisma-model-methods, LLMClientCache._remove_key, release-tag.sh, ship/v1.83.10 PR target, antd migration, sessionStorage, schema.prisma, TRANSPORT.OPENAPI, MCP credential table, no-pipe-curl-to-shell, check_valid_key, litellm-proxy-extras migration lag. --- CLAUDE.md | 520 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 301 insertions(+), 219 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 28252b06daeb..ef405d8780d0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,243 +1,325 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +Guidance for Claude Code (claude.ai/code) when working in this repo. +This is an internal fork of `BerriAI/litellm` pinned to the upstream +`v1.83.10-stable` tag, with internal fixes layered on top. + +## Hot path — read this before doing anything + +- Every fix PR targets **`ship/v1.83.10`**, not `internal/v1.83.10-stable` + (has 1700+ upstream-sync commits) and not `litellm_internal_staging` + (pure upstream tracker). +- Run `uv run black .` before committing — CI enforces. +- Full-stack scenarios (DB schema, background jobs, real HTTP) go under + `e2e/cases/NN_*.md`, not into bespoke `tests/` integration files. +- Proxy DB access uses Prisma model methods (`prisma_client.db.`) + only — no raw SQL. +- `LLMClientCache._remove_key()` must never close HTTP/SDK clients; + in-flight requests still hold them. + +## Agent constraints (this fork) + +Do not take these actions autonomously without explicit instruction: + +- Push to `ship/v1.83.10`, `internal/v1.83.10-stable`, or any tag. + Always work via a `fix/*` branch + PR. +- Run `scripts/release-tag.sh`. It has an interactive `[y/N]` prompt + meant for the human releaser — suggest `! scripts/release-tag.sh v...` + so they own the confirmation. +- Edit files under `litellm/proxy/_experimental/out/`. Those are + upstream Next.js build artifacts; ignore the `git status` noise there. +- Skip pre-commit hooks (`--no-verify`), bypass signing, or amend an + already-pushed commit. + +## Development workflow -## Development Commands - -### Installation -- `make install-dev` - Install core development dependencies -- `make install-proxy-dev` - Install proxy development dependencies with full feature set -- `make install-test-deps` - Install the full local test environment and generate the Prisma client - -### Testing -- `make test` - Run all tests -- `make test-unit` - Run unit tests (tests/test_litellm) with 4 parallel workers -- `make test-integration` - Run integration tests (excludes unit tests) -- `pytest tests/` - Direct pytest execution - -### Code Quality -- `make lint` - Run all linting (Ruff, MyPy, Black, circular imports, import safety) -- `make format` - Apply Black code formatting -- `make lint-ruff` - Run Ruff linting only -- `make lint-mypy` - Run MyPy type checking only -- **Before committing, always run `uv run black .` to format your code.** Black formatting is enforced in CI. - -### Single Test Files -- `uv run pytest tests/path/to/test_file.py -v` - Run specific test file -- `uv run pytest tests/path/to/test_file.py::test_function -v` - Run specific test - -### Running Scripts -- `uv run python script.py` - Run Python scripts (use for non-test files) - -### GitHub Issue & PR Templates -When contributing to the project, use the appropriate templates: - -**Bug Reports** (`.github/ISSUE_TEMPLATE/bug_report.yml`): -- Describe what happened vs. what you expected -- Include relevant log output -- Specify your LiteLLM version - -**Feature Requests** (`.github/ISSUE_TEMPLATE/feature_request.yml`): -- Describe the feature clearly -- Explain the motivation and use case - -**Pull Requests** (`.github/pull_request_template.md`): -- Add at least 1 test in `tests/litellm/` -- Ensure `make test-unit` passes - -### Branching strategy (internal fork) - -This fork pins to the upstream `v1.83.10-stable` tag and ships internal -fixes on top of it. +```bash +# Install +make install-dev # core dev deps +make install-proxy-dev # proxy with full feature set +make install-test-deps # full local test env + Prisma client + +# Run tests +make test-unit # tests/test_litellm with 4 workers +make test-integration # everything except unit +uv run pytest tests/path/to/test_file.py -v +uv run pytest tests/path/to/test_file.py::test_function -v + +# Lint / format +make lint # Ruff + MyPy + Black + circular-import + import-safety +make format # Black only +uv run black . # MANDATORY before commit +uv run python script.py # for non-test scripts +``` -**Branches:** +## Branching strategy | Branch | Purpose | Stays clean? | |---|---|---| | `v1.83.10-stable` (tag) | Immutable upstream pin | yes — never moves | -| `ship/v1.83.10` | Long-term ship branch — starts at the tag, only advances via merges of internal `fix/*` PRs | yes | -| `internal/v1.83.10-stable` | Upstream-sync working branch — may collect upstream commits via teammate / CI sync | **no** — can have hundreds of upstream commits | +| `ship/v1.83.10` | Long-term ship — advances only via merged `fix/*` PRs | yes | +| `internal/v1.83.10-stable` | Upstream-sync working branch | no — collects upstream commits | | `litellm_internal_staging` | Pure upstream tracker for `BerriAI/litellm` | tracks upstream | -| `fix/` | Per-bug feature branch | yes — merged into `ship/v1.83.10` via PR merge commit | - -**PR target:** every internal fix PR **must target `ship/v1.83.10`**, not -`internal/v1.83.10-stable` (which has 1700+ upstream-sync commits on top -of the tag) and not `litellm_internal_staging` (pure upstream). +| `fix/` | Per-bug feature branch | yes — merged into `ship/v1.83.10` | ```bash -# Default new fix branch from the latest ship state git checkout -b fix/ ship/v1.83.10 - -# Open PR gh pr create --base ship/v1.83.10 --head fix/ ``` -**Conflicts:** `ship/v1.83.10` only moves when a `fix/*` PR merges, so it -stays exactly TAG + (merged fixes). Fixes never have to rebase against -moving upstream; the upstream-sync churn lives entirely on -`internal/v1.83.10-stable`. +`ship/v1.83.10` only moves when a `fix/*` PR merges, so it stays exactly +`TAG + merged fixes`. Fixes never have to rebase against moving upstream +— the upstream-sync churn lives on `internal/v1.83.10-stable`. -### Cutting an internal release +## Cutting an internal release -**Always use `scripts/release-tag.sh`. Never `git tag` by hand.** +Always use `scripts/release-tag.sh`. Never `git tag` by hand. ```bash -# After a fix/* PR has been merged into ship/v1.83.10 and you've -# fast-forwarded local ship/v1.83.10 to match origin: +# After fix/* PR merged into ship/v1.83.10 and local ship is up-to-date: scripts/release-tag.sh v1.83.10-internal.N # N = next integer ``` -The script preflight-checks current branch (must be `ship/*`), worktree -cleanliness, local/origin sync, and tag non-existence; auto-generates -the changelog from the previous `internal.N` tag; then prompts `[y/N]` -before creating the annotated tag and pushing it. Pushing the tag -triggers `.github/workflows/release-docker.yml`, which builds a -multi-arch image and pushes: +Tag format is enforced: `^v[0-9]+\.[0-9]+\.[0-9]+-internal\.[0-9]+$`. +`N` is monotonically increasing — don't reset, skip, or reuse. The +current latest: + +```bash +git tag -l 'v1.83.10-internal.*' --sort=-v:refname | head -1 +``` +Pushing the tag triggers `.github/workflows/release-docker.yml` → +multi-arch image published as: - `zsk2026/litellm:vX.Y.Z-internal.N` -- `zsk2026/litellm:vX.Y.Z-stable` (rolling pointer to the latest - `-internal.N` on the same base) - -**Tag format is enforced:** `^v[0-9]+\.[0-9]+\.[0-9]+-internal\.[0-9]+$` -— the workflow trigger (`v*-internal.*`) and the script's regex both -key off this. Anything else won't release. - -**Versioning:** `N` is a monotonically increasing integer on top of the -upstream base (`v1.83.10`). Don't reset, don't skip, don't reuse. -`git tag -l 'v1.83.10-internal.*' --sort=-v:refname | head -1` -shows the current. - -Because the script is interactive (`read -rp "Proceed? [y/N]"`), prefer -asking the user to run it directly (`! scripts/release-tag.sh v...`) so -they own the y/N confirmation — never auto-answer it on their behalf. - -## Architecture Overview - -LiteLLM is a unified interface for 100+ LLM providers with two main components: - -### Core Library (`litellm/`) -- **Main entry point**: `litellm/main.py` - Contains core completion() function -- **Provider implementations**: `litellm/llms/` - Each provider has its own subdirectory -- **Router system**: `litellm/router.py` + `litellm/router_utils/` - Load balancing and fallback logic -- **Type definitions**: `litellm/types/` - Pydantic models and type hints -- **Integrations**: `litellm/integrations/` - Third-party observability, caching, logging -- **Caching**: `litellm/caching/` - Multiple cache backends (Redis, in-memory, S3, etc.) - -### Proxy Server (`litellm/proxy/`) -- **Main server**: `proxy_server.py` - FastAPI application -- **Authentication**: `auth/` - API key management, JWT, OAuth2 -- **Database**: `db/` - Prisma ORM with PostgreSQL/SQLite support -- **Management endpoints**: `management_endpoints/` - Admin APIs for keys, teams, models -- **Pass-through endpoints**: `pass_through_endpoints/` - Provider-specific API forwarding -- **Guardrails**: `guardrails/` - Safety and content filtering hooks -- **UI Dashboard**: Served from `_experimental/out/` (Next.js build) - -## Key Patterns - -### Provider Implementation -- Providers inherit from base classes in `litellm/llms/base.py` -- Each provider has transformation functions for input/output formatting -- Support both sync and async operations -- Handle streaming responses and function calling - -### Error Handling -- Provider-specific exceptions mapped to OpenAI-compatible errors -- Fallback logic handled by Router system -- Comprehensive logging through `litellm/_logging.py` - -### Configuration -- YAML config files for proxy server (see `proxy/example_config_yaml/`) -- Environment variables for API keys and settings -- Database schema managed via Prisma (`proxy/schema.prisma`) - -## Development Notes - -### Code Style -- Uses Black formatter, Ruff linter, MyPy type checker -- Pydantic v2 for data validation -- Async/await patterns throughout -- Type hints required for all public APIs -- **Avoid imports within methods** — place all imports at the top of the file (module-level). Inline imports inside functions/methods make dependencies harder to trace and hurt readability. The only exception is avoiding circular imports where absolutely necessary. -- **Use dict spread for immutable copies** — prefer `{**original, "key": new_value}` over `dict(obj)` + mutation. The spread produces the final dict in one step and makes intent clear. -- **Guard at resolution time** — when resolving an optional value through a fallback chain (`a or b or ""`), raise immediately if the resolved result being empty is an error. Don't pass empty strings or sentinel values downstream for the callee to deal with. -- **Extract complex comprehensions to named helpers** — a set/dict comprehension that calls into the DB or manager (e.g. "which of these server IDs are OAuth2?") belongs in a named helper function, not inline in the caller. -- **FastAPI parameter declarations** — mark required query/form params with `= Query(...)` / `= Form(...)` explicitly when other params in the same handler are optional. Mixing `str` (required) with `Optional[str] = None` in the same signature causes silent 422s when the required param is missing. - -### Testing Strategy -- Unit tests in `tests/test_litellm/` -- Integration tests for each provider in `tests/llm_translation/` -- Proxy tests in `tests/proxy_unit_tests/` -- Load tests in `tests/load_tests/` -- **End-to-end tests in `e2e/` (root, not under `tests/`)** — Claude-driven runbook harness, not pytest. Cases live in `e2e/cases/NN_*.md`; tools in `e2e/tools/` (`proxy`, `keys`, `teams`, `metrics`, `call`, `run-all-cases`). Docker Compose brings up Postgres + a litellm container built from local source. **Whenever a fix needs full-stack verification (DB schema, background jobs, real HTTP flow), add a new case under `e2e/cases/` rather than spinning up bespoke integration infra under `tests/`.** Update the index in `e2e/cases/README.md`. See `e2e/README.md` for the harness contract. -- **Always add tests when adding new entity types or features** — if the existing test file covers other entity types, add corresponding tests for the new one -- **Keep monkeypatch stubs in sync with real signatures** — when a function gains a new optional parameter, update every `fake_*` / `stub_*` in tests that patch it to also accept that kwarg (even as `**kwargs`). Stale stubs fail with `unexpected keyword argument` and mask real bugs. -- **Test all branches of name→ID resolution** — when adding server/resource lookup that resolves names to UUIDs, test: (1) name resolves and UUID is allowed, (2) name resolves but UUID is not allowed, (3) name does not resolve at all. The silent-fallback path is where access-control bugs hide. - -### UI / Backend Consistency -- When wiring a new UI entity type to an existing backend endpoint, verify the backend API contract (single value vs. array, required vs. optional params) and ensure the UI controls match — e.g., use a single-select dropdown when the backend accepts a single value, not a multi-select - -### UI Component Library -- **Always use `antd` for new UI components** — we are migrating off of `@tremor/react`. Do not introduce new `Badge`, `Text`, `Card`, `Grid`, `Title`, or other imports from `@tremor/react` in any new or modified file. Use `antd` equivalents: `Tag` for labels, `Typography.Text` / `Typography.Title` / `Typography.Paragraph` for textual content (avoid plain text-only ``, `

`, `` when Typography fits), and `Card` from `antd`. Note that `antd` has no `"yellow"` Tag color — use `"gold"` for amber/yellow. - -### MCP OAuth / OpenAPI Transport Mapping -- `TRANSPORT.OPENAPI` is a UI-only concept. The backend only accepts `"http"`, `"sse"`, or `"stdio"`. Always map it to `"http"` before any API call (including pre-OAuth temp-session calls). -- FastAPI validation errors return `detail` as an array of `{loc, msg, type}` objects. Error extractors must handle: array (map `.msg`), string, nested `{error: string}`, and fallback. -- When an MCP server already has `authorization_url` stored, skip OAuth discovery (`_discovery_metadata`) — the server URL for OpenAPI MCPs is the spec file, not the API base, and fetching it causes timeouts. -- `client_id` should be optional in the `/authorize` endpoint — if the server has a stored `client_id` in credentials, use that. Never require callers to re-supply it. - -### MCP Credential Storage -- OAuth credentials and BYOK credentials share the `litellm_mcpusercredentials` table, distinguished by a `"type"` field in the JSON payload (`"oauth2"` vs plain string). -- When deleting OAuth credentials, check type before deleting to avoid accidentally deleting a BYOK credential for the same `(user_id, server_id)` pair. -- Always pass the raw `expires_at` timestamp to the client — never set it to `None` for expired credentials. Let the frontend compute the "Expired" display state from the timestamp. -- Use `RecordNotFoundError` (not bare `except Exception`) when catching "already deleted" in credential delete endpoints. - -### Browser Storage Safety (UI) -- Never write LiteLLM access tokens or API keys to `localStorage` — use `sessionStorage` only. `localStorage` survives browser close and is readable by any injected script (XSS). -- Shared utility functions (e.g. `extractErrorMessage`) belong in `src/utils/` — never define them inline in hooks or duplicate them across files. - -### Database Migrations -- Prisma handles schema migrations -- Migration files auto-generated with `prisma migrate dev` -- Always test migrations against both PostgreSQL and SQLite - -### Proxy database access -- **Do not write raw SQL** for proxy DB operations. Use Prisma model methods instead of `execute_raw` / `query_raw`. -- Use the generated client: `prisma_client.db.` (e.g. `litellm_tooltable`, `litellm_usertable`) with `.upsert()`, `.find_many()`, `.find_unique()`, `.update()`, `.update_many()` as appropriate. This avoids schema/client drift, keeps code testable with simple mocks, and matches patterns used in spend logs and other proxy code. -- **No N+1 queries.** Never query the DB inside a loop. Batch-fetch with `{"in": ids}` and distribute in-memory. -- **Batch writes.** Use `create_many`/`update_many`/`delete_many` instead of individual calls (these return counts only; `update_many`/`delete_many` no-op silently on missing rows). When multiple separate writes target the same table (e.g. in `batch_()`), order by primary key to avoid deadlocks. -- **Push work to the DB.** Filter, sort, group, and aggregate in SQL, not Python. Verify Prisma generates the expected SQL — e.g. prefer `group_by` over `find_many(distinct=...)` which does client-side processing. -- **Bound large result sets.** Prisma materializes full results in memory. For results over ~10 MB, paginate with `take`/`skip` or `cursor`/`take`, always with an explicit `order`. Prefer cursor-based pagination (`skip` is O(n)). Don't paginate naturally small result sets. -- **Limit fetched columns on wide tables.** Use `select` to fetch only needed fields — returns a partial object, so downstream code must not access unselected fields. -- **Check index coverage.** For new or modified queries, check `schema.prisma` for a supporting index. Prefer extending an existing index (e.g. `@@index([a])` → `@@index([a, b])`) over adding a new one, unless it's a `@@unique`. Only add indexes for large/frequent queries. -- **Keep schema files in sync.** Apply schema changes to all `schema.prisma` copies (`schema.prisma`, `litellm/proxy/`, `litellm-proxy-extras/`, `litellm-js/spend-logs/` for SpendLogs) with a migration under `litellm-proxy-extras/litellm_proxy_extras/migrations/`. - -### Setup Wizard (`litellm/setup_wizard.py`) -- The wizard is implemented as a single `SetupWizard` class with `@staticmethod` methods — keep it that way. No module-level functions except `run_setup_wizard()` (the public entrypoint) and pure helpers (color, ANSI). -- Use `litellm.utils.check_valid_key(model, api_key)` for credential validation — never roll a custom completion call. -- Do not hardcode provider env-key names or model lists that already exist in the codebase. Add a `test_model` field to each provider entry to drive `check_valid_key`; set it to `None` for providers that can't be validated with a single API key (Azure, Bedrock, Ollama). - -### Enterprise Features -- Enterprise-specific code in `enterprise/` directory -- Optional features enabled via environment variables -- Separate licensing and authentication for enterprise features - -### CI Supply-Chain Safety -- **Never pipe a remote script into a shell** (`curl ... | bash`, `wget ... | sh`). Download the artifact to a file, verify its SHA-256 checksum, then install. -- **Pin every external tool to a specific version** with a full URL (not `latest` or `stable`). Unversioned downloads silently change under you. -- **Verify checksums for all downloaded binaries.** Use the provider's official `.sha256` / `.sha256sum` sidecar file when available; otherwise compute and hardcode the digest. -- **Prefer reusable CircleCI commands** (`commands:` section) so a tool is installed and verified in exactly one place, then referenced everywhere with `- install_` or `- wait_for_service`. -- **Don't add tools just because they were there before.** Audit whether an external dependency is still needed. If it can be replaced with a shell one-liner or a tool already in the image, remove it. -- These rules apply to every download in CI: binaries, install scripts, language version managers, package repos. No exceptions. - -### HTTP Client Cache Safety -- **Never close HTTP/SDK clients on cache eviction.** `LLMClientCache._remove_key()` must not call `close()`/`aclose()` on evicted clients — they may still be used by in-flight requests. Doing so causes `RuntimeError: Cannot send a request, as the client has been closed.` after the 1-hour TTL expires. Cleanup happens at shutdown via `close_litellm_async_clients()`. - -### Troubleshooting: DB schema out of sync after proxy restart -`litellm-proxy-extras` runs `prisma migrate deploy` on startup using **its own** bundled migration files, which may lag behind schema changes in the current worktree. Symptoms: `Unknown column`, `Invalid prisma invocation`, or missing data on new fields. - -**Diagnose:** Run `\d "TableName"` in psql and compare against `schema.prisma` — missing columns confirm the issue. - -**Fix options:** -1. **Create a Prisma migration** (permanent) — run `prisma migrate dev --name ` in the worktree. The generated file will be picked up by `prisma migrate deploy` on next startup. -2. **Apply manually for local dev** — `psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF NOT EXISTS ..."` after each proxy start. Fine for dev, not for production. -3. **Update litellm-proxy-extras** — if the package is installed from PyPI, its migration directory must include the new file. Either update the package or run the migration manually until the next release ships it. +- `zsk2026/litellm:vX.Y.Z-stable` (rolling pointer to latest `internal.N`) + +## Architecture notes + +- Provider transformations live in `litellm/llms//` and inherit + from `litellm/llms/base.py`. Adding a provider = new subdir + base + subclass + input/output transforms. +- `Router` (`litellm/router.py`, sync-friendly) vs `proxy_server` + (`litellm/proxy/proxy_server.py`, async FastAPI) — never call sync + Router methods from async proxy code. +- Internal code in this fork lives under `litellm_extras/` to keep + upstream `litellm/` untouched. Don't import `litellm_extras` from + inside `litellm/`. +- UI is a Next.js build under `litellm/proxy/_experimental/out/` — + committed to git by upstream, occasionally changes format (`.html` ↔ + `/index.html`). Add `litellm/proxy/_experimental/out/` to + `.git/info/exclude` locally to silence the noise. + +## Code style + +- Black formatter, Ruff linter, MyPy type checker. +- Pydantic v2 for data validation; type hints required on public APIs. +- **Avoid imports within methods** — module-level imports only. Inline + imports hide dependencies and break static analysis. Only exception: + breaking a circular import where unavoidable. +- Prefer `{**original, "key": new_value}` over `dict(obj)` + mutation. +- Guard at resolution time: when resolving an optional via fallback + chain (`a or b or ""`), raise immediately if the resolved result + being empty is an error. Don't pass empty strings or sentinels + downstream. +- Extract complex comprehensions that call into the DB/manager into a + named helper — don't inline them. +- FastAPI handlers mixing required and optional params: mark required + ones with `= Query(...)` / `= Form(...)` explicitly. Otherwise you'll + get silent 422s when the required param is missing. + +## Test discipline + +Tests live in `tests/test_litellm/` (unit), `tests/llm_translation/` +(per-provider integration), `tests/proxy_unit_tests/` (proxy), and +`tests/load_tests/`. Full-stack scenarios go in `e2e/cases/`. + +- **Write assertions from the spec, not the impl.** For new features, + the `e2e/cases/NN_*.md` runbook IS the spec — write it before the + fixture and the impl. For bug fixes, the issue's repro steps are the + spec. Tests reverse-engineered from controller code can never expose + a doc-vs-impl gap because they were generated from the gap. +- **Lock known doc-vs-impl gaps with `pytest.mark.xfail(strict=True)`**, + never `pytest.mark.skip` or a `TODO` comment. Skipped tests vanish + from CI signal and rot. `xfail(strict=True)` keeps the gap visible + AND flips to a build failure (XPASS) the moment the impl catches up, + forcing cleanup. Include a `reason=` that points at the upstream + issue or internal ticket. Pair with a plain `test_*_current_behavior` + that pins the wrong-but-current behavior so drift surfaces too: + ```python + def test_x_current_behavior(): + assert actual == BUGGY_VALUE # codifies the bug + + @pytest.mark.xfail(strict=True, reason="BerriAI/litellm#NNNNN") + def test_x_correct_behavior(): + assert actual == EXPECTED_VALUE # flips XPASS when impl catches up + ``` +- **"Test exposed a bug" ≠ "bug is fixed".** Adding a failing test (or + a strict-xfail) documents a gap; it does not close one. Fix the impl + in the same PR, or call out the deferral in the PR summary + ("exposes #N, fix deferred to #M"). +- Keep monkeypatch stubs in sync with real signatures. When a function + gains a new optional param, update every `fake_*` / `stub_*` to accept + it (even as `**kwargs`). Stale stubs fail with `unexpected keyword + argument` and mask real bugs. +- Test all branches of name→ID resolution: (1) name resolves and UUID + allowed, (2) name resolves but UUID not allowed, (3) name doesn't + resolve. The silent-fallback path is where access-control bugs hide. +- Always add tests when introducing a new entity type — if existing + test files cover other entity types, add corresponding cases. + +## Proxy database access + +Use Prisma model methods. **Never raw SQL** (`execute_raw`/`query_raw`). + +- Client: `prisma_client.db.` with `.upsert`/`.find_many`/ + `.find_unique`/`.update`/`.update_many`. +- **No N+1 queries.** Batch-fetch with `{"in": ids}` and distribute + in-memory. +- Batch writes via `create_many`/`update_many`/`delete_many` (these + return counts only; `update_many`/`delete_many` no-op silently on + missing rows). Multiple writes to the same table in `batch_()` → + order by primary key to avoid deadlocks. +- Push filter/sort/group/aggregate work into SQL. Verify Prisma + generates expected SQL — e.g. prefer `group_by` over + `find_many(distinct=...)` (the latter does client-side processing). +- For results > ~10 MB, paginate. Prefer cursor-based pagination + (`skip` is O(n)). Always include explicit `order`. +- Use `select` on wide tables to fetch only needed columns. Downstream + code must not access unselected fields. +- Check index coverage in `schema.prisma`. Prefer extending an + existing index over adding a new one (unless `@@unique`). Only add + indexes for large/frequent queries. +- **Schema changes must update all four `schema.prisma` copies** + (`schema.prisma`, `litellm/proxy/`, `litellm-proxy-extras/`, + `litellm-js/spend-logs/` for SpendLogs) plus a migration under + `litellm-proxy-extras/litellm_proxy_extras/migrations/`. + +## HTTP client cache safety + +**`LLMClientCache._remove_key()` must not call `close()` / `aclose()` +on evicted clients** — in-flight requests still hold them, and closing +mid-flight raises `RuntimeError: Cannot send a request, as the client +has been closed.` after the 1-hour TTL expires. Cleanup happens at +shutdown via `close_litellm_async_clients()`. + +## MCP OAuth / OpenAPI transport mapping + +- `TRANSPORT.OPENAPI` is a UI-only concept. The backend only accepts + `"http"`, `"sse"`, or `"stdio"`. Map to `"http"` before any API call + (including pre-OAuth temp-session calls). +- FastAPI validation errors return `detail` as `[{loc, msg, type}, ...]`. + Error extractors must handle: array (map `.msg`), string, nested + `{error: string}`, and a fallback. +- If an MCP server has `authorization_url` stored, skip OAuth discovery + (`_discovery_metadata`) — the server URL for OpenAPI MCPs is the spec + file, not the API base, and fetching it causes timeouts. +- `client_id` is optional in `/authorize` — if the server has stored + `client_id` in credentials, use that. Never require callers to + re-supply it. + +## MCP credential storage + +- OAuth and BYOK credentials share `litellm_mcpusercredentials`, + distinguished by `"type"` in the JSON payload (`"oauth2"` vs plain + string). When deleting OAuth credentials, check type first to avoid + deleting a BYOK credential for the same `(user_id, server_id)` pair. +- Pass raw `expires_at` timestamps to the client — never `None` for + expired credentials. The frontend computes the "Expired" display + state from the timestamp. +- Catch `RecordNotFoundError` (not bare `except Exception`) for + "already deleted" in credential delete endpoints. + +## Browser storage safety (UI) + +**Never write LiteLLM access tokens or API keys to `localStorage`** — +use `sessionStorage` only. `localStorage` survives browser close and is +readable by any injected script (XSS). + +Shared utility functions (e.g. `extractErrorMessage`) belong in +`src/utils/` — never define inline in hooks or duplicate across files. + +## UI component library + +New UI work uses `antd`. We are migrating off `@tremor/react` — do not +introduce new `Badge`/`Text`/`Card`/`Grid`/`Title` imports from +`@tremor/react` in any new or modified file. Use `antd` equivalents: +`Tag` for labels, `Typography.Text`/`Typography.Title`/ +`Typography.Paragraph` for textual content (avoid plain ``/`

`/ +`` when Typography fits), `Card` from `antd`. `antd` has no +`"yellow"` Tag color — use `"gold"`. + +## UI / backend consistency + +When wiring a new UI entity to an existing backend endpoint, verify the +backend contract (single value vs. array, required vs. optional) and +match UI controls — e.g. single-select dropdown when the backend +accepts a single value, not a multi-select. + +## Setup wizard (`litellm/setup_wizard.py`) + +- Single `SetupWizard` class with `@staticmethod` methods. No + module-level functions except `run_setup_wizard()` and pure helpers + (color, ANSI). +- Validate credentials via `litellm.utils.check_valid_key(model, api_key)`, + not a custom completion call. +- Don't hardcode provider env-key names or model lists. Add a + `test_model` field per provider entry to drive `check_valid_key`; + set to `None` for providers that can't be validated with a single + key (Azure, Bedrock, Ollama). + +## CI supply-chain safety + +These rules apply to every download in CI — binaries, install scripts, +language version managers, package repos. No exceptions. + +- **Never pipe a remote script into a shell** (`curl ... | bash`). + Download to a file, verify SHA-256, then install. +- **Pin every external tool to a specific version** with a full URL. + No `latest` / `stable` — those silently change under you. +- **Verify checksums on downloaded binaries.** Use the provider's + `.sha256` sidecar if available, otherwise compute and hardcode. +- Prefer reusable CircleCI `commands:` so a tool is installed/verified + in exactly one place, referenced everywhere with `- install_`. +- Don't add tools just because they were there. Audit each external + dependency on every CI touch — remove if a shell one-liner or an + in-image tool can replace it. + +## Enterprise features + +Enterprise-only code in `enterprise/`. Optional features enable via env +vars; separate licensing and authentication. + +## Database migrations + +Prisma handles schema migrations. Migration files auto-generate with +`prisma migrate dev`. Always test migrations against both PostgreSQL +and SQLite. + +## Troubleshooting: DB schema out of sync after proxy restart + +`litellm-proxy-extras` runs `prisma migrate deploy` on startup using +**its own** bundled migrations, which may lag behind schema changes in +the current worktree. Symptoms: `Unknown column`, `Invalid prisma +invocation`, missing data on new fields. + +Diagnose: `\d "TableName"` in psql vs `schema.prisma` — missing columns +confirm. + +Fix: +1. **Permanent** — `prisma migrate dev --name ` in the + worktree. The generated file is picked up by `prisma migrate deploy` + on next startup. +2. **Local dev** — `psql -d litellm -c "ALTER TABLE ... ADD COLUMN IF + NOT EXISTS ..."` after each proxy start. Dev-only, not production. +3. **PyPI install** — if `litellm-proxy-extras` is installed from PyPI, + its migration directory must include the new file. Update the + package or apply the migration manually until the next release. + +## GitHub templates + +- Bug reports: `.github/ISSUE_TEMPLATE/bug_report.yml` — what happened + vs. expected + log output + LiteLLM version. +- Feature requests: `.github/ISSUE_TEMPLATE/feature_request.yml` — + describe + motivation + use case. +- PRs: `.github/pull_request_template.md` — at least 1 test in + `tests/litellm/`; `make test-unit` must pass.