Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ Same thing for bug fixes. The tests should make it so that this specific bug can

`tests/test_litellm/` mirrors `litellm/` in a parallel path (see `tests/test_litellm/readme.md`). Name tests `test_<filename>.py`, but always match the existing test file in the directory you touch — many provider dirs use longer descriptive names (e.g. `test_anthropic_chat_transformation.py`) to avoid ambiguity across sibling folders. For bug fixes, extend the existing mapped test file rather than creating a new one. Only create a new test file for a new feature (provider, endpoint, or transformation module) that has no mapped test yet, following that directory's naming convention (or `test_<filename>.py` if you're the first test there). One focused regression test beats many shallow ones

End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `CLAUDE.md`

When creating PRs, don't set base to `main`. `litellm_internal_staging` serves that purpose

Always use @.github/pull_request_template.md as a guide for your PR body
Expand Down
27 changes: 20 additions & 7 deletions tests/e2e/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,18 @@ When contributing to this directory, please first discuss the change you wish to

## Setup

The suites run against a live proxy, so bring one up first. `docker-compose.yml` here starts that proxy with its Postgres and Redis, serving `gateway/litellm-config.yml`; add any model, pricing override, or guardrail your test needs to that file and read it back in the test rather than hardcoding values. `gateway/` holds proxy configuration only, so never put tests there
The suites run against a live proxy, so bring one up first. `docker-compose.yml` here starts that proxy with a throwaway Postgres and Redis; `docker compose down -v` resets everything, so no state leaks between runs. The proxy config is inlined in the compose file under `configs`, prewired with example models (`gpt-5.5`, `claude-haiku-4-5`, `gemini-2.5-flash`, `openai-text-embedding-3-small`) whose keys come from your `.env`. If your test needs another model, a pricing override, or a guardrail declared up front, add it to that inline config and read it back in the test rather than hardcoding values

## Running the tests locally

1. Create a .env file and add provider keys:
1. Create a `.env` file in this directory with the provider keys the example models use:

```bash
OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-..."

GEMINI_API_KEY="..."
```

2. Bring the stack up from this directory:

```bash
Expand Down Expand Up @@ -123,7 +126,17 @@ Mark live tests with `@pytest.mark.e2e` (on the class or the module). Pure cover

Before you push

- Run basedpyright over your changes; the harness is fully typed and new code must not add `Any` or widen the budgets
- Bring the stack up with docker-compose from this directory and run your suite locally against it, so you exercise the same skip-vs-fail path CI does
- Use the config at `tests/e2e/gateway/litellm-config.yml` if your feature needs a model, pricing override, guardrail, or other proxy setting declared up front; add the deployment there and read it back in the test rather than hardcoding values
- Capture screenshots of the tests passing and attach them to the PR as proof of fix
1. Run basedpyright over your changes; the harness is fully typed and new code must not add `Any` or widen the budgets

2. Add the models your test needs to the inline config in `docker-compose.yml`

3. Bring the stack up and run your suite against it:

```bash
docker compose up -d
uv run pytest tests/e2e/<your_suite>/ -v
```

4. Capture screenshots of the test run and attach them to the PR as proof

5. If a test fails because it surfaced a real issue in the product, flag that explicitly in the PR rather than reworking the test until it passes
87 changes: 87 additions & 0 deletions tests/e2e/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# local setup to run e2e tests
configs:
litellm_config:
content: |
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
store_prompts_in_spend_logs: true

litellm_settings:
drop_params: true
num_retries: 3
request_timeout: 600
cache: true
cache_params:
type: redis
host: redis
port: 6379

router_settings:
routing_strategy: simple-shuffle
num_retries: 3
allowed_fails: 5
cooldown_time: 30
fallbacks:
- gemini-2.5-flash: ["gpt-5.5", "claude-haiku-4-5"]

model_list:
- model_name: gpt-5.5
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEY

- model_name: claude-haiku-4-5
litellm_params:
model: anthropic/claude-haiku-4-5
api_key: os.environ/ANTHROPIC_API_KEY

- model_name: gemini-2.5-flash
litellm_params:
model: gemini/gemini-2.5-flash
api_key: os.environ/GEMINI_API_KEY

- model_name: openai-text-embedding-3-small
litellm_params:
model: openai/text-embedding-3-small
api_key: os.environ/OPENAI_API_KEY

services:
litellm:
image: ghcr.io/berriai/litellm:main-latest
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
env_file: .env
environment:
LITELLM_MASTER_KEY: sk-1234
DATABASE_URL: postgresql://litellm:litellm@db:5432/litellm
ports:
- "4000:4000"
Comment thread
mubashir1osmani marked this conversation as resolved.
configs:
- source: litellm_config
target: /app/config.yaml
command: ["--config", "/app/config.yaml", "--port", "4000"]

# throwaway db
db:
image: postgres:16
environment:
POSTGRES_USER: litellm
POSTGRES_PASSWORD: litellm
POSTGRES_DB: litellm
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 3s
timeout: 3s
retries: 20

redis:
image: redis:7
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 3s
timeout: 3s
retries: 20
Loading
Loading