Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/e2e/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,6 @@ other.<area>.<case>.<assertion>

- when it comes to typing an input schema for an api endpoint, have it type X = A | B | C ... where X = exhaustive union of all supported input schemas and A, B, C typically are composed by a base type. types are only pretty for a api request / response body. make sure to compose types instead of repeating the same base attributes over and over again.

- use the docker-compose to your advantage and spin up a local proxy, make sure all tests pass. if a test fails due to an internally found issue, let users know to create a linear ticket for it.
- spin up a local proxy by running the litellm proxy locally (`litellm --config <your-e2e-config>.yml --port 4000`; see CONTRIBUTING.md), make sure all tests pass. if a test fails due to an internally found issue, let users know to create a linear ticket for it.

- do not use xfail markers, tests should be written in a form that the end user expects it to pass
38 changes: 18 additions & 20 deletions tests/e2e/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,26 +9,33 @@ When contributing to this directory, please first discuss the change you wish to

## Setup

The suites run against a live proxy, so bring one up first. `docker-compose.yml` here starts that proxy with a throwaway Postgres and Redis; `docker compose down -v` resets everything, so no state leaks between runs. The proxy config is inlined in the compose file under `configs`, prewired with example models (`gpt-5.5`, `claude-haiku-4-5`, `gemini-2.5-flash`, `openai-text-embedding-3-small`) whose keys come from your `.env`. If your test needs another model, a pricing override, or a guardrail declared up front, add it to that inline config and read it back in the test rather than hardcoding values
The suites run against a live proxy, so bring one up first by running the litellm proxy locally. Point it at a config that prewires the example models the suites use (`gpt-5.5`, `claude-haiku-4-5`, `gemini-2.5-flash`, `openai-text-embedding-3-small`) with keys from your `.env`, and enables prompt storage, a redis cache, and the fast budget rescheduler the quota suites rely on. If your test needs another model, a pricing override, or a guardrail declared up front, add it to that config and read it back in the test rather than hardcoding values

## Running the tests locally

1. Create a `.env` file in this directory with the provider keys the example models use:
1. Create a `.env` file in this directory with the provider keys the example models use, plus the master key and the Postgres/Redis coordinates your config reads back:

```bash
LITELLM_MASTER_KEY="sk-1234"
DATABASE_URL="postgresql://llmproxy:dbpassword9090@localhost:5432/litellm"
REDIS_HOST="localhost"
REDIS_PORT="6379"
OPENAI_API_KEY="sk-..."
ANTHROPIC_API_KEY="sk-..."
GEMINI_API_KEY="..."
```

2. Bring the stack up from this directory:
2. Bring up a Postgres and a Redis for the proxy to use. The repo-root `docker-compose.yml` already defines a Postgres on `5432`; a `docker run -p 6379:6379 redis:7` covers Redis. Point `DATABASE_URL` / `REDIS_HOST` / `REDIS_PORT` at whatever you run

3. Start the litellm proxy locally against your config and confirm it is live:

```bash
docker compose up -d
set -a && source .env && set +a
litellm --config <your-e2e-config>.yml --port 4000
curl -fs http://localhost:4000/health/liveliness
```

3. Run a suite against it; the harness reads `LITELLM_PROXY_URL` (default `http://localhost:4000`):
4. Run a suite against it; the harness reads `LITELLM_PROXY_URL` (default `http://localhost:4000`):

```bash
uv run pytest tests/e2e/llm_translation/ -v
Expand All @@ -41,20 +48,11 @@ The suites run against a live proxy, so bring one up first. `docker-compose.yml`
uv run playwright install chromium
```

They also need a proxy whose bundled UI contains the change under test. The published `main-latest` image ships the UI from the last release; to test local UI changes, build the image from your branch and point the compose stack at it:

```bash
docker build -t litellm-local .
LITELLM_E2E_IMAGE=litellm-local docker compose up -d
```
They also need a proxy whose bundled UI contains the change under test, so run the proxy from your branch (an editable install serves the UI your checkout builds)

4. Tear it down when you're done:

```bash
docker compose down -v
```
Some suites need extra services the bare proxy does not start. The `logging/` OTEL trace-completeness tests read spans back from a jaeger query API at `http://localhost:16686` (override with `E2E_OTEL_QUERY_URL`); run a `jaegertracing/all-in-one` and point `PHOENIX_COLLECTOR_HTTP_ENDPOINT` at its OTLP ingest. The `mcp/` suite needs the deterministic upstream MCP server in `mcp_tests/mcp_e2e_upstream_server.py` reachable by the proxy

Tests marked `@pytest.mark.e2e` hard-fail when no proxy answers `/health/liveliness`, so a run that goes red with `No live proxy` at setup means the stack isn't up; they never skip for a missing proxy, so an absent stack can't be mistaken for a pass
Tests marked `@pytest.mark.e2e` hard-fail when no proxy answers `/health/liveliness`, so a run that goes red with `No live proxy` at setup means the proxy isn't up; they never skip for a missing proxy, so an absent proxy can't be mistaken for a pass

## What a complete test looks like

Expand Down Expand Up @@ -142,12 +140,12 @@ Before you push

1. Run `make lint-e2e-basedpyright` (or `make pre-commit` with your changes staged); the harness is fully typed and the gate allows zero basedpyright errors, enforced in CI on any PR touching `tests/e2e/**/*.py`

2. Add the models your test needs to the inline config in `docker-compose.yml`
2. Add the models your test needs to the config your local proxy loads

3. Bring the stack up and run your suite against it:
3. Start the litellm proxy locally and run your suite against it:

```bash
docker compose up -d
litellm --config <your-e2e-config>.yml --port 4000
uv run pytest tests/e2e/<your_suite>/ -v
```

Expand Down
165 changes: 0 additions & 165 deletions tests/e2e/docker-compose.yml

This file was deleted.

Loading