chore(v1.87.0): port e2e/ test harness core (Wave 2) - #47
Merged
Conversation
Adds a top-level e2e/ test harness designed to be driven by Claude
Code: scripts are single-purpose Unix tools, scenarios live as
markdown runbooks (cases/*.md), no pytest framework lock-in.
Why this exists separate from tests/:
- tests/test_litellm/ uses mocks and never makes real provider calls,
so it cannot catch integration bugs where mock-based unit tests pass
but the real /metrics HTTP path or success-callback wiring is broken
- Real-provider tests cost money per run and depend on external API
availability — they must NEVER auto-run in `make test-unit` or CI
- Long-running development on this fork needs a reusable harness, not
one-off smoke scripts
Layout:
e2e/
├── README.md ← how to run, env contract
├── _config/
│ └── docker-compose.yml ← litellm built from local source + Postgres
├── tools/ ← Unix-style single-purpose CLIs
│ ├── proxy ← lifecycle: start|stop|status|logs|rebuild
│ ├── call ← one chat-completions request → JSON
│ ├── metrics ← /metrics: snapshot|diff|get
│ ├── keys ← virtual-key CRUD + hash
│ └── teams ← team CRUD
└── cases/
├── README.md ← case index
└── 01..12_*.md ← runbooks Claude executes
Configuration via e2e/.env (gitignored). Postgres is ephemeral by
design — every `proxy stop` wipes data so test runs are reproducible
and stale virtual keys can't poison later cases.
Initial case set (12 cases):
01-04: Anthropic prompt-cache metrics (5m/1h TTL, read, no-cache baseline)
05-06: OpenAI prompt-cache metrics (cached_tokens, no creation metric)
07: /metrics endpoint smoke
08-09: virtual-key + per-team label isolation (needs Postgres)
10: cost_breakdown must include cache_read_cost/cache_creation_cost
when the static model_cost entry has cache rates (GREEN guard)
11: spend_logs.error_information.error_message must not be silently
empty on auth failures — companion regression case for PR #2
12: custom_pricing path must not drop cache pricing when a
dashboard-added deployment lacks cache_*_input_token_cost in
litellm_params — currently RED, guards the future fix to
router.py:7237 (deployment-UUID entry merge)
Also documents the fork's branching strategy in CLAUDE.md: ship/v1.83.10
is the long-term ship branch (TAG + accumulated fix/* merges);
internal/v1.83.10-stable is the upstream-sync working branch (1700+
upstream commits, NOT to be used as fix base); litellm_internal_staging
is pure upstream tracking. All internal fix PRs target ship/v1.83.10.
.gitignore additions cover the rendered config (e2e/_config/
.litellm.rendered.yaml, produced by `proxy start` from .env values)
and tool pycache.
Test plan:
- Cases 01-07 verified GREEN against real Anthropic + OpenAI/GLM
providers (Bug #1 metrics fix)
- Case 11 GREEN after PR #2 merge (Bug #3 error_message fix)
- Case 10 GREEN against current ship state
- Case 12 deliberately RED — guards future router.py fix
- No changes to tests/test_litellm/ or any other ci-managed paths
Two related improvements to the e2e test harness, both driven by
real-prod observations of the corp Anthropic gateway behavior:
1. `e2e/tools/call` learns `--user-id`. Forwarded to the proxy as the
OpenAI `user` field, which litellm in turn maps to Anthropic's
`metadata.user_id`. The corp gateway at maasapi.* uses this field
for sticky upstream-key load balancing — requests sharing the
same user_id land on the same upstream API key, so the second
request's prefix can read the first request's cache write. Without
sticky routing, the gateway round-robins anonymous requests across
~10+ upstream accounts each with its own cache namespace, and
cache_read never hits in test scenarios.
This was misdiagnosed initially as "gateway doesn't support cache
reads". Prod logs show that requests carrying device_id in their
end_user blob do achieve cache_read>0 (e.g. spend_logs entries
e0ab6f96 and eec6a70d both hitting cache_read=128k-135k tokens
under the same b0cb8e4d... device_id). Without device_id, the
same gateway shows cache_read=0 on subsequent calls.
2. All Anthropic e2e cases (01, 02, 03, 04, 08, 09) now pass
`--user-id` so cache-related assertions are deterministic
regardless of upstream-account routing. Case 03 (cache READ) flips
from SKIP-on-miss to required GREEN now that cache_read can be
reliably triggered.
3. `e2e/tools/run-all-cases` ships as a first-class tool (was a
/tmp scratch script). One PASS/FAIL/SKIP line per case, summary
at the bottom, exits 0 iff every case PASSes (SKIPs allowed).
Per-case logic split into `case_NN()` functions; adding a new case
means dropping a fixture + adding one function + one invocation.
Fixes a metric-snapshotting bug in the old runner: the previous
`snap()` returned the LAST matching `/metrics` series, which silently
compared two different series across before/after snapshots once
later cases minted new virtual keys / teams. New `snap_sum()` sums
across all series matching the label selector, giving stable totals
even as the series set grows. Case 01 now passes cleanly through
the runner.
`--skip-paid` flag skips real-provider cases (05, 06, 13) for a
~$0 smoke pass against the harness itself.
4. `e2e/cases/data/11_error_information_message_populated.sh` polls
spend_logs up to 20s instead of a flat 2s sleep — async logger lag
was producing flake.
Test plan
- `e2e/tools/run-all-cases` → 13/13 PASS
- Case 03 cache_read deterministically hits 1827 tokens with --user-id
- Case 01 no longer false-FAIL when runner is invoked after case 09
has minted virtual keys/teams
- Black 24.10.0 formatting clean on e2e/tools/call
…rror Verifies the cherry-pick of BerriAI#26346 at the full-stack level: seed a key with `budget_limits` set, wait two ticks of the background `ResetBudgetJob.reset_budget_windows`, then grep the proxy container's logs for either the raw `prisma.errors.MissingRequiredValueError` exception or the "Failed to reset budget windows" wrapper line. Empty grep = PASS. The case is intentionally key-path only — the team path is symmetric and covered by unit tests (`test_reset_budget_windows_resets_expired_team_window`, `test_reset_budget_windows_query_error_does_not_break_team_path`). Seeding a team via `/team/new` with `budget_limits` hits an unrelated Prisma serialization bug in the team endpoint, which would mask the result of this regression check. Companion changes that the case relies on: - `e2e/_config/docker-compose.yml`: pin `PROXY_BUDGET_RESCHEDULER_MIN_TIME=10` / `PROXY_BUDGET_RESCHEDULER_MAX_TIME=15`. Upstream default is ~600s (10 min) which makes the case impossible to verify inside a sane observation window; for dev/e2e there's no production reason to wait that long. The fixture skips (exit 77, not fail) if a stale container still has the upstream default — so an out-of-date environment doesn't masquerade as a regression. - `e2e/tools/proxy`: split rebuild into two commands. `build` is the cached path (30-90s, default for source-only edits); `rebuild` keeps the original `--no-cache` semantics (3-5 min, for Dockerfile / dep changes). The cached path was always implicitly possible via `docker compose build`, but only `--no-cache` was surfaced through the tool, forcing a full rebuild on every fix-and-verify cycle. - `CLAUDE.md`: document that root-level `e2e/` is the project's Claude-driven end-to-end harness, and that full-stack regression verification (DB schema, background jobs, real HTTP flow) belongs under `e2e/cases/` rather than under `tests/`.
This was referenced Jun 4, 2026
songkuan-zheng
added a commit
that referenced
this pull request
Jun 4, 2026
…ning (#55) Ports the current state of ` CLAUDE.md`, ` scripts/upstream-sync-check.sh`, and ` .github/pull_request_template.md` from ship/v1.83.10 to ship/v1.87.0, and flips the Current pinning block to point at v1.87.0. Before this PR the policy doc was orphaned: Wave 2 (#47) cherry-picked an OLD version of CLAUDE.md from f67766f (when our policy doc was a 50-line skeleton). PRs #43, #45 added the Tier classification, Current pinning, and Upstream sync cadence sections on ship/v1.83.10, but they never crossed over to ship/v1.87.0. PR #44 added ` scripts/upstream-sync-check.sh` on ship/v1.83.10 only. After this PR, ship/v1.87.0 has the full policy state, and the sync script auto-discovers ` v1.87.0` from CLAUDE.md (verified locally — reports 100 commits on upstream/main since the new pin, 0 missed backports, 0 newer minor lines). ## Changes - ` CLAUDE.md` - Current pinning block flipped to v1.87.0 (Upstream pin, Ship branch, Upstream-sync branch, Internal release tag pattern). - Branching strategy table examples updated to ` v1.87.0` / ` ship/v1.87.0` / ` internal/v1.87.0-stable`. The pin tag column notes the historical ` vX.Y.Z-stable` form (used pre-1.84.0) so older grep results still resolve. - Cutting an internal release example updated to ` v1.87.0-internal.1` (N restarts at 1 under the new pin per the release-tag.sh contract). - Upstream sync cadence example covers both the new ` vX.Y.W` and the legacy ` .patch.N` patch-tag conventions. - ` git tag -l 'v1.87.0-internal.*'` for the latest-release lookup. - ` scripts/upstream-sync-check.sh` — copied verbatim from ship/v1.83.10. Re-reads the new pin without any code change. - ` .github/pull_request_template.md` — copied verbatim from ship/v1.83.10 (Tier classification checkbox section). Tier: B (internal policy / branding doc; not pushed upstream).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier classification
litellm_extras/only)litellm/corelitellm/coreIf Tier C or D, did you try upstream first?
Summary
Second wave of the v1.87.0 bump. Ports the core e2e/ test harness (Claude-
driven, real-provider runbooks) onto
ship/v1.87.0.Scope intentionally narrower than original Wave 2 plan: ships the
base harness + sticky routing + case 16 only. The in-network mock
provider (
7e43f094f6+713dfd859e) is deferred to a follow-up PRbecause it touches case-index lines for cases 13-22 that we're porting
in later waves (avoids cascading conflicts).
Cherry-picks (chronological, 3 commits)
ffffa7906a07e65a92fd3675d2c8c720bbcc0b22(AsyncMock/MagicMockimport for case 16 tests) was ano-op against v1.87.0 — that import is already present in upstream's
tests/test_litellm/proxy/common_utils/test_reset_budget_job.py.Skipped.
What's in this PR
e2e/README.md,e2e/.env.example,e2e/_config/docker-compose.yml—harness contract docs + container topology.
e2e/tools/proxy|call|metrics|keys|teams|run-all-cases— Unix-stylesingle-purpose CLIs.
e2e/cases/01..12, 16— 13 runbook scenarios (Prometheus prompt-cachemetrics, cost breakdown, error_information, custom pricing, budget
reset regression).
e2e/cases/data/*— fixtures for cases 03, 10, 11, 12, 16.CLAUDE.md— early e2e section. Will be fully overwritten by theCLAUDE.md migration wave (which ports our current PR-docs(claude): version-agnostic refs + Tier classification + sync cadence #43/docs(claude): reframe internal/<pin>-stable as opt-in bump sandbox #45 policy
doc); the content here is just placeholder.
.gitignore— e2e/.env, .litellm.rendered.yaml etc.Conflict resolution
3675d2c8c7(case 16) had conflicts ine2e/cases/README.mdande2e/tools/run-all-casesbecause the original commit expected cases13/14/15 to already exist as context. Resolved by accepting only case
16's entries; cases 13/14/15 will be added back in their feature waves.
Deferred to Wave 2.5
7e43f094f6feat(e2e): in-network mock provider (feat(e2e): in-network mock provider (OpenAI + Anthropic + Gemini) #34)713dfd859efix(e2e): mock-only preflight, case 23 fixture, mock providers in tools/call (fix(e2e): mock-only preflight, case 23 fixture, mock providers in tools/call #36)Reason: the mock provider PR's README/runner edits assume cases 13-22
are present at port time. Porting it now without those cases produces
a chain of conflicts. Will port together with one of the case-coupled
feature waves (likely after Wave 3-4).
Verification
```bash
e2e/tools/run-all-cases --help # works
e2e/tools/proxy --help # works
ls e2e/cases/ # 13 .md files (01-12, 16)
```
The harness can be exercised end-to-end without the mock provider — it
just means
--mock-onlymode is unavailable until Wave 2.5.Pre-Submission checklist
litellm/core code touched (test/infra only).make test-unitunaffected.Type
🚄 Infrastructure / ✅ Test