test(e2e): add weekly session-anomaly load test against real providers

[mateo-berri]

Relevant issues

Linear ticket

Resolves LIT-4562

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Green-path proof captured at commit 48295df against a live proxy hitting the real Anthropic API and real Bedrock Invoke (no mocks; a full run costs roughly $0.70 of real provider spend). The run takes ~3 minutes because the spend gate waits for the recorded total to hold stable for 75s before trusting it

Boot the proxy and run the weekly test:

$ litellm --config tests/e2e/load/weekly_anomaly_config.yml --port 39441
$ LITELLM_PROXY_URL=http://localhost:39441 E2E_WEEKLY_ANOMALY=1 \
    pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py -v -rA

tests/e2e/load/test_weekly_session_anomaly_e2e.py::TestWeeklySessionAnomaly::test_session_load_stays_within_baselines[anthropic] PASSED [ 50%]
tests/e2e/load/test_weekly_session_anomaly_e2e.py::TestWeeklySessionAnomaly::test_session_load_stays_within_baselines[bedrock_invoke] PASSED [100%]

anthropic anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=367210, warm_cache_creation_tokens=58445, p95_turn_seconds=2.6713846249040216)
bedrock_invoke anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=375432, warm_cache_creation_tokens=59680, p95_turn_seconds=2.8544898750260472)

======================== 2 passed in 198.52s (0:03:18) =========================

Re-ran the same green path at 1255094 (per-turn retries threaded through the driver, cron moved to Saturday), fresh proxy and database, real provider spend again:

$ LITELLM_PROXY_URL=http://localhost:29980 E2E_WEEKLY_ANOMALY=1 \
    pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py -v -rA

anthropic anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=364472, warm_cache_creation_tokens=58011, p95_turn_seconds=2.3629595418460667)
bedrock_invoke anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=362048, warm_cache_creation_tokens=57767, p95_turn_seconds=2.9469174998812377)

======================== 2 passed in 191.25s (0:03:11) =========================

Bugbot then caught that the pure harness tests could not actually run without a proxy: a session-scoped autouse fixture in the load conftest registered the throughput mock model for every test collected in the package. At this PR's head 33fb380 that fixture is wired into load_key instead, the documented no-proxy run passes, and the throughput and weekly cells were re-run live at the same commit and stayed green

$ LITELLM_PROXY_URL=http://localhost:59999 pytest tests/e2e/load/test_session_anomaly.py -q   # dead port, no proxy

8 errors in 0.09s    # at 1255094de3, every case errors in _ensure_load_model setup
8 passed in 0.01s    # at 33fb38056d

$ LITELLM_PROXY_URL=http://localhost:37234 pytest tests/e2e/load/test_chat_completions_throughput_e2e.py -q
1 passed in 65.79s (0:01:05)

$ LITELLM_PROXY_URL=http://localhost:37234 E2E_WEEKLY_ANOMALY=1 \
    pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py -v -rA

anthropic anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=378197, warm_cache_creation_tokens=60114, p95_turn_seconds=2.9170170000288635)
bedrock_invoke anomaly report: AnomalyReport(planned_turns=36, attempted_turns=36, failed_turns=0, warm_turns=30, warm_uncached_input_tokens=60, warm_cache_read_tokens=370226, warm_cache_creation_tokens=58999, p95_turn_seconds=2.642204290954396)

======================== 2 passed in 190.82s (0:03:10) =========================

The test also discriminates the exact incident it exists to catch. Each checkpoint below is a live proxy booted from a detached worktree at that commit with the same weekly config and a fresh database, driven by this PR's test with real provider traffic. Percentages are the warm cache read share against the 65% floor. The regression PR #31364 only touched Bedrock Invoke, so the anthropic route staying green at the buggy checkpoints is the correct outcome

Checkpoint	Proxy commit	Expected	anthropic	bedrock_invoke
Before regression PR #31364	80d3b69	test passes	PASS, 86.2%	PASS, 86.3%
After regression PR #31364	64d8d7f	test fails	PASS, 86.2%	FAIL, 45.5%
Before fix PR #32578	ab53d06	test fails	PASS	FAIL, 45.4%
After fix PRs	48295df	test passes	PASS, 86.3%	PASS, 86.3%

At the failing checkpoints the failure is exactly the collapse signature: warm cache creation balloons (238,865 and 225,514 tokens against ~59,000 healthy) while reads halve, because the hoisted mid-conversation system entries mutate the cached prefix every turn. The old checkpoints ran with LITELLM_LOCAL_MODEL_COST_MAP=False since packaged cost maps before Jul 1 lack claude-sonnet-5 pricing; booting the Jun 27 proxy with the pinned local map instead made the gateway record $0.00 for a full paid run and the spend gate flagged that too, which is the spend-tracking-dead anomaly working as intended

The gates also trip on demand. At the earlier commit b572cb8, forcing a low spend ceiling through the env override:

$ LITELLM_PROXY_URL=http://localhost:34354 E2E_WEEKLY_ANOMALY=1 \
    E2E_ANOMALY_MAX_KEY_SPEND_USD=0.10 \
    pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py -k anthropic -q

E       AssertionError: anthropic: gateway recorded $0.3379 for 36 turns, above the $0.1 ceiling; spend per session is anomalously high (cache regressions surface here as 2-3x spend)
E       assert 0.3378937000000001 <= 0.1
1 failed, 1 deselected in 20.86s

During development the error-rate gate also tripped for real: an earlier iteration placed the mid-conversation system reminder after the assistant turn, both providers rejected it with a 400, and the test went red with bedrock_invoke: 6/12 turns failed (50.0% > 5.0% allowed)

A regular e2e run cannot spend this money by accident; without the opt-in env the cells deselect:

$ pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py --collect-only -q
no tests collected (2 deselected) in 0.01s

Type

🚄 Infrastructure
✅ Test

Changes

This is the weekly anomaly load test from the mid-conversation-system cache-collapse RCA: a customer's cache hit rate fell from ~90% to 25-45% and daily spend rose 2-3x for a week before a human noticed. The existing Locust throughput test could not have caught it because it drives a mocked model with single-turn pings, and cache read/write, spend, and turn-time anomalies only exist against real providers under realistic multi-turn traffic

The new test (tests/e2e/load/test_weekly_session_anomaly_e2e.py, driver in session_anomaly.py) runs 6 concurrent Claude Code shaped sessions of 6 turns each per route, parametrized over anthropic/claude-sonnet-5 and bedrock/invoke/us.anthropic.claude-sonnet-5 (both registered through /model/new and torn down after). Each session grows a real conversation: a unique cacheable system prompt above the 1024-token minimum, a cache_control breakpoint that moves to the latest user turn, and a role: "system" <system-reminder> attached after every user turn, mirroring what Claude Code sends mid-session. Each turn's request retries transient non-success results up to 3 attempts (E2E_ANOMALY_TURN_ATTEMPTS, 2s pause between attempts), so a single provider blip cannot fail the weekly run; recorded turn latency spans all attempts, so retry storms still surface in the p95 gate, and a turn that stays failed after its attempts still aborts its session. Per turn it records latency, usage token splits, and failures; per route it asserts the anomaly ceilings: error ratio at most 5% of planned turns (a session that aborts early counts every unattempted turn as a failure, so a lost session cannot hide by shrinking the denominator), warm-turn cache read share at least 65% of billed input tokens (healthy measures ~86% at every healthy checkpoint; the regression reproduced live measures ~45%, so the floor sits at the midpoint), non-zero warm cache writes, p95 turn time at most 30s, and gateway-recorded key spend non-zero, at most $0.60 (healthy measures ~$0.35; the RCA regression was 2-3x), and only trusted once it has held stable for 75s, longer than a spend batch-flush interval, so a partial total sitting between asynchronous flushes is not mistaken for final. Every threshold is overridable through E2E_ANOMALY_* env vars in e2e_config.py. The accounting and retry rules are pinned by pure harness regression tests in tests/e2e/load/test_session_anomaly.py, which carry no e2e marker and run without a proxy; to keep that true, the load suite's mock-model registration fixture hangs off load_key rather than being package-wide autouse

The test is marked e2e, load, and the new weekly marker; the load-suite conftest deselects weekly cells unless E2E_WEEKLY_ANOMALY is set, so ordinary suite runs never spend this money. .github/workflows/weekly_load_anomaly.yml runs it every Saturday 12:00 UTC (04:00 or 05:00 Pacific depending on DST, plus workflow_dispatch), early Saturday morning so a red run lands before the Saturday stable release cut. The cadence stays weekly rather than joining the 6-hourly e2e schedule because the load intensity is expected to grow. A red run is the anomaly alert. The workflow needs ANTHROPIC_API_KEY and AWS_BEARER_TOKEN_BEDROCK repo secrets; zizmor reports no findings on it

A coverage-registry row reliability.perf.session_anomaly.under_slo is added and the grammar/suite docs in tests/e2e/CLAUDE.md are updated to match

QA runbook

• tests/e2e/load/test_weekly_session_anomaly_e2e.py::TestWeeklySessionAnomaly::test_session_load_stays_within_baselines[anthropic] (and [bedrock_invoke], identical flow) - concurrent Claude Code shaped multi-turn sessions against a real provider stay within the error, cache, latency, and spend baselines

  • Boot a proxy with a DB: litellm --config tests/e2e/load/weekly_anomaly_config.yml --port 4000 (needs ANTHROPIC_API_KEY, AWS_BEARER_TOKEN_BEDROCK, DATABASE_URL, LITELLM_MASTER_KEY in the env)
  • Register the deployment: curl -X POST http://localhost:4000/model/new -H "Authorization: Bearer sk-1234" -d '{"model_name": "qa-anomaly", "litellm_params": {"model": "anthropic/claude-sonnet-5"}}' (for the bedrock cell use "model": "bedrock/invoke/us.anthropic.claude-sonnet-5", "aws_region_name": "us-east-1")
  • Create a scoped key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"models": ["qa-anomaly"]}'
  • Drive a session of /v1/messages calls with that key: a system array holding one large unique text block with "cache_control": {"type": "ephemeral"}, and on each turn the full prior conversation plus a fresh user turn carrying the cache_control breakpoint, with a {"role": "system", "content": [{"type": "text", "text": "<system-reminder>...</system-reminder>"}]} turn after every user turn
  • Expect every turn to return 200 in a few seconds, and from turn 2 on expect usage.cache_read_input_tokens to cover most billed input tokens (~86% across warm turns) with non-zero usage.cache_creation_input_tokens
  • Poll GET /key/info?key=<key> until spend is non-zero and has not changed for at least 75 seconds; for the test's 36 turns per route expect roughly $0.35, under the $0.60 ceiling
  • Run pytest tests/e2e/load/test_weekly_session_anomaly_e2e.py --collect-only -q without E2E_WEEKLY_ANOMALY and expect both cells deselected
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/load/test_session_anomaly.py - pure harness coverage (no proxy, no e2e marker): a session aborted on its first turn counts all of its planned turns as failures in the error ratio, a spend total that pauses between asynchronous batch flushes is not accepted until it has been stable for the full settle window, and per-turn retries return the first success, stop at the attempt cap returning the last failure, and never fire after a first-try success

  • Run pytest tests/e2e/load/test_session_anomaly.py -q with no proxy running and expect 8 passed
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Changes are confined to e2e tests, CI workflow, and documentation; production proxy behavior is unchanged, and paid provider traffic only runs when explicitly opted in or via the scheduled workflow.

Overview
Introduces a weekly session-anomaly perf check aimed at catching mid-conversation prompt-cache regressions (high error rate, collapsed cache reads, slow turns, or runaway gateway spend) that single-turn mock load tests cannot see.

The new driver in session_anomaly.py runs concurrent multi-turn /v1/messages sessions with cacheable system prefixes, moving user-turn cache breakpoints, and mid-session system-reminder turns—mirroring Claude Code traffic. It records per-turn latency and token usage, retries transient failures, and aggregates error ratio, warm-turn cache read share, p95 turn time, and settled key spend against E2E_ANOMALY_* thresholds. Pure harness tests in test_session_anomaly.py pin accounting and retry/settle behavior without a proxy.

Live coverage lives in test_weekly_session_anomaly_e2e.py (Anthropic + Bedrock Invoke), marked weekly and deselected unless E2E_WEEKLY_ANOMALY=1. .github/workflows/weekly_load_anomaly.yml boots Postgres, starts the proxy with weekly_anomaly_config.yml, and runs that test on a Saturday cron (with workflow_dispatch). The load suite’s mock-model setup is no longer package-wide autouse—it hangs off load_key so unit tests in the package can run proxy-free. Docs, pytest markers, and reliability.perf.session_anomaly.under_slo registry coverage are updated to match.

^{Reviewed by Cursor Bugbot for commit 33fb380. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds an opt-in weekly load test for multi-turn session anomalies against real providers. The main changes are:

• Concurrent Anthropic and Bedrock session traffic with retries
• Checks for errors, prompt-cache behavior, latency, and settled key spend
• Pure harness tests for aborted sessions, retries, and delayed spend updates
• A scheduled workflow with explicit weekly opt-in
• Scoped mock-model setup so pure harness tests run without a proxy

Confidence Score: 5/5

This looks safe to merge.

• Aborted sessions now count every uncompleted planned turn as failed.
• Spend must remain unchanged and nonzero for the configured settlement period before evaluation.
• Focused harness tests cover both fixes.
• No blocking issues were found in the updated code.

Important Files Changed

Filename	Overview
tests/e2e/load/session_anomaly.py	Adds the concurrent session driver, retry handling, planned-turn accounting, anomaly aggregation, and spend settlement.
tests/e2e/load/test_session_anomaly.py	Covers aborted-session accounting, retry limits, and delayed spend updates without requiring a proxy.
tests/e2e/load/test_weekly_session_anomaly_e2e.py	Runs the real-provider session test and checks cache use, errors, latency, and final key spend.
tests/e2e/load/conftest.py	Limits mock-model registration to tests using the load key and deselects weekly tests unless enabled.
.github/workflows/weekly_load_anomaly.yml	Schedules the weekly test and provisions its database, proxy, dependencies, and provider credentials.

_{Reviews (5): Last reviewed commit: "fix(e2e): register the load mock model t..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit_4562_weekly_anomaly_load_test (33fb380) with litellm_internal_staging (a780d4e)}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Unit tests require live proxy
  • The autouse _ensure_load_model fixture now short-circuits when no e2e-marked tests are collected in the session, so pure harness tests in test_session_anomaly.py no longer trigger proxy calls during setup.

Or push these changes by commenting:

@cursor push cab0e972de

Preview (cab0e972de)

diff --git a/tests/e2e/load/conftest.py b/tests/e2e/load/conftest.py
--- a/tests/e2e/load/conftest.py
+++ b/tests/e2e/load/conftest.py
@@ -53,8 +53,12 @@
 
 @pytest.fixture(scope="session", autouse=True)
 def _ensure_load_model(  # pyright: ignore[reportUnusedFunction]  # pytest autouse session fixture, wired by name
+    request: pytest.FixtureRequest,
     client: LoadClient,
 ) -> Iterator[None]:
+    if not any(item.get_closest_marker("e2e") is not None for item in request.session.items):
+        yield
+        return
     proxy = client.proxy
     if _model_is_servable(proxy, LOAD_MODEL):
         yield

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 33fb380. Configure here.}