test(e2e): harness fixes for stage job green (skips + router/UI/budget)

[mubashir1osmani]

Relevant issues

Stage e2e was 24 failed / 226 passed. This PR targets pragmatic job green (0 failed, skips allowed), not full product green on every Claude/router cell

Linear ticket

Product bugs tracked separately (do not fix in this PR):

• LIT-4521 complexity classifier SIMPLE on short hard prompts
• LIT-4522 Bedrock Invoke /v1/messages tool_search type normalize
• LIT-4523 Bedrock Converse document requires text block
• LIT-4524 Bedrock Converse stream Content block is not a text block

Plan A (pragmatic job green)

1. Merge this PR (test(e2e): harness fixes for stage job green (skips + router/UI/budget) #33634)
2. Keep Linear product bugs skipped until Mateo ships LIT-4521 through LIT-4524
3. After the next stage e2e run, fix or skip remaining non-Linear reds (DataDog delivery, management control-plane 500s, custom pricing /model/new 500)
4. Success criteria: stage e2e 0 failed (skips allowed)

What this PR does not claim: product cells that are skipped are not fixed; re-enable those tests only after live proof on LIT tickets

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

E2e-only harness changes (no product path under litellm/). Baseline stage fail list from e2e pod log Logs-2026-07-16 18_08_57.txt (24 failed)

After merge, re-run stage e2e and expect:

• harness paths green (complexity registration when not skipped for LIT-4521, create-key UI, multi-window budget on gpt-5.5)
• known product/config cells skipped rather than failed
• remaining non-Linear reds (if any) handled in a follow-up under plan A step 3

Type

Test

Changes

Scope is tests/e2e/** only

Harness fixes:

• complexity-smart-router session fixture: Gateway.create_model + data-plane list + chat callability probe; test key scoped to router + tier backends
• create-key UI: open via + Create New Key (no flaky /ui/api-keys/?create=true navigation race)
• multi-window budget: poll gpt-5.5 with max_tokens=1 so the reset wait measures budget windows instead of Claude latency / ALB idle timeout; removed ALB 502 skip masking

Skips (stage red / product / config) until owners unskip:

• Linear product: LIT-4521 (complexity semantic tier), LIT-4522 (tool_search bedrock invoke), LIT-4523 (pdf bedrock converse), LIT-4524 (thinking bedrock converse)
• Claude matrix also skipped where stage is red without a Linear ticket yet: long_context_1m x5, tool_search azure/vertex, count_tokens vertex, passthrough azure

Not in this PR:

• product fixes under litellm/
• management control-plane write retries (removed; does not fix root cause)
• DataDog / custom pricing / management 500 infra (plan A step 3 after next stage run)

QA runbook

• tests/e2e/router/conftest.py + test_complexity_router_e2e.py - registers complexity-smart-router when missing; LIT-4521 may leave the semantic-tier assertion skipped until Mateo lands

  • On a proxy without static complexity-smart-router, suite setup registers the model and /v1/models lists it
  • Chat with a key that includes complexity-smart-router is not Invalid model name when the fixture completed
  • Sanity check: registration failure fails setup with a clear registration error

• tests/e2e/management/test_key_models_dropdown_e2e.py::TestKeyModelsDropdownUI::test_create_teamless_key_offers_proxy_scope_and_persists

  • Open /ui/api-keys/, click + Create New Key, modal appears without ?create=true navigation abort
  • Models offers All Proxy Models and gpt-5.5; not All Team Models
  • Create key; /key/info shows models all-proxy-models and no team
  • Sanity check: product path is the button, not deep-link create=true

• tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py::test_short_window_blocks_then_resets

  • Key with 30s tiny window + roomy 1m window on gpt-5.5
  • Exhaust short window to budget_exceeded
  • After reset, chat succeeds again with max_tokens=1
  • Sanity check: non-budget errors still fail; we do not treat ALB 502 as success

• Skipped claude_code / Linear cells

  • pytest collection shows skip (not fail) for LIT-4522/23/24 cells and the other stage-red skips listed under Changes
  • Sanity check: re-enable only after the matching product ticket or stage config is fixed

Final Attestation

• For the harness paths this PR owns, tests check the right things; product gaps are skipped and tracked on Linear rather than papered over with retries

---

Note

Low Risk
Changes are confined to e2e tests and documentation; they adjust CI/stage behavior via skips and fixture stability without modifying runtime proxy or product logic.

Overview
Targets stage e2e 0 failed (skips allowed) with tests/e2e only changes—no product code under litellm/.

Harness fixes: The complexity router session setup now uses Gateway.create_model, verifies the virtual model with a real chat probe (not just /v1/models), and adds a complexity_key fixture scoped to the router plus tier backends; spend-log assertions accept alias vs provider-prefixed model names. The create-key Playwright flow lands on /ui/api-keys/ and clicks + Create New Key instead of the flaky ?create=true deep link. Multi-window budget polling uses CHEAP_OPENAI_MODEL with max_tokens=1, a tiny short-window cap, and stricter non-budget error reporting so ALB 502s are not mistaken for budget behavior.

Skips: Stage-red or Linear-tracked Claude matrix cells are marked @pytest.mark.skip (long_context_1m across providers, Vertex count_tokens/tool_search, Azure passthrough/tool_search, Bedrock Invoke tool_search, Bedrock Converse PDF/thinking, complexity LLM classifier LIT-4521) with reasons pointing at product/config gaps.

Docs: tests/e2e/CLAUDE.md notes harness unit trees under claude_code and that pure harness tests need no @pytest.mark.e2e; coverage registry README drops the E2E_RESULT vs coverage blurb.

^{Reviewed by Cursor Bugbot for commit e11b760. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR targets a pragmatic stage e2e green (0 failed, skips allowed) by fixing three harness regressions and skipping known product/config failures tracked on Linear. All changes are confined to tests/e2e/; no production code under litellm/ is touched.

• Harness fixes: _router_is_callable now probes the virtual router with a real chat call (using isinstance(result, Success)) instead of relying on /v1/models list alone, resolving the reload-race false-positive; the create-key UI flow navigates to /ui/api-keys/ and clicks the button rather than using the flaky ?create=true deep-link; the multi-window budget test drops the cap to 1e-9 with max_tokens=1 on a cheap OpenAI model so exhaustion is guaranteed within the first two calls and ALB idle-timeout 502s can no longer be mistaken for budget signals.
• Skips: Eleven stage-red or Linear-tracked cells are marked @pytest.mark.skip with explicit reasons and ticket references (LIT-4521 through LIT-4524), covering long-context 1M across providers, tool-search on Azure/Vertex/Bedrock Invoke, Bedrock Converse PDF/thinking, Vertex count-tokens, and Azure passthrough.

Confidence Score: 5/5

Safe to merge — all changes are in the test harness only, with no production code modified.

Every change is scoped to tests/e2e/. The three harness fixes correctly address previously identified weaknesses (list-only servability probe, flaky SPA deep-link, insufficiently tight budget cap). Skips are clearly documented with Linear ticket references or stage-red explanations. No regressions are introduced in production paths.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/router/conftest.py	Replaced the list-only servability check with a real chat probe (_router_is_callable) and switched from the bespoke /model/new call to Gateway.create_model; adds a complexity_key fixture for per-test scoped keys. Addressed all three concerns from the previous review thread.
tests/e2e/router/test_complexity_router_e2e.py	Swapped scoped_key for the new complexity_key fixture, changed to len(served)==1 and served[0] in LLM_TIER_MODELS (exact-count + membership guard), updated the probe prompt, and skips the test pending LIT-4521. Previous assertion-weakening concern is resolved by the explicit length check.
tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py	Switches model to CHEAP_OPENAI_MODEL with max_tokens=1 and lowers the tight-window cap to 1e-9 — any real call spend exceeds the cap, guaranteeing exhaustion within the first two iterations and removing the ALB-timeout false-positive risk.
tests/e2e/management/test_key_models_dropdown_e2e.py	Replaces the flaky ?create=true deep-link navigation with a two-step flow: land on /ui/api-keys/ then click the "+ Create New Key" button. Avoids SPA auth-redirect races on stage.
tests/e2e/claude_code/tool_search/test_bedrock_invoke.py	Skips with LIT-4522 product bug reference; skip reason is specific and actionable.
tests/e2e/claude_code/pdf_input/test_bedrock_converse.py	Skips with LIT-4523 product bug reference; skip reason is specific and actionable.
tests/e2e/claude_code/thinking/test_bedrock_converse.py	Skips with LIT-4524 product bug reference; skip reason is specific and actionable.
tests/e2e/claude_code/long_context_1m/test_anthropic.py	Adds @pytest.mark.skip with a clear stage-red reason; no functional changes.
tests/e2e/coverage_registry/README.md	Drops the E2E_RESULT vs coverage blurb; doc-only change, no code impact.
tests/e2e/CLAUDE.md	Notes harness-unit-test trees under claude_code/ and that pure harness tests carry no @pytest.mark.e2e; doc-only update.

_{Reviews (2): Last reviewed commit: "test(e2e): use a tradeoff-decision promp..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_harness_fixes (90bebc4) with litellm_internal_staging (0223383)}

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e11b760. Configure here.}