test(e2e): harness fixes for stage failures (long_context, router, UI, retries) - #33630
Conversation
… unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests
Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps
Keep management, router, budget, and shared conftest harness fixes only
Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines
Greptile SummaryThis PR hardens the e2e test harness against 24 failures observed in a stage run, with changes scoped entirely to
Confidence Score: 4/5Safe to merge — all changes are confined to tests/e2e/ with no production code touched. The transient-error retry logic treats an empty response body as retryable, which could mask real proxy crash-loops. The tests/e2e/management/management_client.py (retry-condition inconsistency and empty-body suppression) and tests/e2e/conftest.py (removal of structured result logging).
|
| Filename | Overview |
|---|---|
| tests/e2e/management/management_client.py | Adds _is_transient_control_plane_error helper and retry loops for write operations; minor inconsistency between llm_only_key retry patterns and the shared helper. |
| tests/e2e/router/conftest.py | Replaces list-only wait with Gateway.create_model plus a live chat probe; adds complexity_key per-test fixture with deferred cleanup. |
| tests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py | Treats HTTP 502/503/504 during the reset-wait loop as transient ALB noise; result.status_code access is safe since StreamingResponse always carries it. |
| tests/e2e/router/test_complexity_router_e2e.py | Broadens tier model constants to frozensets; switches to complexity_key fixture; assertion is now set-membership rather than exact-string equality. |
| tests/e2e/claude_code/test_config.yaml | Replaces sonnet-4-5 with sonnet-4-6 entries for long_context_1m; removes redundant anthropic-beta static header from all model configs. |
| tests/e2e/claude_code/count_tokens/test_vertex_ai.py | Adds _is_upstream_token_count_unsupported to classify Vertex capability gaps as not_applicable; real proxy errors still fail. |
| tests/e2e/management/test_key_models_dropdown_e2e.py | Replaces ?create=true navigation with button-click modal open to avoid SPA auth redirect race on stage. |
| tests/e2e/conftest.py | Removes pytest_runtest_makereport hook and e2e_result_reporter import, dropping structured Loki/Grafana result logging. |
| tests/e2e/test_e2e_gateway.py | New unit tests for Gateway.create_model, delete_model, and spend_logs_window using a typed fake transport. |
| tests/e2e/test_lifecycle.py | New unit test verifying run_case releases partially-registered resources when init() fails partway through. |
| tests/e2e/test_transport.py | New unit tests for is_control_plane_path routing logic. |
| tests/e2e/claude_code/_pr_gate_unit_tests/test_bash_tool_restrictions.py | Security pin tests enforcing exact-match Bash(echo pong) allow rules and dontAsk permission mode on all Bash-using cells. |
Comments Outside Diff (1)
-
tests/e2e/conftest.py, line 85 (link)pytest_runtest_makereporthook removed — E2E_RESULT logfmt lines no longer emittedThe hook that printed one structured
E2E_RESULTlogfmt line per finished test (consumed by Loki for the Grafana status-history panels) has been removed alongside thee2e_result_reporterimport. The coverage_registry README change documents the intent, but the monitoring effect is that stage pass/fail history panels will stop receiving per-test result events. Was this removed because the reporter was causing stage failures, or is the Loki pipeline being retired in favour of a different signal?Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time! Was
pytest_runtest_makereportremoved becausee2e_result_reporterwas itself causing stage failures, or is the Loki/Grafana result-history pipeline intentionally being retired?
Reviews (1): Last reviewed commit: "test(e2e): restore E2E_RESULT pytest_run..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| return False | ||
| lowered = result.body.lower() | ||
| return ( | ||
| "all connection attempts failed" in lowered | ||
| or "internal server error" in lowered | ||
| or "connecting to redis" in lowered | ||
| or "name resolution" in lowered | ||
| or result.body.strip() == "" | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
result.body.strip() == "" will retry any 500/502/503/504 that returns an empty body, including cases where the proxy genuinely crashed or returned no payload due to a misconfiguration. On stage, an empty body is usually an ALB timeout, but the same condition can mask real proxy startup failures where the container is OOMKilled and the ALB returns 502 with an empty body immediately. Consider requiring at least one of the four string patterns to be present before retrying, or applying the empty-body shortcut only to 502/503 (ALB errors) and not to 500 (proxy own errors).
| def llm_only_key(self) -> str: | ||
| return self.gateway.generate_key(KeyGenerateBody(models=[], allowed_routes=["llm_api_routes"])) | ||
| last_error: str | None = None | ||
| for attempt in range(_TRANSIENT_WRITE_ATTEMPTS): | ||
| try: | ||
| return self.gateway.generate_key( | ||
| KeyGenerateBody(models=[], allowed_routes=["llm_api_routes"]) | ||
| ) | ||
| except AssertionError as exc: | ||
| last_error = str(exc) | ||
| message = last_error.lower() | ||
| if ( | ||
| "all connection attempts failed" in message | ||
| or "internal server error" in message | ||
| ) and attempt + 1 < _TRANSIENT_WRITE_ATTEMPTS: | ||
| time.sleep(0.5 * (attempt + 1)) | ||
| continue | ||
| raise | ||
| raise AssertionError(last_error or "llm_only_key failed after retries") | ||
|
|
||
| def update_key_models(self, key: str, models: list[str]) -> None: | ||
| last: Result[NoBody] | None = None | ||
| for attempt in range(5): | ||
| for attempt in range(_TRANSIENT_WRITE_ATTEMPTS): |
There was a problem hiding this comment.
llm_only_key retry conditions are narrower than _is_transient_control_plane_error
The inline retry in llm_only_key only checks for "all connection attempts failed" and "internal server error" in the raised AssertionError message, while _is_transient_control_plane_error also covers "connecting to redis" and "name resolution". If gateway.generate_key raises with a message like "connecting to redis", llm_only_key will re-raise immediately instead of retrying, even though the equivalent write operation in create_team/create_user/create_org would retry on the same string.
There was a problem hiding this comment.
dont need transient failures, need to rate limit calls to model. fixed the code
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Transient 500 retries do not fix the underlying control plane failures
…latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s
|
Superseded by same-repo PR #33634 (branch on BerriAI/litellm). |
1 participant
Relevant issues
Hardens the e2e harness against failures seen on the stage run (24 failed / 226 passed). Product bugs found during that triage are tracked separately as LIT-4521 through LIT-4524 and are not fixed in this PR
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
E2e-only harness changes (no product path). Stage failure list and assertions were taken from the e2e pod log
Logs-2026-07-16 18_08_57.txt(24 failed). After merge, re-run the stage e2e job and expect green (or clearer failures) for:Product-side reds (Bedrock tool_search type normalize, Converse document+text, Converse streaming text block, complexity classifier SIMPLE quality) stay product tickets LIT-4522, LIT-4523, LIT-4524, LIT-4521
Type
✅ Test
Changes
Scope is
tests/e2e/**only; nothing underlitellm/Complexity router session fixture registers
complexity-smart-routerwithGateway.create_model, waits for data-plane list, then probes a real chat so list-only false positives cannot pass setup. The live test uses a key scoped to the virtual router and both tier backendsKey models dropdown create path no longer navigates to
?create=true(ALB/auth often interrupts that goto). It lands on the api-keys list, waits for+ Create New Key, and opens the modal from the buttonManagement create_team / create_user / create_org / llm_only_key retry transient control-plane 500/502/503 (
All connection attempts failed, internal server error, redis/DNS blips)Multi-window budget reset wait treats ALB 502/503/504 as transient instead of hard-failing as non-budget errors
Claude Code long_context_1m models move from sonnet-4-5 (200k) to sonnet-4-6 (1M) for Anthropic, Azure, Bedrock Invoke/Converse, and Vertex;
test_config.yamladds the matching aliasesVertex count_tokens rows that get upstream
not supported for token countingare recorded asnot_applicableinstead of failing the celltool_search model lists drop Azure Haiku (workspace tool_search_server gap) and Bedrock Invoke Opus 4.7 (unsupported); Bedrock Invoke probes Sonnet 4.5 only until product LIT-4522 lands
Adds unit coverage for the e2e gateway, lifecycle, transport, and Claude Code matrix builder / CLI driver / PR-gate helpers so harness regressions fail offline without a proxy
QA runbook
tests/e2e/router/test_complexity_router_e2e.py::TestComplexityRouterLlmClassifier::test_llm_classifier_runs_and_routes_by_semantic_tier - stage without static complexity-smart-router can still register and call the virtual model; spend log shows a higher-tier backend when the LLM classifier runs
complexity-smart-router, run the router e2e suite (or only this node)/v1/modelseventually listscomplexity-smart-routerafter suite setupcomplexity-smart-routerdoes not return Invalid model nameclaude-haiku-4-5(or anthropic-prefixed) for promptIs P equal to NP?when the classifier is healthy (if the log shows gpt-5.5 with signals llm-classifier:SIMPLE, that is product LIT-4521, not this registration fix)tests/e2e/management/test_key_models_dropdown_e2e.py::TestKeyModelsDropdownUI::test_create_teamless_key_offers_proxy_scope_and_persists - create-key UI opens without navigation race
+ Create New Keyand expect the create modal (no aborted navigation to?create=true)All Proxy Modelsandgpt-5.5; notAll Team Models/key/infoshowsmodels: ["all-proxy-models"]and no teamPage.goto ...?create=trueinterrupted by redirect to/ui; button open is the product path users hittests/e2e/quota_management/budgets/test_multi_window_budget_e2e.py::test_short_window_blocks_then_resets - budget reset wait does not die on a single ALB 502
tests/e2e/claude_code/long_context_1m/test_anthropic.py::test_long_context_1m_anthropic (and azure/bedrock_*/vertex siblings) - 1M beta path uses models that actually support 1M
claude-sonnet-4-6(and provider-suffixed aliases fromtests/e2e/claude_code/test_config.yaml)--betas context-1m-2025-08-07and ~210k paddingtests/e2e/claude_code/count_tokens/test_vertex_ai.py::test_count_tokens_vertex_ai - Vertex capability gap is not_applicable, not fail
/v1/messages/count_tokensforclaude-haiku-4-5-vertex/claude-sonnet-4-5-vertextests/e2e/management/management_client.py create_* retries - team/user/org creation survives brief control-plane 500s
Final Attestation