test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips - #37957
Conversation
… where a hit is the assertion
The e2e proxy runs with the response cache on, so any test that re-sends an
identical chat, messages, responses, completions, embeddings or rerank body
reads back a redis copy of an earlier call instead of reaching the provider.
Five tests in the last week failed that way. Default cache: {"no-cache": true}
on those request models and pass cache=None only in the two tests whose
assertion is the cache hit itself.
Both routes wait on providers that can legitimately take longer than the 60s transport-wide request timeout (gpt-image edits, Azure Document Intelligence), and a client-side read timeout there fails a green request. post/upload now accept a per-call timeout like get already does; only those two call sites use it.
Assertion failures still fail on the first attempt; only an outcome whose error string carries the e2e_http network kind or a 5xx status gets one more try. Test Engine records every attempt, so the flake rate stays visible while a single provider blip no longer reds the rc run.
…t why The burst is the precondition, not the property: one 5xx among six concurrent calls still leaves five workers racing the cold counter, which is what the reseed assertion measures. Two or more failures still abort, and the failing bodies are now in the message instead of only the status codes.
Greptile SummaryThis PR hardens the live E2E harness by disabling response-cache reuse by default, extending selected provider timeouts, retrying selected transient failures, and tolerating one failed reseed-burst request
Confidence Score: 4/5The PR needs correction before merging because its rerun filters miss common transient failure representations; the reseed relaxation also weakens regression coverage Common send and stream failures render as "status -1" or "status 500", so the new regexes do not provide the promised retry, while the reseed test accepts one unclassified product failure Files Needing Attention: tests/e2e/pytest.ini, tests/e2e/quota_management/budgets/test_spend_counter_reseed_e2e.py
|
| Filename | Overview |
|---|---|
| tests/e2e/pytest.ini | Adds selective reruns, but its regexes miss common StreamingResponse network and 5xx failure messages |
| tests/e2e/e2e_http.py | Unchanged supporting transport code demonstrates that raw and streaming failures use text incompatible with the new rerun filters |
| tests/e2e/models.py | Adds supported no-cache defaults to shared request models, with optional None allowing cache-specific tests to omit the field |
| tests/e2e/llm_translation/endpoints_client.py | Extends no-cache defaults to endpoint request models and applies the slow timeout to image edits |
| tests/e2e/transport.py | Adds optional post and upload timeout overrides and forwards them correctly through split transports |
| tests/e2e/quota_management/budgets/test_spend_counter_reseed_e2e.py | Permits one arbitrary failed synchronized request, weakening the existing product-regression signal |
| tests/e2e/quota_management/spend_tracking/spend_e2e_client.py | Propagates cache controls while allowing the cache-hit test to opt back into response caching |
| tests/e2e/router/reliability_support.py | Disables response caching by default for reliability requests while retaining an explicit cache opt-in |
| tests/e2e/proxy_client.py | Applies the configured slow-provider timeout to OCR requests |
Reviews (1): Last reviewed commit: "test(e2e): let the reseed burst survive ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # uv run pytest tests/e2e/quota_management/spend_tracking/ -v | ||
| # The e2e marker is also registered in conftest.py for runs rooted elsewhere. | ||
| addopts = --strict-markers --strict-config | ||
| addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]" |
There was a problem hiding this comment.
Streaming failures render as status -1 or status 500, so neither regex matches and transient network or upstream failures are not rerun.
| failed = [r for r in burst_results if not r.ok] | ||
| assert len(failed) <= BURST_TOLERATED_FAILURES, ( | ||
| "too many burst calls failed; cannot exercise concurrent reseed. " | ||
| f"statuses={[r.status_code for r in burst_results]} " | ||
| f"bodies={[r.body[:300] for r in failed]}" | ||
| ) |
There was a problem hiding this comment.
The threshold accepts any failed response, so one proxy-side regression can pass while the aggregate counter assertion still succeeds.
Rule Used: What: Flag any modifications to existing tests and... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
poll_traces_for_call already waits up to POLL_TIMEOUT for spans to land, but a single refused connection to the query API failed the test on the spot. Jaeger restarted twice during today's gate runs (19:05 and 19:41 UTC, each under a minute) and took ten and three otel tests with it while the same tests passed on the rc build minutes later. A network failure now counts as not-yet inside the same deadline; if Jaeger is still unreachable when the deadline passes the test fails with that error, and any non-network failure still fails immediately.
7aef79b
into
litellm_internal_staging
2 participants
TLDR
Problem this solves:
How it solves it (one commit each, bisectable):
cache: {"no-cache": true}on every cacheable request body (ChatBody,AnthropicMessagesBody,EmbedBody,ResponsesRequest,MessagesRequest,RichMessagesRequest,CompletionsRequest,EmbeddingsRequest,RerankRequest— the litellm cache's supported call types). The two tests whose assertion is the cache hit (test_cache_hit_is_zero_cost_and_suffixed,test_exact_cache_returns_cached) passcache=Noneexplicitly. Five tests failed this way last week (test(e2e): bypass the proxy response cache in the mid-conversation system and fallback tests #37915 fixed two models; this covers the rest).Per-call timeout on
post/upload, same shapegetalready has, used only by/v1/images/editsand/v1/ocr(E2E_SLOW_PROVIDER_TIMEOUT, default 180 s). The proxy's own timeout still bounds the call.--reruns 1scoped with--only-reruntokind='network'andstatus_code=5xx. AnAssertionErrorabout behaviour never reruns; a provider blip gets one more try. Test Engine records both attempts, so flake rate stays visible.Reseed burst tolerates one failed call out of six and prints the failing bodies. Five concurrent workers still race the cold counter, which is what the assertion measures; two or more failures still abort.
Jaeger poll survives a transient query failure.
poll_traces_for_callalready waits up to 120 s for spans to settle; a refused connection mid-poll now counts as not-yet inside that same deadline instead of failing on the spot. Jaeger restarted twice during this PR's gate runs (19:05 and 19:41 UTC, each under a minute) and took 10 then 3 otel tests with it while the identical tests passed on the rc build minutes later. If it is still down at the deadline the test fails with that error.Test-harness only; no product code.
Relevant issues
Follow-up to #37915. Evidence per flake: litellm-e2e builds 43, 45, 47, 50 and litellm-e2e-pr 111, 113.
Pre-Submission checklist