ci(llm_responses_api_testing): bound live re-record calls and rerun timeout-only failures to stop 15m no-output kills

[mateo-berri]

Relevant issues

Context: ci/circleci: llm_responses_api_testing failed on PR #32389 at dadf926 (https://circleci.com/gh/BerriAI/litellm/2010679) even though that PR only touches Bedrock cost-map lookup ordering and the suite has zero Bedrock usage; a retrigger with no relevant change passed (https://circleci.com/gh/BerriAI/litellm/2013953), confirming a flake

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests (no test surface; this is a CI-config fix, proven with live loop runs below)
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

The failure signature, from the CircleCI step log of job 2010679: 168 of 170 tests finished within about 30 seconds of the run starting (all [VCR HIT] cassette replays), one test started and never finished, and the step then produced zero output until CircleCI killed the job

tests/llm_responses_api_testing/test_openai_responses_api.py::test_basic_openai_responses_with_websearch[False]
...
[gw3] [ 98%] PASSED tests/llm_responses_api_testing/test_openai_responses_api.py::test_openai_compact_responses_api[False]
[VCR HIT] played=1 entries=1 :: tests/llm_responses_api_testing/test_openai_responses_api.py::test_openai_compact_responses_api[False]

Too long with no output (exceeded 15m0s): context deadline exceeded

The same signature killed this job 6 times in the last 8 days, with 4 different tests hanging (CircleCI job number, hung test): 2010679 and 2012169 and 1993606 test_basic_openai_responses_with_websearch[False], 2004033 and 1947340 test_responses_api_multi_turn_with_reasoning_and_structured_output, 2013288 test_azure_responses_api.py::TestAzureResponsesAPITest::test_responses_api_shell_tool, 1960817 test_gemini_3_responses_api_streaming_with_thought_signatures. So the fix targets the shared mechanism (unbounded live re-record calls), not one test

Before, captured at 4b0ac8b (origin/litellm_internal_staging). The stall itself is a rare provider-connection event, so a healthy loop passes; this establishes the baseline behavior and cost of the live call path (real OpenAI gpt-5.5 calls with the web_search tool, no cassette, no mocks)

$ cd tests/llm_responses_api_testing
$ for i in $(seq 1 20); do .venv/bin/python -m pytest test_openai_responses_api.py::test_basic_openai_responses_with_websearch -q; done
iter 1: pass 5s ... iter 20: pass 4s
BEFORE fix: 20 passed, 0 failed of 20

What happens today when that live call stalls: nothing bounds it under litellm's 6000s default request timeout, so the worker sits silent until the 15m no-output kill. Demonstrated at 4b0ac8b by forcing every live call to be slower than the bound; with REQUEST_TIMEOUT set the stall surfaces as a visible litellm.Timeout in under a second instead of silence

$ REQUEST_TIMEOUT=0.3 .venv/bin/python -m pytest "test_openai_responses_api.py::test_basic_openai_responses_with_websearch[False]" -q
E  litellm.exceptions.Timeout: litellm.Timeout: Connection timed out. Timeout passed=0.3, time taken=0.338 seconds
1 failed in 0.74s

After, captured at 3271a77, same forced-stall scenario but with the new job flags; both the streaming and non-streaming variants time out fast, get rerun once, and the failure output stays visible the whole time (in a real one-off stall the rerun records on a fresh connection and passes, keeping the job green)

$ REQUEST_TIMEOUT=0.3 .venv/bin/python -m pytest test_openai_responses_api.py::test_basic_openai_responses_with_websearch -q --reruns 1 --only-rerun Timeout
2 failed, 2 rerun in 2.59s

After, captured at 3271a77, healthy-path loop with the exact new knobs (real OpenAI calls)

$ for i in $(seq 1 20); do REQUEST_TIMEOUT=180 .venv/bin/python -m pytest test_openai_responses_api.py::test_basic_openai_responses_with_websearch -q --reruns 1 --only-rerun Timeout; done
iter 1: pass 5s ... iter 20: pass 4s
AFTER fix: 20 passed, 0 failed of 20

And a full live (cassette-disabled) run of the OpenAI and Gemini test files with the new flags, to simulate the worst case where every cassette expired on the same day, captured at 3271a77

$ REQUEST_TIMEOUT=180 .venv/bin/python -m pytest test_openai_responses_api.py test_google_ai_studio_responses_api.py -q -n 8 --reruns 1 --only-rerun Timeout
4 failed, 62 passed, 11 skipped in 24.82s

The 4 failures are pre-existing pure-live failures, byte-for-byte the same set on unmodified origin/litellm_internal_staging (4b0ac8b) without the new flags (they pass in CI via cassette replay), so the new knobs change nothing on healthy runs

$ .venv/bin/python -m pytest "test_openai_responses_api.py::TestOpenAIResponsesAPITest::test_basic_openai_responses_delete_endpoint" "test_openai_responses_api.py::TestOpenAIResponsesAPITest::test_responses_api_multi_turn_with_reasoning_and_structured_output" test_openai_responses_api.py::test_openai_streaming_logging -q
4 failed in 6.03s

Type

🚄 Infrastructure

Changes

This suite replays provider traffic from the Redis-backed VCR cassette cache. Cassettes expire 24 hours after their last write and the TTL is deliberately not refreshed on read (tests/_vcr_redis_persister.py), so roughly once per day each test re-records against the live provider. That re-record call runs with litellm's default request timeout of 6000 seconds (litellm/constants.py:381, applied on the responses path at litellm/responses/main.py:1109), so when the provider connection stalls (server-side web_search execution, reasoning calls, mid-stream reads) nothing client-side aborts it. Every other test finishes within a minute via cassette hits, the lone xdist worker sits silent, and CircleCI kills the whole job at no_output_timeout: 15m. The kill also means junit.xml is never written, so the failed job reports no failing test, which is why this flake kept looking random

Two scoped knobs on the job fix the mechanism. REQUEST_TIMEOUT: "180" in the job environment bounds every live call at 180 seconds (cassette replays never hit the network and are unaffected; the slowest legitimate live requests in this suite run well under a minute). --reruns 1 --only-rerun Timeout reruns only timeout-class failures once, so a one-off dead connection re-records on a fresh connection and the job stays green, while a persistent provider incident still fails the job, now within minutes and with a readable litellm.Timeout traceback instead of a silent 15 minute hang. The rerun is deliberately not blanket: non-timeout failures are real failures and still fail immediately

[greptile-apps]

Greptile Summary

This PR fixes a recurring CI flake in llm_responses_api_testing where a live VCR re-record call stalled indefinitely (up to litellm's 6000 s default timeout), causing CircleCI's 15-minute no-output kill to fire with no junit.xml and no visible failing test.

• Adds REQUEST_TIMEOUT: "180" as a job-level environment variable, which litellm/constants.py reads at import time to cap every live provider call at 180 seconds; VCR cassette replays are unaffected since they never hit the network.
• Adds --reruns 1 --only-rerun Timeout to the pytest invocation so a one-off stall retries once and records a cassette on the second attempt, while non-timeout failures still fail immediately; pytest-rerunfailures==15.1 is already in pyproject.toml.

Confidence Score: 5/5

CI-only change with no production code touched; the two new knobs are scoped entirely to one job and correctly wired to litellm's existing REQUEST_TIMEOUT environment variable.

Both mechanisms are verified in the codebase: REQUEST_TIMEOUT is read from the environment in litellm/constants.py:381 and pytest-rerunfailures 15.1 is already declared in pyproject.toml. The 180 s bound is well above observed healthy live-call times and well below the 15-minute no-output kill.

No files require special attention.

Important Files Changed

Filename	Overview
.circleci/config.yml	Adds REQUEST_TIMEOUT=180 job env var and --reruns 1 --only-rerun Timeout pytest flags to the llm_responses_api_testing job to bound live re-record calls and prevent silent 15-minute kills

_{Reviews (1): Last reviewed commit: "ci(llm_responses_api_testing): bound liv..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_fix_responses_api_flake_live_rerecord_timeout (3271a77) with litellm_internal_staging (b2e2a38)}