From 24dfb4bb1e240b460ce85cbda7ba781667f4ef0f Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 17 Jun 2026 20:33:34 +0000 Subject: [PATCH 1/2] test(proxy): poll for image-gen spend instead of a fixed 5s sleep test_key_info_spend_values_image_generation failed once on litellm_internal_staging (pipeline 82282) with "spend did not increase on an identical repeat image call" (assert 0.24966 > 0.24966). The test made the second image call, slept 5s, then read the key's spend once. Response caching is commented out in proxy_server_config.yaml and no sibling test enables it, so the likely cause is async/batched spend logging not having flushed the repeat call's cost within 5s, which the build_and_test job aggravates by running every tests/test_*.py against one shared proxy under pytest -n 4. Poll the key's spend for up to 60s and break as soon as it grows. This removes the timing flake while preserving the canary: if the repeat were genuinely unbilled (for example the proxy response cache being on), spend never grows, the poll times out, and the assertion still fails. --- tests/test_keys.py | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/tests/test_keys.py b/tests/test_keys.py index 89977d43676e..003e27110550 100644 --- a/tests/test_keys.py +++ b/tests/test_keys.py @@ -621,17 +621,23 @@ async def retry_request(func, *args, _max_attempts=5, **kwargs): assert spend > 0 # The record/replay proxy serves this identical second call from its - # cassette (free), but the proxy must still bill it. If the proxy's own - # response cache were on, the repeat would be a $0 cache hit and spend - # would not move, silently zeroing recorded-call spend; assert it grows. + # cassette (free), but the proxy must still bill it. Spend logging is + # async/batched, so poll for the increase rather than reading once after a + # fixed sleep; a spend that never grows means the repeat was not billed + # (e.g. the proxy response cache is on), which this still catches. await image_generation(session=session, key=key) - await asyncio.sleep(5) - key_info = await retry_request( - get_key_info, session=session, get_key=key, call_key=key - ) - assert key_info["info"]["spend"] > spend, ( - "spend did not increase on an identical repeat image call; the proxy " - "response cache appears to be ON, which would zero recorded-call spend" + spend_after = spend + for _ in range(12): + await asyncio.sleep(5) + key_info = await retry_request( + get_key_info, session=session, get_key=key, call_key=key + ) + spend_after = key_info["info"]["spend"] + if spend_after > spend: + break + assert spend_after > spend, ( + "spend did not increase on an identical repeat image call; the repeat " + "was not billed (the proxy response cache may be on)" ) From d7ade9df345bb0056bb5a5313e365065d855f250 Mon Sep 17 00:00:00 2001 From: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Date: Wed, 17 Jun 2026 20:33:34 +0000 Subject: [PATCH 2/2] test(pass_through): raise ruby assistants client request_timeout to 600s The streaming assistants example in openai_assistants_passthrough_spec.rb hit Net::ReadTimeout on litellm_internal_staging (pipeline 82280), failing at roughly 125s which is ruby-openai's default request_timeout of 120s. An assistants run with the code_interpreter tool can occasionally take longer than that to stream its first content back through the pass-through. Raise the client's request_timeout to 600s, matching the 600s timeout the Python pass-through e2e tests already use, so a slow-but-healthy streaming run no longer trips the default read timeout. --- .../spec/openai_assistants_passthrough_spec.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/pass_through_tests/ruby_passthrough_tests/spec/openai_assistants_passthrough_spec.rb b/tests/pass_through_tests/ruby_passthrough_tests/spec/openai_assistants_passthrough_spec.rb index 1cfaeb5e209f..5a4dc0395f80 100644 --- a/tests/pass_through_tests/ruby_passthrough_tests/spec/openai_assistants_passthrough_spec.rb +++ b/tests/pass_through_tests/ruby_passthrough_tests/spec/openai_assistants_passthrough_spec.rb @@ -5,7 +5,8 @@ let(:client) do OpenAI::Client.new( access_token: "sk-1234", - uri_base: "http://0.0.0.0:4000/openai" + uri_base: "http://0.0.0.0:4000/openai", + request_timeout: 600 ) end