Skip to content

test: backport stale and state-leaking test fixes to rc/1.104.0 (#43266) - #43267

Merged
yuneng-berri merged 1 commit into
rc/1.104.0from
litellm_rc_1_104_0_stale_test_fixes
Sep 26, 2026
Merged

yuneng-berri merged 1 commit into
rc/1.104.0from
litellm_rc_1_104_0_stale_test_fixes

Conversation

@yuneng-berri

@yuneng-berri yuneng-berri commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Backport of #43266 to rc/1.104.0: git cherry-pick -x 7fc2206171 (the merged main commit), applied cleanly as 45d78ca

Problem this solves:

  • 30 tests red on every scheduled main CircleCI run, including the rc/1.104.0 SHA
  • Two tests leak global state into their neighbors on the same worker
  • One parametrize list still names OCR harness modules that were deleted
  • The Windows job dies on the 10 minute no-output limit while compiling Rust

How it solves it:

  • test_update_config_success_callback_normalization patches proxy_server with monkeypatch
  • The prometheus test module no longer turns on DEBUG logging at import
  • The OCR entries leave the harness importability list
  • test_get_model_info_bedrock_region patches model_cost and clears the model info caches
  • Windows install and wheel steps allow 30 minutes of silence

Screenshots / Proof of Fix

Tests-only change, so the proof is the failing CircleCI suites reproduced locally on the merge base and the same commands green on the tip. CircleCI reference: pipeline 90394 on main at c19ce71 (the rc/1.104.0 tip) had all of these red, and a same-SHA rerun (pipeline 90444) reproduced them

Before (c19ce71)

Leaked proxy_logging_obj mock

  1. pytest -n 1 --dist=loadscope tests/unit/proxy/test_proxy_server.py tests/unit/proxy/_experimental/mcp_server/test_mcp_server.py tests/unit/proxy/auth/test_auth_checks.py tests/unit/proxy/management_endpoints/test_jwt_key_mapping.py
  2. On main's merge base this was 14 failed, 224 passed, 13 skipped; the same 14 fail at c19ce71 in CircleCI pipeline 90394: the 10 JWT mapping tests, 2 test_get_end_user_object params and 2 call_mcp_tool tests, same set as CircleCI

Import-time DEBUG logging

  1. pytest -n 1 --dist=loadscope tests/unit/enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py tests/unit/router_strategy/complexity_router/test_jev_classifier.py tests/unit/llms/vertex_ai/files/test_vertex_ai_files_streaming.py::TestPathSourcedStreaming tests/unit/a2a_protocol/test_main.py::test_asend_message_counts_usage_off_the_event_loop tests/unit/integrations/compression_interception/test_compression_interception_handler.py::test_pre_call_hook_counts_tokens_off_the_event_loop tests/unit/router_utils/pre_call_checks/test_prompt_caching_deployment_check.py
  2. On main's merge base this was 8 failed, 109 passed; the same 8 fail at c19ce71 in CircleCI pipeline 90394: 3 JEV caplog params, the vertex memory ratio, and 4 event-loop lag checks

OCR harness list and bedrock region cache

  1. pytest -p no:randomly tests/test_rust_python_harness.py tests/local_testing/<warm lookup of us.anthropic.claude-haiku-4-5> tests/local_testing/test_get_model_info.py::test_get_model_info_bedrock_region
  2. On main's merge base this was 3 failed; at c19ce71 CircleCI shows both OCR importability params red, and the region test red on 2 of 4 Sep 25 main runs: importability params and test_get_model_info_bedrock_region

After (45d78ca)

Leaked proxy_logging_obj mock

  1. Same command
  2. 238 passed, 13 skipped

Import-time DEBUG logging

  1. Same command
  2. 117 passed

OCR harness list and bedrock region cache

  1. pytest tests/test_rust_python_harness.py tests/local_testing/test_get_model_info.py::test_get_model_info_bedrock_region
  2. 10 passed

Type

Test
Infrastructure

Caveats (if any)

Low

@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Test fixes for state isolation and CI timeouts.

The PR appears safe to merge, though the documented full harness suite retains stale OCR imports.

Findings

  1. P2 Stale OCR imports remain ▶

Summary

This PR isolates test state, removes stale OCR importability cases, and extends the no-output timeout for two Windows build steps.

  • The remaining OCR references in the separately documented harness suite still need cleanup.

Reviews (1) · Last reviewed commit: "test: fix stale and state-leaking tests ..."

Comment on lines 39 to 40
"tests.rust-python-harness.strategies.trace_parity.sdk.messages.case",
"tests.rust-python-harness.strategies.trace_parity.sdk.chat_completions.case",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Stale OCR imports remain

Removing these entries fixes this importability test, but the documented full harness check also collects trace_parity/test_runner.py and trace_parity/sdk/test_core_scenario_matrix.py. Those tests still import the deleted OCR case module, so they raise ModuleNotFoundError. Please update those references too; otherwise the full harness check remains broken.

…3266)

test_update_config_success_callback_normalization replaced
proxy_server.proxy_logging_obj with a MagicMock and never restored it.
Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping,
end-user and MCP tests on the same xdist worker awaited that mock and
failed. The test now uses monkeypatch.

test_prometheus_logging_callbacks set verbose_logger to DEBUG and
litellm.set_verbose at import, so every worker in the unit job ran with
DEBUG on. That broke caplog equality in the JEV classifier test, the
vertex streaming memory ratio, and four event-loop lag checks. The
module-level setup is removed; nothing in the file depended on it.

#43081 removed the OCR harness modules but left them in the
importability parametrize list.

test_get_model_info_bedrock_region reassigned litellm.model_cost and set
LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared
the get_model_info caches, so it failed whenever an earlier test had
looked up the regional model. It now uses monkeypatch and invalidates
the caches; the local_testing isolation fixture also invalidates them
after restoring model_cost.

The Windows job hit CircleCI's 10 minute no-output limit while cargo
compiles the Rust crates inside uv sync and uv build. Those two steps now
allow 30 minutes of silence.

(cherry picked from commit 7fc2206)
@yuneng-berri
yuneng-berri force-pushed the litellm_rc_1_104_0_stale_test_fixes branch from 12eb8bb to 45d78ca Compare September 26, 2026 02:12
@yuneng-berri yuneng-berri changed the title test: fix stale and state-leaking tests red on scheduled CircleCI (rc/1.104.0) test: backport stale and state-leaking test fixes to rc/1.104.0 (#43266) Sep 26, 2026
@yuneng-berri
yuneng-berri merged commit 4b3c72a into rc/1.104.0 Sep 26, 2026
8 of 54 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_rc_1_104_0_stale_test_fixes branch September 26, 2026 02:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant