Repository navigation
test: backport stale and state-leaking test fixes to rc/1.104.0 (#43266) - #43267
Conversation
|
| "tests.rust-python-harness.strategies.trace_parity.sdk.messages.case", | ||
| "tests.rust-python-harness.strategies.trace_parity.sdk.chat_completions.case", |
There was a problem hiding this comment.
Removing these entries fixes this importability test, but the documented full harness check also collects trace_parity/test_runner.py and trace_parity/sdk/test_core_scenario_matrix.py. Those tests still import the deleted OCR case module, so they raise ModuleNotFoundError. Please update those references too; otherwise the full harness check remains broken.
…3266) test_update_config_success_callback_normalization replaced proxy_server.proxy_logging_obj with a MagicMock and never restored it. Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping, end-user and MCP tests on the same xdist worker awaited that mock and failed. The test now uses monkeypatch. test_prometheus_logging_callbacks set verbose_logger to DEBUG and litellm.set_verbose at import, so every worker in the unit job ran with DEBUG on. That broke caplog equality in the JEV classifier test, the vertex streaming memory ratio, and four event-loop lag checks. The module-level setup is removed; nothing in the file depended on it. #43081 removed the OCR harness modules but left them in the importability parametrize list. test_get_model_info_bedrock_region reassigned litellm.model_cost and set LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared the get_model_info caches, so it failed whenever an earlier test had looked up the regional model. It now uses monkeypatch and invalidates the caches; the local_testing isolation fixture also invalidates them after restoring model_cost. The Windows job hit CircleCI's 10 minute no-output limit while cargo compiles the Rust crates inside uv sync and uv build. Those two steps now allow 30 minutes of silence. (cherry picked from commit 7fc2206)
12eb8bb to
45d78ca
Compare
TLDR
Backport of #43266 to rc/1.104.0:
git cherry-pick -x 7fc2206171(the merged main commit), applied cleanly as 45d78caProblem this solves:
How it solves it:
test_update_config_success_callback_normalizationpatchesproxy_serverwith monkeypatchtest_get_model_info_bedrock_regionpatchesmodel_costand clears the model info cachesScreenshots / Proof of Fix
Tests-only change, so the proof is the failing CircleCI suites reproduced locally on the merge base and the same commands green on the tip. CircleCI reference: pipeline 90394 on main at c19ce71 (the rc/1.104.0 tip) had all of these red, and a same-SHA rerun (pipeline 90444) reproduced them
Before (c19ce71)
Leaked proxy_logging_obj mock
pytest -n 1 --dist=loadscope tests/unit/proxy/test_proxy_server.py tests/unit/proxy/_experimental/mcp_server/test_mcp_server.py tests/unit/proxy/auth/test_auth_checks.py tests/unit/proxy/management_endpoints/test_jwt_key_mapping.py14 failed, 224 passed, 13 skipped; the same 14 fail at c19ce71 in CircleCI pipeline 90394: the 10 JWT mapping tests, 2test_get_end_user_objectparams and 2call_mcp_tooltests, same set as CircleCIImport-time DEBUG logging
pytest -n 1 --dist=loadscope tests/unit/enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py tests/unit/router_strategy/complexity_router/test_jev_classifier.py tests/unit/llms/vertex_ai/files/test_vertex_ai_files_streaming.py::TestPathSourcedStreaming tests/unit/a2a_protocol/test_main.py::test_asend_message_counts_usage_off_the_event_loop tests/unit/integrations/compression_interception/test_compression_interception_handler.py::test_pre_call_hook_counts_tokens_off_the_event_loop tests/unit/router_utils/pre_call_checks/test_prompt_caching_deployment_check.py8 failed, 109 passed; the same 8 fail at c19ce71 in CircleCI pipeline 90394: 3 JEV caplog params, the vertex memory ratio, and 4 event-loop lag checksOCR harness list and bedrock region cache
pytest -p no:randomly tests/test_rust_python_harness.py tests/local_testing/<warm lookup of us.anthropic.claude-haiku-4-5> tests/local_testing/test_get_model_info.py::test_get_model_info_bedrock_region3 failed; at c19ce71 CircleCI shows both OCR importability params red, and the region test red on 2 of 4 Sep 25 main runs: importability params andtest_get_model_info_bedrock_regionAfter (45d78ca)
Leaked proxy_logging_obj mock
238 passed, 13 skippedImport-time DEBUG logging
117 passedOCR harness list and bedrock region cache
pytest tests/test_rust_python_harness.py tests/local_testing/test_get_model_info.py::test_get_model_info_bedrock_region10 passedType
Test
Infrastructure
Caveats (if any)
Low
test_e2e_langfuse_callbacks_in_db: the mock server lacks the Langfuse OTLP route feat(langfuse)!: migrate the sdk callback to langfuse v4 #36741 now uses, fixed in feat(langfuse): accept OTLP trace exports for the Langfuse v4 callback example_openai_endpoint#5langfuse_logging_unit_testshang and the Mythos parity data (fixed by fix(cost-map): retirement dates, chatgpt reasoning flags, bing pricing, bedrock mantle and mythos, azure gpt-5.6 alias, anthropic batch rates, new nebius, openrouter and xai rows #42951)