Repository navigation
ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests - #42903
Merged
Conversation
…list and coverage upload on failure
… synthetic values and integration and e2e coverage
…d wire the gemini pass-through split
…ts/unit and run them from litellm-tests under their legacy flags
…ts/unit and run them from litellm-tests under their legacy flags
…-db shards from litellm-tests
Contributor
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…ll_trees # Conflicts: # .circleci/tests.yml
devin-ai-integration
Bot
force-pushed
the
litellm_ci_mig_3_small_trees
branch
from
September 24, 2026 08:27
bb62bba to
8f8d572
Compare
devin-ai-integration
Bot
force-pushed
the
litellm_ci_mig_4_proxy
branch
from
September 24, 2026 08:27
f80e605 to
d9b14ca
Compare
Contributor
…ll_trees # Conflicts: # .circleci/tests.yml
Contributor
1 similar comment
Contributor
Author
Contributor
Author
devin-ai-integration
Bot
force-pushed
the
litellm_ci_mig_3_small_trees
branch
from
September 24, 2026 22:39
d1e2cda to
715975f
Compare
devin-ai-integration
Bot
force-pushed
the
litellm_ci_mig_4_proxy
branch
from
September 24, 2026 22:39
7edd12a to
9be8944
Compare
Contributor
Author
yuneng-berri
approved these changes
Sep 24, 2026
Contributor
Author
yuneng-berri
added a commit
that referenced
this pull request
Sep 26, 2026
…3266) test_update_config_success_callback_normalization replaced proxy_server.proxy_logging_obj with a MagicMock and never restored it. Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping, end-user and MCP tests on the same xdist worker awaited that mock and failed. The test now uses monkeypatch. test_prometheus_logging_callbacks set verbose_logger to DEBUG and litellm.set_verbose at import, so every worker in the unit job ran with DEBUG on. That broke caplog equality in the JEV classifier test, the vertex streaming memory ratio, and four event-loop lag checks. The module-level setup is removed; nothing in the file depended on it. #43081 removed the OCR harness modules but left them in the importability parametrize list. test_get_model_info_bedrock_region reassigned litellm.model_cost and set LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared the get_model_info caches, so it failed whenever an earlier test had looked up the regional model. It now uses monkeypatch and invalidates the caches; the local_testing isolation fixture also invalidates them after restoring model_cost. The Windows job hit CircleCI's 10 minute no-output limit while cargo compiles the Rust crates inside uv sync and uv build. Those two steps now allow 30 minutes of silence.
yuneng-berri
added a commit
that referenced
this pull request
Sep 26, 2026
…3266) test_update_config_success_callback_normalization replaced proxy_server.proxy_logging_obj with a MagicMock and never restored it. Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping, end-user and MCP tests on the same xdist worker awaited that mock and failed. The test now uses monkeypatch. test_prometheus_logging_callbacks set verbose_logger to DEBUG and litellm.set_verbose at import, so every worker in the unit job ran with DEBUG on. That broke caplog equality in the JEV classifier test, the vertex streaming memory ratio, and four event-loop lag checks. The module-level setup is removed; nothing in the file depended on it. #43081 removed the OCR harness modules but left them in the importability parametrize list. test_get_model_info_bedrock_region reassigned litellm.model_cost and set LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared the get_model_info caches, so it failed whenever an earlier test had looked up the regional model. It now uses monkeypatch and invalidates the caches; the local_testing isolation fixture also invalidates them after restoring model_cost. The Windows job hit CircleCI's 10 minute no-output limit while cargo compiles the Rust crates inside uv sync and uv build. Those two steps now allow 30 minutes of silence. (cherry picked from commit 7fc2206)
yuneng-berri
added a commit
that referenced
this pull request
Sep 26, 2026
…3266) (#43267) test_update_config_success_callback_normalization replaced proxy_server.proxy_logging_obj with a MagicMock and never restored it. Since the proxy unit tests joined tests/unit (#42903), 14 JWT mapping, end-user and MCP tests on the same xdist worker awaited that mock and failed. The test now uses monkeypatch. test_prometheus_logging_callbacks set verbose_logger to DEBUG and litellm.set_verbose at import, so every worker in the unit job ran with DEBUG on. That broke caplog equality in the JEV classifier test, the vertex streaming memory ratio, and four event-loop lag checks. The module-level setup is removed; nothing in the file depended on it. #43081 removed the OCR harness modules but left them in the importability parametrize list. test_get_model_info_bedrock_region reassigned litellm.model_cost and set LITELLM_LOCAL_MODEL_COST_MAP without restoring either, and never cleared the get_model_info caches, so it failed whenever an earlier test had looked up the regional model. It now uses monkeypatch and invalidates the caches; the local_testing isolation fixture also invalidates them after restoring model_cost. The Windows job hit CircleCI's 10 minute no-output limit while cargo compiles the Rust crates inside uv sync and uv build. Those two steps now allow 30 minutes of silence. (cherry picked from commit 7fc2206)
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack 4/5 of the split of #42831 (reference PR, stays open until the stack lands). Below: #42902. Above: #42904. Carved from
litellm_ci_migration_proxy_miscat c17f440 withgit checkout <branch> -- <paths>andgit checkout -pfor the shared CI files, no rewritesTLDR
Problem this solves:
tests/proxy_unit_testslives outsidetests/unitand only runs from GitHub ActionsHow it solves it:
tests/proxy_unit_tests->tests/unit/proxywith itsconftest.py,test_configs,.env, fixtures and audio filestest_skills_db.py->tests/unit/skillsproxy-db-*groups inunit_selection.sh, onetests.ymljob each withreruns: 2and Prisma setuptest-unit-proxy-db.ymlkeeps its check names, runs the moved paths on fork PRs onlyMakefile,classify_changes.sh,assert_ci_coverage.py,test-unit.yml,test_circleci_path_filter.py,user_api_key_auth_code_coverage.py,test_azure_routes.py, skills READMERenames
git diff -M --stat litellm_ci_mig_3_small_trees litellm_ci_mig_4_proxy: 73 renames, 69 at 100 percent. The four below 100 percent are path edits only:test_jwt_key_mapping.py(sys.path.insertgains two../),test_proxy_config_unit_test.py(os.listdirfiltered to yaml files because the package now has__init__.py),test_proxy_server.py(two sibling imports becometests.unit.proxy...absolute imports),test_skills_db.py(Path(__file__).parents[2]for the fixtures dir). The emptytest_key_generate_dynamodb.pyis deleted rather than movedType
🚄 Infrastructure
✅ Test
Screenshots / Proof of Fix
litellm-testson this head: https://app.circleci.com/pipelines/github/BerriAI/litellm/90352Locally, all twelve
proxy-db-*groups throughunit_selection.shunder the sameenv -iallowlist with outbound network blocked are green,assert_ci_coverage.pyreports every test file owned by a job, andtest_assert_ci_coverage.pyplustest_circleci_path_filter.pypass (101 tests)Greptile follow-up in d9b14ca:
make test-proxy-unit-arunstests/unit/proxyminustest_[p-z]*.pyandtest-proxy-unit-baddstests/unit/skills, so together they collect all 1129 tests of the moved tree instead of only the top-level filesCaveats (if any)
Low
test_proxy_token_counter.pycarries two failures inherited from main, unchanged by the move (see stack 2)Pre-Submission checklist
Final Attestation
Link to Devin session: https://app.devin.ai/sessions/d541aba7e5844703bab65085ef2d54b2
Open in Devin Desktop: https://app.devin.ai/desktop/session/d541aba7e5844703bab65085ef2d54b2?variant=devin