ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests by devin-ai-integration[bot] · Pull Request #42902 · BerriAI/litellm

devin-ai-integration · 2026-09-24T07:52:39Z

Stack 3/5 of the split of #42831 (reference PR, stays open until the stack lands). Below: #42901. Above: #42903. Carved from litellm_ci_migration_proxy_misc at c17f440 with git checkout <branch> -- <paths> and git checkout -p for the shared CI files, no rewrites

TLDR

Problem this solves:

caching, proxy-extras, gateway and enterprise tests live outside tests/unit
they only run from GitHub Actions, not from litellm-tests

How it solves it:

.circleci/scripts/unit_selection.sh with caching-local, proxy-extras, enterprise-package, enterprise-routing, proxy-infra
one tests.yml job per flag with reruns: 2; general unit shards exclude the legacy-owned paths
four tests/local_testing caching files -> tests/unit/caching
tests/litellm-proxy-extras -> tests/unit/litellm_proxy_extras, tests/test_gateway -> tests/unit/gateway
tests/enterprise and tests/test_litellm/enterprise -> tests/unit/enterprise
fork-flag in _test-unit-base.yml and test-unit.yml: drained GHA jobs keep their names and still run the moved paths on fork PRs
tests/unit/conftest.py allows localhost (pytest-rerunfailures xdist status server), outbound hosts stay blocked
matching assert_ci_coverage.py, Makefile hunks and the coverage test that ships with the script

Renames

git diff -M --stat litellm_ci_mig_2_key_cleanup litellm_ci_mig_3_small_trees: 34 renames, 33 at 100 percent. The one at 99 percent is test_litellm_proxy_extras_utils.py, where three __file__-relative paths gain one ../

 .circleci/scripts/unit_selection.sh                                                                  | 63 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 .circleci/tests.yml                                                                                  | 33 ++++++++++++++++++++++++---
 .github/scripts/assert_ci_coverage.py                                                                | 12 +++++++++-
 .github/workflows/_test-unit-base.yml                                                                | 33 +++++++++++++++++++++++----
 .github/workflows/test-unit.yml                                                                      | 22 ++++++++++--------
 Makefile                                                                                             |  2 +-
 tests/test_litellm/test_assert_ci_coverage.py                                                        |  2 +-
 tests/{test_litellm/enterprise/proxy => unit/caching}/__init__.py                                    |  0
 tests/{local_testing => unit/caching}/test_cache_preset_key.py                                       |  0
 tests/{local_testing => unit/caching}/test_caching_handler.py                                        |  0
 tests/{local_testing => unit/caching}/test_responses_stream_cache_keys.py                            |  0
 tests/{local_testing => unit/caching}/test_unit_test_caching.py                                      |  0
 tests/unit/conftest.py                                                                               |  2 +-
 tests/{ => unit}/enterprise/conftest.py                                                              |  0
 tests/unit/enterprise/enterprise_callbacks/send_emails/__init__.py                                   |  0
 tests/{test_litellm => unit}/enterprise/enterprise_callbacks/send_emails/test_base_email.py          |  0
 tests/{test_litellm => unit}/enterprise/enterprise_callbacks/send_emails/test_endpoints.py           |  0
 tests/{test_litellm => unit}/enterprise/enterprise_callbacks/send_emails/test_resend_email.py        |  0
 tests/{test_litellm => unit}/enterprise/enterprise_callbacks/send_emails/test_sendgrid_email.py      |  0
 .../litellm_enterprise => unit/enterprise}/enterprise_callbacks/test_prometheus_logging_callbacks.py |  0
 tests/unit/enterprise/integrations/__init__.py                                                       |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/integrations/test_custom_guardrail.py       |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/integrations/test_prometheus.py             |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/integrations/test_prometheus_unit_tests.py  |  0
 tests/unit/enterprise/proxy/__init__.py                                                              |  0
 tests/unit/enterprise/proxy/auth/__init__.py                                                         |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/proxy/auth/test_route_checks.py             |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/proxy/auth/test_user_api_key_auth.py        |  0
 tests/unit/enterprise/proxy/guardrails/__init__.py                                                   |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/proxy/guardrails/conftest.py                |  0
 .../litellm_enterprise => unit/enterprise}/proxy/guardrails/test_apply_guardrail_endpoint.py         |  0
 .../litellm_enterprise => unit/enterprise}/proxy/guardrails/test_bedrock_apply_guardrail.py          |  0
 tests/unit/enterprise/proxy/hooks/__init__.py                                                        |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/proxy/hooks/test_managed_files.py           |  0
 tests/unit/enterprise/proxy/management_endpoints/__init__.py                                         |  0
 .../enterprise}/proxy/management_endpoints/test_internal_user_endpoints.py                           |  0
 .../enterprise}/proxy/management_endpoints/test_project_endpoints_prisma.py                          |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_afile_retrieve_returns_unified_id.py              |  0
 tests/{enterprise/litellm_enterprise => unit/enterprise}/proxy/test_audit_logging_endpoints.py       |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_batch_retrieve_input_file_id.py                   |  0
 .../{test_litellm => unit}/enterprise/proxy/test_batch_retrieve_registers_missing_output_file_id.py  |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_batch_retrieve_returns_unified_input_file_id.py   |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_batch_update_db_managed_output_file_id.py         |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_deleted_file_returns_403_not_404.py               |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_enterprise_routes.py                              |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_file_deletion_blocking.py                         |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_managed_files_access_check.py                     |  0
 tests/{test_litellm => unit}/enterprise/proxy/test_managed_files_hook.py                             |  0
 tests/unit/gateway/__init__.py                                                                       |  0
 tests/{test_gateway => unit/gateway}/test_launch.py                                                  |  0
 tests/unit/litellm_proxy_extras/__init__.py                                                          |  0
 tests/{litellm-proxy-extras => unit/litellm_proxy_extras}/test_litellm_proxy_extras_logging.py       |  0
 tests/{litellm-proxy-extras => unit/litellm_proxy_extras}/test_litellm_proxy_extras_utils.py         |  6 ++---
 53 files changed, 152 insertions(+), 23 deletions(-)

Type

🚄 Infrastructure
✅ Test

Screenshots / Proof of Fix

litellm-tests on this head: https://app.circleci.com/pipelines/github/BerriAI/litellm/90348

Locally, each flag through unit_selection.sh with the CircleCI pytest flags under the same env -i allowlist with outbound network blocked: caching-local 44 passed, proxy-extras 78 passed, enterprise-routing 188 passed, enterprise-package 286 passed 4 skipped, proxy-infra 10 passed. assert_ci_coverage.py reports every test file owned by a job

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/d541aba7e5844703bab65085ef2d54b2
Open in Devin Desktop: https://app.devin.ai/desktop/session/d541aba7e5844703bab65085ef2d54b2?variant=devin

…list and coverage upload on failure

… synthetic values and integration and e2e coverage

…d wire the gemini pass-through split

…ts/unit and run them from litellm-tests under their legacy flags

devin-ai-integration · 2026-09-24T07:52:42Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

codecov · 2026-09-24T07:54:02Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-09-24T07:57:20Z

Confidence Score: 4/5

The PR’s test routing appears sound, but the repository’s comment requirement should be satisfied before merging

Findings

Unnecessary fork-flag comment ▶

Summary

Moves caching, proxy-extras, gateway, and enterprise tests into tests/unit and assigns them to dedicated CircleCI flags, while retaining fork coverage through GitHub Actions

_{Reviews (6) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

greptile-apps · 2026-09-24T07:57:44Z

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

Gateway tests violate unit requirements tests/unit/gateway/test_launch.py:35 ▶

Moving these tests into tests/unit brings in a fake PgBouncer subprocess and loopback sockets. The unit-test guide requires in-process tests with no network or subprocesses, so this requirement must be satisfied before merging

…ll_trees # Conflicts: # .circleci/tests.yml

mateo-berri · 2026-09-24T08:29:03Z

@greptileai

…no files

…ll_trees # Conflicts: # .circleci/tests.yml

mateo-berri · 2026-09-24T08:42:36Z

@greptileai

mateo-berri · 2026-09-24T08:53:55Z

On the gateway finding: tests/test_gateway/test_launch.py is unchanged, this PR only moves it. The destination follows the reviewed split plan, so it stays a rename here

mateo-berri · 2026-09-24T08:53:56Z

@greptileai

CLAassistant · 2026-09-24T21:57:59Z

All committers have signed the CLA.

devin-ai-integration · 2026-09-24T22:02:26Z

@greptileai

…l_trees

devin-ai-integration · 2026-09-24T22:40:32Z

@greptileai

greptile-apps · 2026-09-24T22:43:11Z

+# `fork-flag` names the `.circleci/tests.yml` job that now runs part of the
+# shard under the same Codecov flag. CircleCI does not build pull requests from
+# forks, so the shard still runs those files there and skips them elsewhere.


Unnecessary fork-flag comment The new comment repeats the workflow input description. AGENTS.md permits comments only for complex logic, tool inputs, or TODOs; remove it before merging

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

yuneng-berri added 5 commits September 24, 2026 07:35

ci: fix the litellm-tests unit job with sysmon coverage, an env allow…

1ad9006

…list and coverage upload on failure

test: replace key-dependent proxy, enterprise and mcp unit tests with…

0df19f6

… synthetic values and integration and e2e coverage

test: drop key reads at the legacy proxy, enterprise and mcp paths an…

3833a4c

…d wire the gemini pass-through split

ci: move caching, proxy-extras, gateway and enterprise tests into tes…

c4364ee

…ts/unit and run them from litellm-tests under their legacy flags

ci: move caching, proxy-extras, gateway and enterprise tests into tes…

d9dd7d6

…ts/unit and run them from litellm-tests under their legacy flags

This was referenced Sep 24, 2026

ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests #42903

Merged

test: take keys out of the legacy proxy, enterprise and mcp unit tests before moving them #42901

Merged

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/scripts/assert_ci_coverage.py

yuneng-berri added 4 commits September 24, 2026 08:24

ci: fail the unit shard when circleci tests split errors

8f84ef6

Merge branch 'litellm_ci_mig_1_ci_fix' into litellm_ci_mig_2_key_cleanup

372f717

test: drop restating comments from the gemini pass-through split

d2054d6

Merge branch 'litellm_ci_mig_2_key_cleanup' into litellm_ci_mig_3_sma…

8f8d572

…ll_trees # Conflicts: # .circleci/tests.yml

devin-ai-integration Bot force-pushed the litellm_ci_mig_2_key_cleanup branch from 4b0e503 to d2054d6 Compare September 24, 2026 08:27

devin-ai-integration Bot force-pushed the litellm_ci_mig_3_small_trees branch from bb62bba to 8f8d572 Compare September 24, 2026 08:27

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread .circleci/tests.yml

yuneng-berri added 3 commits September 24, 2026 08:41

ci: exit the unit shard cleanly when circleci tests split assigns it …

5553b66

…no files

Merge branch 'litellm_ci_mig_1_ci_fix' into litellm_ci_mig_2_key_cleanup

f2d1187

Merge branch 'litellm_ci_mig_2_key_cleanup' into litellm_ci_mig_3_sma…

d4e2447

…ll_trees # Conflicts: # .circleci/tests.yml

Base automatically changed from litellm_ci_mig_2_key_cleanup to main September 24, 2026 21:56

yuneng-berri requested a review from a team September 24, 2026 21:56

devin-ai-integration Bot requested review from mateo-berri and ryan-crabbe-berri as code owners September 24, 2026 21:57

devin-ai-integration Bot requested review from kerry-berri and yuneng-berri as code owners September 24, 2026 21:57

Merge remote-tracking branch 'origin/main' into litellm_ci_mig_3_smal…

715975f

…l_trees

devin-ai-integration Bot force-pushed the litellm_ci_mig_3_small_trees branch from d1e2cda to 715975f Compare September 24, 2026 22:39

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

yuneng-berri approved these changes Sep 24, 2026

View reviewed changes

yuneng-berri merged commit bf01870 into main Sep 24, 2026
102 of 110 checks passed

yuneng-berri deleted the litellm_ci_mig_3_small_trees branch September 24, 2026 22:50

yuneng-berri mentioned this pull request Sep 25, 2026

ci: run migrated unit selections on every event in legacy GHA shards #43182

Merged

2 tasks done

devin-ai-integration Bot mentioned this pull request Oct 4, 2026

test(ci): fix six CircleCI test regressions on main #44429

Merged

5 of 7 tasks

Conversation

devin-ai-integration Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Renames

Type

Screenshots / Proof of Fix

Pre-Submission checklist

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 24, 2026

Uh oh!

codecov Bot commented Sep 24, 2026

Codecov Report

Uh oh!

greptile-apps Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comments Outside Diff

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

CLAassistant commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot commented Sep 24, 2026

Uh oh!

devin-ai-integration Bot commented Sep 24, 2026

Uh oh!

greptile-apps Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

devin-ai-integration Bot commented Sep 24, 2026 •

edited

Loading

greptile-apps Bot commented Sep 24, 2026 •

edited

Loading

greptile-apps Bot commented Sep 24, 2026 •

edited

Loading

CLAassistant commented Sep 24, 2026 •

edited

Loading