fix(tests): stop DATABASE_URL env pollution from read-replica tests breaking DB e2e tests

[mateo-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This fixes a CI-only test flake, so the proof is the deterministic reproduction of the worker poisoning itself; there is no proxy behavior change to demonstrate

CI occurrences of the flake, all on PRs that touch nothing related: https://github.com/BerriAI/litellm/actions/runs/28994072776/job/86039902861 (first attempt, poisoned worker gw1), https://github.com/BerriAI/litellm/actions/runs/28994072776/job/86048515541 (full job rerun, poisoned worker gw0), https://github.com/BerriAI/litellm/actions/runs/29036129022/job/86181354539 (different branch, same failure). Each shows 1 failed, 4242 passed, ..., 2 rerun with

FAILED tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py::TestDeprecatedKeyLookupDbE2E::test_deprecated_key_grace_period_cache_hit_path - httpx.ConnectError: All connection attempts failed
{"is_panic":false,"message":"Can't reach database server at `writer.aurora.local`:`5432`...","error_code":"P1001"}

writer.aurora.local is a fake hostname that only exists in tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py

Before, captured at 60729f7 (the branch point). PYTEST_XDIST_WORKER=gw0 mirrors a CI xdist worker (it disables the single-process litellm module reload in tests/test_litellm/conftest.py), and popping DATABASE_URL after importing litellm mirrors CI where the proxy-infra job defines no DATABASE_URL (locally a .env can set one at import time)

$ env -u DATABASE_URL PYTEST_XDIST_WORKER=gw0 uv run python -c "
import os, litellm, pytest
os.environ.pop('DATABASE_URL', None)
rc1 = pytest.main(['-q', '-p', 'no:xdist', '-p', 'no:pytest-retry', 'tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py'])
print('routing file exit:', rc1, '| DATABASE_URL now:', os.environ.get('DATABASE_URL'))
rc2 = pytest.main(['-q', '-p', 'no:xdist', '-p', 'no:pytest-retry', 'tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py::TestDeprecatedKeyLookupDbE2E'])
print('e2e exit:', rc2)
"
39 passed, 3 warnings in 2.45s
routing file exit: 0 | DATABASE_URL now: postgresql://litellm:TOKEN@writer.aurora.local:5432/litellm
...
{"is_panic":false,"message":"Can't reach database server at `writer.aurora.local`:`5432`\n\nPlease make sure your database server is running at `writer.aurora.local`:`5432`.","meta":{"database_host":"writer.aurora.local","database_port":5432},"error_code":"P1001"}
FAILED tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py::TestDeprecatedKeyLookupDbE2E::test_deprecated_key_grace_period_cache_hit_path
1 failed, 2 warnings in 11.16s
e2e exit: 1

That is the exact CI failure, reproduced deterministically in one process: the routing wrapper tests pass but leave the fake URL in os.environ, and the DB e2e test, which skips when DATABASE_URL is unset, arms itself on the poisoned value and dies

Regression guard demonstration, captured at 60729f7 plus only the new tests/test_litellm/proxy/db/conftest.py (no other change): the leaking tests now fail at teardown, at the culprit, instead of poisoning a distant test

_ ERROR at teardown of test_writer_get_rds_iam_token_defaults_port_when_unset __
E  AssertionError: tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py::test_writer_get_rds_iam_token_defaults_port_when_unset leaked DB env vars past monkeypatch teardown: {'DATABASE_URL': 'postgresql://litellm:TOKEN@writer.aurora.local:5432/litellm'}. ...
_ ERROR at teardown of test_writer_get_rds_iam_token_uses_database_host_env_vars _
E  AssertionError: ... leaked DB env vars past monkeypatch teardown: {'DATABASE_URL': 'postgresql://litellm:WRITER-TOKEN@writer.aurora.local:5432/litellm?schema=public'}. ...
39 passed, 3 warnings, 2 errors in 2.72s

After, captured at 4058667 (this PR)

$ env -u DATABASE_URL PYTEST_XDIST_WORKER=gw0 uv run python -c "
import os, litellm, pytest
os.environ.pop('DATABASE_URL', None)
rc1 = pytest.main(['-q', '-p', 'no:xdist', '-p', 'no:pytest-retry', 'tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py'])
print('routing file exit:', rc1, '| DATABASE_URL now:', os.environ.get('DATABASE_URL'))
rc2 = pytest.main(['-q', '-rs', '-p', 'no:xdist', '-p', 'no:pytest-retry', 'tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py::TestDeprecatedKeyLookupDbE2E'])
print('e2e exit:', rc2)
"
39 passed, 3 warnings in 2.58s
routing file exit: 0 | DATABASE_URL now: None
SKIPPED [1] tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py:582: DATABASE_URL not set; skipping DB-backed key-rotation E2E test.
1 skipped, 2 warnings in 0.03s
e2e exit: 0

And the CI-shaped run at 4058667 (2 xdist workers, loadscope, whole db dir plus the e2e file, DATABASE_URL unset)

$ env -u DATABASE_URL uv run pytest tests/test_litellm/proxy/db/ tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py -q -n 2 --dist=loadscope -rs
SKIPPED [1] tests/test_litellm/proxy/common_utils/test_key_rotation_e2e.py:582: DATABASE_URL not set; skipping DB-backed key-rotation E2E test.
326 passed, 1 skipped, 17 warnings in 6.64s

Type

🐛 Bug Fix
✅ Test

Changes

PrismaWrapper.get_rds_iam_token (litellm/proxy/db/prisma_client.py:329) writes the freshly minted writer URL into os.environ["DATABASE_URL"] as a side effect. Two tests in tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py exercise that path with the fake host writer.aurora.local and guarded the env var with monkeypatch.delenv("DATABASE_URL", raising=False). When the key is absent, which is the case in the proxy-infra CI job, delenv(raising=False) records nothing, so monkeypatch has nothing to restore and the fake URL written by product code survives the test. The pytest-xdist worker process is then permanently poisoned: TestDeprecatedKeyLookupDbE2E::test_deprecated_key_grace_period_cache_hit_path skips itself when DATABASE_URL is unset, but on a poisoned worker it sees the fake URL, connects Prisma to writer.aurora.local:5432 and fails with P1001. pytest-rerunfailures reruns the failed test inside the same worker process where os.environ is still poisoned, so the two configured reruns can never rescue it; and --dist=loadscope assigns the routing module and the single-test e2e class to workers based on runtime timing, so the two only sometimes land on the same worker, which is why unrelated PRs fail intermittently (gw1 in one linked run, gw0 in its rerun)

The fix has two layers. First, a new tests/test_litellm/proxy/db/conftest.py makes the bug class impossible for the whole directory: a pytest_runtest_setup/pytest_runtest_teardown hookwrapper pair snapshots the DATABASE_* env keys before any fixture runs and, after all fixture finalizers (including monkeypatch undo) have run, restores the snapshot and fails the test loudly if anything leaked. It has to be a hook pair rather than an autouse fixture because isolate_host_aws_config in the parent conftest requests monkeypatch, so the shared monkeypatch instance always finalizes after any module-level fixture and a fixture-based guard would compare against pre-undo state. Second, the two leaking tests now use a unset_database_url fixture that registers a setenv+delenv pair, giving monkeypatch a restore record even when the key started out unset

Running the guard over the directory also exposed two more latent leaks of the opposite direction (deleting a DATABASE_URL that was set at session start, which matters for anyone running with a .env and for future CI jobs that set DATABASE_URL): the setup_env fixtures in test_rds_iam_token_expiry.py wrote and popped os.environ directly, and _scrub_db_env in test_db_url_settings.py used a hand-rolled snapshot/restore that runs before monkeypatch undo and gets clobbered by it. Both now route through monkeypatch so restoration ordering is correct

The product-code os.environ mutation itself is left alone: the writer wrapper deliberately relies on Prisma re-reading DATABASE_URL from env on reconnect, so removing the side effect is a larger refactor than a test flake fix should carry

---

Note

Low Risk
Changes are confined to test infrastructure and env isolation; no proxy or runtime behavior is modified.

Overview
Adds tests/test_litellm/proxy/db/conftest.py with setup/teardown hooks that snapshot all DATABASE_* env keys, restore them after fixtures (including monkeypatch undo), and fail the test if anything still leaked. Also adds an unset_database_url fixture that registers setenv+delenv so monkeypatch can restore DATABASE_URL when it started unset—fixing the case where PrismaWrapper.get_rds_iam_token writes a fake URL into os.environ and poisons later tests on the same xdist worker.

Updates _scrub_db_env in test_db_url_settings.py and setup_env in test_rds_iam_token_expiry.py to use monkeypatch instead of manual snapshot/pop. The two writer IAM tests in test_routing_prisma_wrapper.py now depend on unset_database_url instead of delenv(..., raising=False) alone.

No production code changes.

^{Reviewed by Cursor Bugbot for commit 4058667. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR fixes an intermittent CI failure where PrismaWrapper.get_rds_iam_token writes a fake DATABASE_URL into os.environ as a side effect, and monkeypatch.delenv("DATABASE_URL", raising=False) leaves no restore record when the key started unset — poisoning every subsequent test in the same pytest-xdist worker process.

• A new tests/test_litellm/proxy/db/conftest.py adds hook-based snapshot/restore guards that compare DB env state before setup against state after all fixture teardowns, failing the culprit test loudly and restoring the environment even on leak, preventing cascading failures downstream.
• The two leaking tests in test_routing_prisma_wrapper.py now use the new unset_database_url fixture (a setenv+delenv pair) so monkeypatch always has a restore record for DATABASE_URL.
• Two opposite-direction latent leaks are also fixed: setup_env in test_rds_iam_token_expiry.py and _scrub_db_env in test_db_url_settings.py are converted from hand-rolled snapshot/restore to monkeypatch.setenv, correcting teardown ordering and eliminating the risk of deleting keys that were set at session start.

Confidence Score: 5/5

Test-only changes with no production code modified; the fix is mechanically correct and matches the failure mode described in the PR.

All changes are confined to test infrastructure for the proxy/db directory. The hook-based guard correctly places the snapshot before any fixture runs and the check after all fixture teardowns. No assertions are weakened and no production behavior changes.

No files require special attention.

Important Files Changed

Filename	Overview
tests/test_litellm/proxy/db/conftest.py	New conftest that snapshots DB env vars before setup and enforces no-leak at teardown via hook wrappers; also provides the unset_database_url fixture that gives monkeypatch a restore record even when the key starts unset.
tests/test_litellm/proxy/db/test_routing_prisma_wrapper.py	Two leaking tests now declare unset_database_url and drop the monkeypatch.delenv(raising=False) that was the root cause of xdist worker poisoning.
tests/test_litellm/proxy/db/test_rds_iam_token_expiry.py	Both setup_env fixtures converted from manual os.environ pop/set to monkeypatch.setenv; cleanup ordering is now correct and hand-rolled finally blocks are removed.
tests/test_litellm/proxy/db/test_db_url_settings.py	The autouse _scrub_db_env fixture converted from hand-rolled try/finally to monkeypatch.setenv+delenv, fixing an ordering bug where the hand-rolled restore was clobbered by monkeypatch's own undo.

_{Reviews (1): Last reviewed commit: "fix(tests): stop DATABASE_URL env pollut..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_db_url_env_pollution_flake (4058667) with litellm_internal_staging (131aa05)}

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4058667. Configure here.}