test: inject the HIBP client and the MCP loop clock so two backend tests stop flaking

[devin-ai-integration]

TLDR

Problem this solves:

• Two backend tests flaked on main in the 24h ending 2026-10-01 09:19 UTC
• The breached-password update test silently called the real HIBP API
• The MCP incomplete-walk test ran every case under a real 200ms budget

How it solves it:

• The single-user update path takes an injected HIBP client, like the bulk path
• The MCP test runs on an event loop whose clock only the test advances
• The only production change is an optional hibp_client parameter defaulting to today's behavior

User Flow

Before: a contributor's unrelated PR goes red on the Unit Tests or Postgres Tests job

1. They push a commit that touches neither user management nor the MCP client
2. The job fails on test_user_update_rejects_breached_password or one case of test_optional_discovery_rejects_incomplete_walks
3. They re-run the job with no change and it passes

After: the same commit gets a green job on the first attempt

1. They push the same commit
2. Both tests now finish the same way on an idle runner, a contended runner, or a runner with no internet access
3. The job passes on the first attempt

Flake evidence (run date 2026-10-01)

tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py::test_user_update_rejects_breached_password

Failed once on main, attempt 1: https://github.com/BerriAI/litellm/actions/runs/36805338275 (Postgres Tests at c168199e33). The log shows Breached-password check skipped, HIBP lookup failed: Timeout on reading data from socket and then TypeError: object MagicMock can't be used in 'await' expression. The same test passed on 18 other main runs of that job in the window, including https://github.com/BerriAI/litellm/actions/runs/36798373650 at the adjacent SHA 6997223068

Mechanism: real network call. The test mocked HIBP with respx, but the HIBP client is built by get_async_httpx_client, whose transport respx never intercepts, so the route was never hit and the test really queried api.pwnedpasswords.com. When that lookup timed out the check failed open by design and the test fell through to the unmocked database calls. Locally, at the merge base, the test passes with network access and fails with the CI signature under unshare -rn (no network), and a probe confirmed the respx route records zero calls

Fix: _update_single_user_helper and _hash_password_in_dict accept an optional hibp_client, the same seam bulk_update_processed_users already exposes. The test injects an AsyncHTTPHandler over httpx.MockTransport and now also asserts exactly one GET https://api.pwnedpasswords.com/range/<prefix> lookup. Production callers pass nothing and keep using the shared HIBP client. Mutation check: skipping the breach check in _hash_password_in_dict makes the test fail

tests/unit/experimental_mcp_client/test_mcp_client.py::test_optional_discovery_rejects_incomplete_walks

Failed 5 times on main, all attempt 1, interleaved with passing runs of the same jobs at adjacent SHAs:

• https://github.com/BerriAI/litellm/actions/runs/36798373382 (Postgres Tests at 6997223068), [False-internal_error-resources/templates/list], assert 0 == 2
• https://github.com/BerriAI/litellm/actions/runs/36805100619 (Unit Tests at 0c515ed7a8), same case and signature
• https://github.com/BerriAI/litellm/actions/runs/36816003415 (Unit Tests at ef6aa4ad66), [True-method_not_found-resources/templates/list], TimeoutError
• https://github.com/BerriAI/litellm/actions/runs/36820064257 (Postgres Tests at 6d7d183a80), [False-internal_error-resources/templates/list], assert 0 == 2
• https://github.com/BerriAI/litellm/actions/runs/36822334825 (Postgres Tests at 0980f756bd), same case and signature

It also failed on two unrelated PR branches, https://github.com/BerriAI/litellm/actions/runs/36808597812 ([True-repeat-resources/templates/list], TimeoutError) and https://github.com/BerriAI/litellm/actions/runs/36830884630 ([False-unauthorized-resources/list], assert 0 == 2)

Mechanism: wall-clock dependence. Every case built the client with timeout=0.2, which is the read timeout for initialize and for each page as well as the deadline for the whole page walk. Only the deadline case is about time, but on a contended runner initialize or the first page of any case could take longer than 200ms of real time and time out before the behavior under test ran. Locally, with 8 busy loops pinned to the same CPU, the merge base failed 9 of 10 runs (36 failing cases, spread across cap, internal_error, unauthorized and others)

Fix: the test runs its coroutine on _ManualClockLoop, an asyncio loop whose time() only moves when the test calls advance. anyio deadlines and the MCP SDK read timeouts schedule on that clock, so no case can time out from scheduler latency. The deadline case advances the clock on purpose: 0.1s while serving page one, then 0.15s when page two arrives and hangs, so the 0.2s walk deadline is due while page two's own read timeout (due at 0.3s) is not, and strict mode gets TimeoutError deterministically. Page two then yields to the loop at most 1,000 times and returns a 500 if nothing cancelled it, so a broken deadline fails in seconds instead of hanging (cancellation arrives after 10 yields today). The loop is driven with run_until_complete, which keeps the test running on Python 3.10. The repeat case's page cap goes from 2 to 3: with a cap of 2 the cap error masked a missing repeated-cursor check, so that check was untested. Mutation checks: removing the repeated-cursor check fails all six repeat cases, and pushing the walk deadline out by 1s fails all six deadline cases within 3 seconds

Not flakes in the window

Real breakages on main, failing on every run until fixed or still failing: tests/unit/interactions/test_openapi_compliance.py (4 tests, 51 failing runs), tests/test_litellm/proxy/batches_endpoints/test_endpoints.py::test_create__non_object_metadata_is_400 (52 failing runs), tests/test_litellm/proxy/test_component_allowlists.py::test_gateway_plus_backend_covers_full_app (13 consecutive runs, then fixed), and tests/unit/test_utils.py::test_aaamodel_prices_and_context_window_json_is_valid (schema rejected new *_ultrafast cost keys). Infra: Codecov upload signature verification failures on Postgres Tests, dependency install timeouts, and the Code Quality undocumented env var check. None of these are touched here

Other open deflake PRs

The auto-router modal integration flake (edit_auto_router_modal.integration.test.tsx, failing on 17 main UI Unit Tests runs in this window) is already covered by #42936 and #43437, and #41219 covers TeamSSOSettings. All three carry a human approval, so they are left untouched under the human-reviewed exception and this PR is the new rolling PR

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

There is no proxy or Admin UI surface for this change: the production edit only adds an optional parameter that every live caller leaves unset. The proof is the flake reproduction itself, run from the repo root at the merge base and at the tip

Before (0980f75)

Breached-password test without network

1. LITELLM_LOCAL_MODEL_COST_MAP=True unshare -rn .venv/bin/python -m pytest "tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py::test_user_update_rejects_breached_password" -q
2. Fails with TypeError: object MagicMock can't be used in 'await' expression after HIBP lookup failed, the CI signature

MCP walk test under CPU contention

1. for i in $(seq 8); do taskset -c 0 sh -c 'while :; do :; done' & done, then 10 times LITELLM_LOCAL_MODEL_COST_MAP=True taskset -c 0 uv run --no-sync pytest tests/unit/experimental_mcp_client/test_mcp_client.py -k test_optional_discovery_rejects_incomplete_walks -n 4 -q -p no:randomly
2. 9 of 10 runs fail, 4 failing cases each

After (9b97b5f)

Breached-password test without network

1. Same command
2. 1 passed

MCP walk test under CPU contention

1. Same commands
2. 10 of 10 runs report 42 passed

Type

✅ Test

Caveats (if any)

Low

• test_claim_token_rejects_breached_password_before_consuming_invite uses the same respx pattern on the onboarding path
  • It did not flake in this window, so it is reported, not changed here

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/b0ccd764fa784fda8926e026f0cec8c9
Open in Devin Desktop: https://app.devin.ai/desktop/session/b0ccd764fa784fda8926e026f0cec8c9?variant=devin

---

Note

Low Risk
Production change is an optional hibp_client defaulting to existing HIBP behavior; remaining edits are test-only determinism fixes.

Overview
Stabilizes two flaky backend tests with a small, backward-compatible production seam and deterministic test harnesses.

HIBP breach check: _hash_password_in_dict and _update_single_user_helper now accept an optional hibp_client (wired into validate_password_not_breached), matching the bulk user-update path. The breached-password update test drops respx and injects a mock AsyncHTTPHandler, so it no longer hits the real Have I Been Pwned API when mocks miss the shared httpx client.

MCP discovery timeouts: test_optional_discovery_rejects_incomplete_walks runs on a new _ManualClockLoop whose time advances only when the test calls advance, so wall-clock scheduler contention cannot fire read/walk timeouts early. The repeat parametrization raises the page cap to 3 so repeated-cursor rejection is actually exercised. Live callers omit hibp_client and behavior stays the same.

^{Reviewed by Cursor Bugbot for commit 9b97b5f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Test infrastructure changes to reduce flakiness in two backend tests.

The PR appears safe to merge; no new actionable failure remains

Summary

The PR injects an HIBP client into the single-user update path and makes two backend tests independent of network access and elapsed wall-clock time

• The breached-password test now checks the lookup method and full URL
• The MCP test uses a controlled loop clock and a bounded escape for the deadline case

_{Reviews (2) · Last reviewed commit: "test: bound MCP deadline checks, support..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_deflake_20261001 (9b97b5f) with main (5107f20)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9b97b5f. Configure here.}

[mateo-berri]

LGTM