Skip to content

test: fix order-dependent flake in passthrough guardrail call-type test - #35317

Merged
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_passthrough_guardrail_flake
Jul 31, 2026
Merged

test: fix order-dependent flake in passthrough guardrail call-type test#35317
mateo-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_passthrough_guardrail_flake

Conversation

@mateo-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • test_pass_through_call_type_resolved_from_logging_obj flakes in the proxy-infra shard
  • it patches a loader that a cached module global bypasses
  • worse, it leaks its mock mapping into that global

How it solves it:

  • patch the module global directly, like sibling tests already do
  • unittest.mock restores the global, so nothing leaks either way

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This fixes a unit test that is order-dependent, so the end-to-end surface is the CI test run itself; the failure it eliminates is this proxy-infra job failure on an unrelated PR, where the test failed plus both its reruns on worker gw0 after test_proxy_logging_hook_detection.py ran first on the same worker

Deterministic reproduction before the fix, at commit 8e28765 (the staging base this branch is cut from), using the same xdist mode CI uses. When the polluter module runs first, the flaky test fails exactly like CI:

$ pytest tests/test_litellm/proxy/test_proxy_logging_hook_detection.py \
    tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_post_call_guardrails.py \
    -n 1 --dist=loadscope -q
E   AssertionError: Expected process_output_response to have been awaited once. Awaited 0 times.
1 failed, 20 passed in 6.25s

The leak also fires in the opposite order at 8e28765. Single-process runs sort tests by name, so the flaky test runs first and permanently replaces the mapping global with its mock, which silently disables unified guardrails for every other call type in that worker; three unrelated guardrail-block tests then fail open:

$ pytest tests/test_litellm/proxy/test_proxy_logging_hook_detection.py \
    tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_post_call_guardrails.py -q
E   Failed: DID NOT RAISE <class 'fastapi.exceptions.HTTPException'>
FAILED ...test_post_call_stream_guardrail_blocks_anthropic_messages_stream
FAILED ...test_post_call_stream_guardrail_reroutes_inherited_apply_guardrail
FAILED ...test_unified_guardrail_iterator_accepts_explicit_guardrail
3 failed, 13 passed in 0.35s

After the fix, at commit 47ebc96, both orderings and the file in isolation pass:

$ pytest ... -n 1 --dist=loadscope -q   # CI direction
21 passed in 30.95s
$ pytest ... -q                          # reverse direction
21 passed in 3.03s
$ pytest tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_post_call_guardrails.py -q
6 passed in 0.25s

Type

✅ Test

Changes

unified_guardrail.py caches load_guardrail_translation_mappings() in the module global endpoint_guardrail_translation_mappings, populated lazily on first hook call and never reset. The flaky test patched the loader function, which is only consulted while the global is still None; whenever any earlier test in the same xdist worker had exercised a unified guardrail hook (in the failing CI run, the streaming-block tests in test_proxy_logging_hook_detection.py), the real mappings were already cached, the patch was a no-op, and the real pass-through handler ran instead of the mock, failing assert_awaited_once on all three attempts since pytest-rerunfailures reruns in the same process

The fix follows the convention already used by test_blocked_response_usage.py and test_unified_guardrail.py: patch the module global itself with the mock mapping via patch.object, which both makes the test immune to whatever ran before it and restores the previous value afterwards, closing the reverse leak where the test's mock mapping poisoned the global for later tests in the worker

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes the passthrough guardrail call-type test order-independent.

  • Replaces the loader-function mock with a scoped patch of the cached module-level translation mapping.
  • Preserves the test’s call-type resolution coverage while ensuring the prior global value is restored afterward.

Confidence Score: 5/5

The PR appears safe to merge because the scoped mock targets the actual cached state and restores it after the test.

The modified test continues to verify passthrough call-type resolution and handler invocation while removing dependence on whether another test initialized the module-level cache first.

Important Files Changed

Filename Overview
tests/test_litellm/proxy/pass_through_endpoints/test_passthrough_post_call_guardrails.py The test now patches the exact cached mapping consumed by the hook, avoiding order-dependent behavior without weakening its assertions.

Reviews (1): Last reviewed commit: "test: patch unified guardrail mapping gl..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri
mateo-berri merged commit 450e173 into litellm_internal_staging Jul 31, 2026
75 checks passed
@mateo-berri
mateo-berri deleted the litellm_fix_passthrough_guardrail_flake branch July 31, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants