Skip to content

test(e2e): prove the global guardrail is live before asserting the team bypass - #34641

Open
devin-ai-integration[bot] wants to merge 2 commits into
litellm_internal_stagingfrom
litellm_e2e_test_quality_fixes
Open

test(e2e): prove the global guardrail is live before asserting the team bypass#34641
devin-ai-integration[bot] wants to merge 2 commits into
litellm_internal_stagingfrom
litellm_e2e_test_quality_fixes

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Guardrail bypass test never proved the guardrail was live

How it solves it:

  • Block the same prompt for a non-team key first

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The test needs a live proxy with gemini credentials, which I don't have provisioned in this environment, so I have not run it live; the QA runbook below is the manual curl sequence it mirrors. Local checks at ac25510: make lint-e2e-basedpyright reports 0 errors and make pre-commit passes

Type

✅ Test

Changes

test_team_with_disable_flag_bypasses_global_guardrail asserted only that a key on the opted-out team got a completion. A guardrail that was never registered, never propagated to the worker, or silently broken produces the same pass, so the test could not distinguish "the opt-out works" from "there was nothing to opt out of". It now runs the existing _assert_eventually_blocked helper against scoped_key first

guardrail_id = client.create_content_filter_guardrail(...)   # default_on, bans a unique keyword
_assert_eventually_blocked(client, scoped_key, banned)       # baseline: 400 for a key outside the team
key = client.create_key_in_team(opted_out_team)
assert unwrap(client.chat(key, MODEL, _prompt_with(banned))).choices

The baseline doubles as the propagation wait the bypass request previously raced, since the helper polls to a 40s deadline while the new guardrail syncs to every worker

This PR originally also reworked TestCacheSettings::test_update_persists_cache_backend_to_get (which wrote back identical settings, so a no-op /cache/settings passed) to write a fresh namespace and read it back. #34664 landed on litellm_internal_staging first and deleted that test outright, on the grounds that no round-trip through this route is safe on a shared proxy: GET cannot express the effective config, so any write assembled from it can drop ssl / redis_startup_nodes and downgrade a TLS cluster to a plaintext node. A partial write with a changed namespace has exactly that hazard, so I took the deletion in the merge rather than resurrecting the test; the registry row stays deliberately uncovered pending LIT-4816

QA runbook

  • tests/e2e/guardrails/test_team_disable_global_guardrail_e2e.py::TestTeamDisableGlobalGuardrail::test_team_with_disable_flag_bypasses_global_guardrail - a default-on guardrail that provably blocks everyone else is bypassed by a team that opted out
    • Create a default-on litellm_content_filter guardrail banning a unique keyword via POST /guardrails (needs gemini credentials for the chat calls)
    • Send POST /v1/chat/completions with a plain key and a prompt containing that keyword; expect a 400 naming the content block (retry for up to 40s while the guardrail syncs to every worker)
    • Create a team with metadata.disable_global_guardrails set, generate a key in it, and send the same prompt; expect a 200 with choices
    • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/16e9ea5475e44d1d9d6745b2436d8bf0

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR strengthens two end-to-end regression tests.

  • It verifies that a newly written cache namespace is returned by the cache-settings endpoint.
  • It establishes that a global guardrail blocks an ordinary key before checking an opted-out team key.

Confidence Score: 3/5

This PR should not merge until the cache-settings test preserves and restores the complete original cache configuration.

The cache endpoint replaces omitted non-secret settings, while the changed test posts and restores only four fields, allowing existing settings to be cleared and an originally absent namespace to remain changed.

Files Needing Attention: tests/e2e/management/test_config_misc_endpoints_e2e.py

Important Files Changed

Filename Overview
tests/e2e/management/test_config_misc_endpoints_e2e.py Strengthens cache persistence validation, but its partial update and restoration payload can destructively alter the shared proxy's cache configuration.
tests/e2e/guardrails/test_team_disable_global_guardrail_e2e.py Adds a valid non-team baseline that proves the global guardrail is active before testing the team-level bypass.

Reviews (1): Last reviewed commit: "test(e2e): make cache-settings and guard..." | Re-trigger Greptile

resources.defer(lambda: self._write_settings(client, captured))

namespace = f"e2e-cache-ns-{unique_marker()}"
target = CacheSettingsValue(type=captured.type, host=captured.host, port=captured.port, namespace=namespace)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Partial update clears cache settings

When the shared proxy has cache settings beyond type, host, port, and namespace, this four-field update clears the omitted non-secret fields because /cache/settings replaces them rather than preserving them. The deferred restore uses the same partial model, so settings such as TTL, database, SSL, or Sentinel configuration remain removed; when the original namespace is null, serialization also omits it and leaves the generated namespace active, changing cache connectivity or key behavior for later tests.

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@devin-ai-integration devin-ai-integration Bot changed the title test(e2e): make cache-settings and guardrail-bypass tests fail on a broken product test(e2e): prove the global guardrail is live before asserting the team bypass Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant