Skip to content

fix(proxy): gate disable_global_guardrails on keys and teams to proxy admins - #42699

Merged
yucheng-berri merged 12 commits into
mainfrom
litellm_gate_disable_global_guardrails_admin
Sep 24, 2026
Merged

yucheng-berri merged 12 commits into
mainfrom
litellm_gate_disable_global_guardrails_admin

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

This fixes a bypass where non-admins could set disable_global_guardrails: true on a key or team and skip global guardrails.

After this PR, only proxy admins can enable disable_global_guardrails: true, including through metadata. Non-admins can keep an exemption that was already granted by an admin, but they can’t create a new one. They can only flip true → false

The UI now hides the option for non-admins, and attempts to enable it return 403.

User Flow

Before: an internal user who is a team admin creates a key that skips the default-on guardrail

  1. They send POST https://litellm-domain/key/generate with their own key and {"team_id": "<team>", "disable_global_guardrails": true}
  2. They get 200 and a new sk-... key whose metadata shows "disable_global_guardrails": true
  3. They send POST https://litellm-domain/v1/chat/completions with that key and a prompt the guardrail would block
  4. They get 200 and a real model answer; the guardrail never ran
  5. Any non-admin could do the same on POST /key/update and POST /key/regenerate, and on POST /team/new

After: the same request is refused and only a proxy admin can grant the exemption

  1. They send POST https://litellm-domain/key/generate with their own key and {"team_id": "<team>", "disable_global_guardrails": true}
  2. They get 403 Only proxy admins can set disable_global_guardrails on a key.
  3. The same 403 comes back when the flag is sent under metadata, or on POST /key/update, POST /key/regenerate, POST /key/service-account/generate and POST /team/new
  4. A key they create without the flag still gets 400 synthetic policy denial from the default-on guardrail on POST /v1/chat/completions
  5. A proxy admin sends the same POST /key/generate with the flag and gets 200; that key still bypasses the guardrail
  6. The non-admin can still rename that admin-exempted key through the edit form: POST /key/update with the full stored metadata echoed back returns 200 and the stored flag is untouched
  7. On https://litellm-domain/ui/?page=api-keys the create and edit key forms no longer show the Disable Global Guardrails switch to non-admins. Before and after screenshots of both forms as an internal user and as a proxy admin are in the PR comment fix(proxy): gate disable_global_guardrails on keys and teams to proxy admins #42699 (comment)

Relevant issues

Affected release

Linear ticket

Resolves LIT-7702

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup, identical on both sides: real proxy with Postgres and Redis, master key creates an internal_user, a team with that user as team admin, a personal key for that user (the caller, allowed routes /key/generate, /key/update, /key/regenerate) and a plain team key. Chat cases use openai/gpt-4o-mini against the real OpenAI API through a local forwarding recorder that counts upstream calls, plus a default_on generic_guardrail_api guardrail pointed at a local sink that always returns BLOCKED. Tokens are redacted

Before (40ec84c)

Non-admin sets the flag on /key/generate

  1. curl -X POST http://127.0.0.1:14010/key/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","disable_global_guardrails":true}'
  2. HTTP_STATUS=200, body "metadata":{"disable_global_guardrails":true}

Non-admin smuggles the flag under metadata

  1. curl -X POST http://127.0.0.1:14010/key/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","metadata":{"disable_global_guardrails":true}}'
  2. HTTP_STATUS=200

Non-admin sets the flag on /key/update

  1. curl -X POST http://127.0.0.1:14010/key/update -H 'Authorization: Bearer <caller>' -d '{"key":"<team-key>","disable_global_guardrails":true}'
  2. HTTP_STATUS=200

Non-admin sets the flag on /key/regenerate

  1. curl -X POST http://127.0.0.1:14010/key/regenerate -H 'Authorization: Bearer <caller>' -d '{"key":"<team-key>","disable_global_guardrails":true}'
  2. HTTP_STATUS=200

Non-admin sets the flag on /key/service-account/generate and /team/new

  1. curl -X POST http://127.0.0.1:14010/key/service-account/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","service_account_id":"svc","disable_global_guardrails":true}'
  2. HTTP_STATUS=200
  3. curl -X POST http://127.0.0.1:14010/team/new -H 'Authorization: Bearer <caller>' -d '{"team_alias":"t2","disable_global_guardrails":true}'
  4. HTTP_STATUS=200

Admin-exempted key, then non-admin re-saves it with the stored metadata echoed back

  1. master: curl -X POST http://127.0.0.1:14010/key/generate -d '{"team_id":"<team>","disable_global_guardrails":true}' returns HTTP_STATUS=200
  2. curl -X POST http://127.0.0.1:14010/key/update -H 'Authorization: Bearer <caller>' -d '{"key":"<exempt-key>","key_alias":"renamed","metadata":{"disable_global_guardrails":true}}'
  3. HTTP_STATUS=200, stored flag still true
  4. curl -X POST http://127.0.0.1:14010/v1/chat/completions -H 'Authorization: Bearer <exempt-key>' -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"synthetic denied marker"}]}' returns HTTP_STATUS=200 with real usage, recorder shows 1 upstream call

Plain team key hits the default-on guardrail

  1. curl -X POST http://127.0.0.1:14010/v1/chat/completions -H 'Authorization: Bearer <team-key>' -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"synthetic denied marker"}]}'
  2. HTTP_STATUS=400, {"error":{"message":"synthetic policy denial",...,"code":"400"}}, recorder shows 0 upstream calls

After (2cfb50e)

Non-admin sets the flag on /key/generate

  1. curl -X POST http://127.0.0.1:14000/key/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","disable_global_guardrails":true}'
  2. HTTP_STATUS=403, {"error":{"message":"{'error': 'Only proxy admins can set disable_global_guardrails on a key.'}",...,"code":"403"}}

Non-admin smuggles the flag under metadata

  1. curl -X POST http://127.0.0.1:14000/key/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","metadata":{"disable_global_guardrails":true}}'
  2. HTTP_STATUS=403, same message

Non-admin sets the flag on /key/update

  1. curl -X POST http://127.0.0.1:14000/key/update -H 'Authorization: Bearer <caller>' -d '{"key":"<team-key>","disable_global_guardrails":true}'
  2. HTTP_STATUS=403, same message

Non-admin sets the flag on /key/regenerate

  1. curl -X POST http://127.0.0.1:14000/key/regenerate -H 'Authorization: Bearer <caller>' -d '{"key":"<team-key>","disable_global_guardrails":true}'
  2. HTTP_STATUS=403, same message

Non-admin sets the flag on /key/service-account/generate and /team/new

  1. curl -X POST http://127.0.0.1:14000/key/service-account/generate -H 'Authorization: Bearer <caller>' -d '{"team_id":"<team>","service_account_id":"svc","disable_global_guardrails":true}'
  2. HTTP_STATUS=403, {"detail":{"error":"Only proxy admins can set disable_global_guardrails on a key."}}
  3. curl -X POST http://127.0.0.1:14000/team/new -H 'Authorization: Bearer <caller>' -d '{"team_alias":"t2","disable_global_guardrails":true}'
  4. HTTP_STATUS=403, Only proxy admins can set disable_global_guardrails on a team.

Admin-exempted key, then non-admin re-saves it with the stored metadata echoed back

  1. master: curl -X POST http://127.0.0.1:14000/key/generate -d '{"team_id":"<team>","disable_global_guardrails":true}' returns HTTP_STATUS=200, "metadata":{"disable_global_guardrails":true}
  2. curl -X POST http://127.0.0.1:14000/key/update -H 'Authorization: Bearer <caller>' -d '{"key":"<exempt-key>","key_alias":"lr-head-2cfb50e902-ex-renamed","metadata":{"disable_global_guardrails":true}}'
  3. HTTP_STATUS=200, "key_alias":"lr-head-2cfb50e902-ex-renamed", stored flag still true
  4. curl -X POST http://127.0.0.1:14000/v1/chat/completions -H 'Authorization: Bearer <exempt-key>' -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"synthetic denied marker"}]}' returns HTTP_STATUS=200, "model":"gpt-4o-mini", "usage":{"completion_tokens":5,"prompt_tokens":19,...}, recorder shows 1 upstream call

Plain team key hits the default-on guardrail

  1. curl -X POST http://127.0.0.1:14000/v1/chat/completions -H 'Authorization: Bearer <team-key>' -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"synthetic denied marker"}]}'
  2. HTTP_STATUS=400, {"error":{"message":"synthetic policy denial","type":"invalid_request_error","param":null,"code":"400"}}, recorder shows 0 upstream calls

Integration regression

tests/integration/authorization/test_key_guardrail_opt_out.py::test_non_admin_cannot_opt_key_out_of_default_on_guardrail (selected through the authorization group in tests/integration/run.py) runs a real proxy with Postgres, Redis and a local scripted guardrail sink. Against a proxy started from 40ec84caa2 it fails at assert generated.status_code == 403 with 200. Against 0ce0fdbd5b it passes

Audit at 0ce0fdb

17 deterministic cells across tests/integration/authorization/test_key_guardrail_opt_out.py, test_key_guardrail_opt_out_runtime.py and test_key_guardrail_opt_out_chaos.py (no covers markers, since the contracts.json manifest was dropped on main in #42711 and this branch merged that at 80ba786dc3; the slow sink check bounds elapsed time by the configured sink delay rather than a fixed constant): every key and team write route with the flag top-level and smuggled, hostile inputs, server-default metadata, resave, omit and revoke sequences, runtime denial and exemption on chat, messages and responses across streaming and the OpenAI, AsyncOpenAI and Anthropic SDKs, team-level flag, spend logs, 40 concurrent writes, cross-worker revocation, guardrail sink outage and slow sink, and a SIGKILLed uvicorn worker. Real proxy, Postgres and Redis, scripted upstream, no real providers. Base e26a6450c8 (current merge base): 9 failed, all at assert 200 == 403 on flag writes, 8 passed. Head 0ce0fdbd5b twice: 17 passed, 17 passed. Not reachable as written: a team admin hitting the new /team/update check, because the team-admin editable-fields gate returns 403 first on both base and head since the flag is not in SUPPORTED_TEAM_ADMIN_EDITABLE_TEAM_FIELDS

The management integration shard (tests/integration/run.py management, seed 4106601) was also run locally at 2cfb50e902 against a fresh proxy pair, Postgres and Redis: 46 passed, 2 failed, all 17 nodes from this PR passed. The 2 failures are owned_redis startup in two unrelated Redis-outage tests because the local Redis is 6.0.16 and the harness passes --set-proc-title, a Redis 7 flag; CircleCI publishes no check on this PR head so that local run stands in for the shard

Mutation check

Stubbing _check_disable_global_guardrails_caller_permission to return makes 11 of the 18 new unit tests fail (every 403 case across generate, update, regenerate, team new and team update, plus the stored-false and explicit-false-with-metadata-true cases); the 7 allow cases stay green. Restoring the helper: 18 passed. Removing isProxyAdminRole(userRole) from either UI form fails the new "hides the switch from a non-admin" tests

Type

🐛 Bug Fix

Caveats (if any)

Severe

  • Non-admins who set disable_global_guardrails today now get 403 on key and team write routes

Medium

Low

  • On /key/regenerate a non-premium non-admin sees the enterprise error before the 403, since the row load sits behind the premium gate
  • A non-admin can still turn an admin-set exemption off by updating a key without the flag; guardrails only ever turn back on
  • Live risk A/B (real OpenAI, real Postgres, zero mocks) at 0ce0fdbd5b vs e26a6450c8 (identical results to the earlier 2cfb50e902 vs 40ec84caa2 run): only the intended 403s differ; /key/bulk_update (already admin-only), /team/key/bulk_update (schema forbids the flag) and PATCH /team/{id} (team-admin editable-fields gate fires first) are unchanged

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/b4c09d500c5c4c8ab366e8eae782ac40
Open in Devin Desktop: https://app.devin.ai/desktop/session/b4c09d500c5c4c8ab366e8eae782ac40?variant=devin
Requested by: @yucheng-berri


Note

High Risk
Changes authorization on guardrail opt-out for keys and teams; non-admins lose a previously allowed path to bypass default-on guardrails, so deploys may break workflows that relied on team admins setting the flag.

Overview
Restricts who can opt keys and teams out of default-on global guardrails. Non–proxy-admins now get 403 when they try to set disable_global_guardrails (top-level or via metadata) on key generate, update, regenerate (including path-based regenerate), service-account generate, and team new / update. Proxy admins are unchanged.

A shared _check_disable_global_guardrails_caller_permission helper mirrors the existing passthrough-routes admin gate: truthy values are blocked (including smuggled metadata), key generate checks caller-supplied metadata before server defaults apply, and updates allow non-admins to re-send an already-stored true so admin UI edits do not break.

The dashboard hides the “Disable Global Guardrails” switch for non-admins on key create/edit and team create/edit; API docs/schema strings now say proxy admin only. Broad unit and integration coverage exercises denial paths, admin success, resave/revoke, concurrency, and runtime guardrail skip behavior.

Reviewed by Cursor Bugbot for commit 0ce0fdb. Bugbot is set up for automated code reviews on this repo. Configure here.

yucheng-berri and others added 2 commits September 23, 2026 07:12
… admins

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...y/management_endpoints/key_management_endpoints.py 88.88% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The latest head appears safe to merge; no new findings remain, and all previous review findings are resolved.

Summary

This PR adds security hardening around privileged guardrail configuration and aligns the dashboard with the server-side authorization policy.

  • Centralizes permission checks for key and team management routes.
  • Preserves compatible update behavior for previously stored configuration and server-provided defaults.
  • Hides privileged controls from non-admin dashboard users.
  • Adds unit, integration, runtime, concurrency, and failure-mode coverage.

Reviews (5) · Last reviewed commit: "test(integration): drop covers markers a..."

greptile-apps[bot]

This comment was marked as resolved.

yucheng-berri and others added 2 commits September 23, 2026 07:38
…guardrail opt-out helper

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@codspeed

codspeed Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_gate_disable_global_guardrails_admin (0ce0fdb) with main (5beac4f)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (966f695) during the generation of this report, so 5beac4f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…ot server defaults

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review at b2c3fad, the server-default metadata finding is addressed there

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread litellm/proxy/management_endpoints/common_utils.py
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Admin UI evidence at b2c3fad vs 40ec84c: the create and edit key forms hide the switch from an internal user, admins keep it

Before (40ec84c) and After (b2c3fad)
Before, internal user sees the switch After, internal user has no switch
before internal create after internal create
After, admin enables it After, admin-created exemption persisted
after admin create after admin persisted
After, internal edit form, no switch After, internal rename kept the exemption
after internal edit after internal renamed

yucheng-berri and others added 4 commits September 23, 2026 08:55
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@yucheng-berri

Copy link
Copy Markdown
Contributor

@greptileai review latest head

@yucheng-berri

Copy link
Copy Markdown
Contributor

bugbot run

Comment thread tests/integration/authorization/test_key_guardrail_opt_out.py Outdated
Comment thread tests/integration/authorization/test_key_guardrail_opt_out_chaos.py Outdated

@cursor cursor Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…y admins

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Team forms at 486a0c6: switch hidden for non-admins, kept for proxy admins, unrelated team saves still work

Before (e26a645) and after (486a0c6), internal user who is org admin and team admin
Before After
Create Team before Create Team after
Team Settings before Team Settings after
Proxy admin at 486a0c6 and non-admin alias save
Admin Create Team Admin Team Settings Alias save persisted
Admin create Admin edit Alias

…to the sink delay

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai review latest head 0ce0fdb

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

bugbot run

Copy link
Copy Markdown
Contributor

@greptileai review latest head

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0ce0fdb. Configure here.

@yucheng-berri
yucheng-berri merged commit cad49ee into main Sep 24, 2026
104 of 105 checks passed
@yucheng-berri
yucheng-berri deleted the litellm_gate_disable_global_guardrails_admin branch September 24, 2026 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants