feat(guardrails): bound every guardrail check with litellm_params.timeout

[devin-ai-integration]

TLDR

Problem this solves:

• A hung guardrail vendor holds every request for up to 600s
• timeout in guardrail litellm_params is ignored by most built-in guardrails

How it solves it:

• The proxy wraps each guardrail check in one wall-clock timeout
• On expiry the request fails with 408, or skips the guardrail with unreachable_fallback: fail_open
• No provider code changes, so every guardrail (built-in or custom) is covered the same way
• Policy pipeline steps get the same limit, and every timeout shows as status="timeout" in guardrail metrics and counts toward the guardrail errors counter, whether the request was blocked or let through
• A TimeoutError a guardrail raises itself, before the limit, is still treated as a normal guardrail error, so fail_open only applies when the proxy-side limit runs out
• custom_code keeps its own execution timeout, so it still returns the 500 exceeded its Ns execution timeout it returns on main. A guardrail class opts out of the proxy limit with enforces_own_timeout = True

This replaces the provider-by-provider approach in #43134, which threads an HTTP timeout through about 74 files. The value lives in a new hook_timeout attribute that the registry sets from litellm_params.timeout, so it never touches the self.timeout that several providers already use for their own HTTP client

User Flow

Before: when any guardrail's vendor hangs during an outage, every chat request waits with it. The steps use openai_moderation, but it's the same for every built-in or custom guardrail

1. The admin sets timeout: 2 on a guardrail, here openai_moderation, and restarts the proxy
2. The guardrail's vendor, here the moderation endpoint, stops answering
3. A developer sends POST https://litellm-domain/v1/chat/completions with that guardrail
4. The request hangs until the vendor call finally returns, 30s in this repro and up to 600s in general

After: the same request is cut off at the configured limit, whichever guardrail is hanging

1. The admin sets timeout: 2 on a guardrail, here openai_moderation, and restarts the proxy
2. The guardrail's vendor, here the moderation endpoint, stops answering
3. A developer sends POST https://litellm-domain/v1/chat/completions with that guardrail
4. After 2s it returns 408 Guardrail '<name>' did not finish within 2.0s, or with unreachable_fallback: fail_open it returns the model's normal 200 answer

Linear ticket

Resolves LIT-8333

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup: a local sink on 127.0.0.1:9911 answers /v1/moderations after sleeping 30s to simulate a hung vendor. The chat model is the real openai/gpt-5.4-mini. Both proxies use this config

model_list:
  - model_name: gpt-5.4-mini
    litellm_params:
      model: openai/gpt-5.4-mini
      api_key: os.environ/OPENAI_API_KEY
guardrails:
  - guardrail_name: moderation-fail-closed
    litellm_params:
      guardrail: openai_moderation
      mode: pre_call
      api_base: http://127.0.0.1:9911/v1
      api_key: sk-unused
      timeout: 2
  - guardrail_name: moderation-fail-open
    litellm_params:
      guardrail: openai_moderation
      mode: pre_call
      api_base: http://127.0.0.1:9911/v1
      api_key: sk-unused
      timeout: 2
      unreachable_fallback: fail_open

Request, with <guardrail> set per case

curl -s -w '\nHTTP %{http_code} in %{time_total}s\n' http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-...' -H 'Content-Type: application/json' \
  -d '{"model": "gpt-5.4-mini", "guardrails": ["<guardrail>"], "messages": [{"role": "user", "content": "Say hi in 3 words"}]}'

Before (db9c307)

fail_closed

1. Send the request with moderation-fail-closed
2. The timeout: 2 is ignored and the request waits for the sink: HTTP 200 in 30.876394s

fail_open

1. Send the request with moderation-fail-open
2. Same wait: HTTP 200 in 30.733296s

After (dbc1583)

fail_closed

1. Send the request with moderation-fail-closed
2. It fails at the limit

{"error": {"message": "litellm.Timeout: Guardrail 'moderation-fail-closed' did not finish within 2.0s", "type": "invalid_request_error", "param": null, "code": "408"}}
HTTP 408 in 2.065323s

fail_open

1. Send the request with moderation-fail-open
2. The guardrail is skipped at the limit and the real model answers

{"model": "gpt-5.4-mini", "content": "Hi there, friend.", "usage": {"total_tokens": 20}}
HTTP 200 in 3.108398s

Integration test

tests/integration/observability/test_guardrail_timeout.py runs a real proxy with owned upstreams for a stalled, a failing and a healthy moderation vendor, plus singulr and a spinning custom_code guardrail

• Merge base db9c307: 2 failed, 4 passed. The stalled fail-closed and fail-open cases wait out the 25s stall and return 200
• Head: 6 passed. Fast vendor 500s still fail, singulr returns 408 on both sides, and custom_code returns the same 500 on both sides

Unit mutations, 15 of 15 killed and green on restore: no wall-clock bound, fail_open ignored, vendor TimeoutError treated as the deadline, Prometheus timeout not counted as an error, registry drops the timeout or the fallback, registry ignores enforces_own_timeout, pipeline pre_call unbounded, during_call and post_call unbounded, metrics wrapper and pre_call status not timeout, error branch overwrites the timeout status, and post_mcp fail-open returns None

Type

🆕 New Feature

Caveats (if any)

Medium

• Configs that already set timeout now also cap the whole check
  • A check with retries or several calls can now end sooner than before
• unreachable_fallback: fail_open now also skips a guardrail that times out
  • Other network errors, such as connection refused, still follow each provider's own handling
• custom_code and any subclass setting enforces_own_timeout ignore unreachable_fallback on timeout
  • They keep main's behavior and fail with their own error
• OpenAI SDK clients retry a 408 twice by default, so a fail-closed caller waits about 3x timeout before seeing the error
  • In the Admin UI Playground the first 408 came back at 2.2s and the error showed at 11.6s
• Streaming post-call hooks are not bounded
  • They wrap the whole response stream, so a per-check limit does not fit
• Realtime apply_guardrail calls and vector-store message scans are not bounded yet
  • Direct lifecycle hooks, post_mcp_call, and policy pipeline steps are

Low

• A guardrail doing blocking sync work cannot be cancelled
  • That already stalls the event loop today
• A pre-call hook cut off with fail_open may leave partial edits on the request

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

REVIEWER MUST KNOW BEFORE APPROVING

• A guardrail with timeout set is now cut off at that limit, including policy pipeline steps
  • Before: most guardrails ignored timeout and waited for the vendor, HTTP 200 in 30.88s against a 30s stall
  • After: HTTP 408 in 2.07s with Guardrail '<name>' did not finish within 2.0s. A config that already carries a stale low timeout now cuts off checks that used to finish, including ones that retry or make several calls
  • Approved by kerry-berri, who asked for this PR in the linked Devin session
• unreachable_fallback: fail_open now lets a timed-out request through for every guardrail, and a timed-out post_mcp_call returns the unscanned tool result
  • Before: openai_moderation and most guardrails ignored it, HTTP 200 in 30.73s after the full stall
  • After: the model answers without the check, HTTP 200 in 3.11s
  • Approved by kerry-berri in the linked Devin session
• Proxy-side timeouts are recorded as status="timeout" and count toward litellm_guardrail_errors_total
  • Before: there was no timeout status, and a guardrail exception was recorded as status="error"
  • After: status="timeout" for both fail_closed and fail_open, so blocked and let-through timeouts share one label
  • Approved by kerry-berri in the linked Devin session
• singulr, and any guardrail whose own HTTP call uses the same timeout, now returns the proxy's timeout message
  • Before: HTTP 408 in 2.05s with litellm.Timeout: Connection timed out. Timeout passed=2.0, time taken=2.002 seconds
  • After: HTTP 408 in 2.04s with litellm.Timeout: Guardrail 'singulr-open-on-error' did not finish within 2.0s, and the metric status moves from error to timeout
  • Pending sign-off from kerry-berri
• A pre-call hook cut off under fail_open can hand a partly edited request to the model
  • Before: not reachable, since nothing cut a hook off mid-run
  • After: traced in code, not reproduced live
  • Pending sign-off from kerry-berri
• custom_code is exempt and does not change: HTTP 500 with exceeded its 2s execution timeout on both legs, and it still ignores fail_open on timeout
  • Approved by kerry-berri, who picked the backward-compatible option in the linked Devin session

Link to Devin session: https://app.devin.ai/sessions/ddf36d5cf5434de19370ceb10b5249b2
Open in Devin Desktop: https://app.devin.ai/desktop/session/ddf36d5cf5434de19370ceb10b5249b2?variant=devin
Requested by: @kerry-berri

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds timeout enforcement to all guardrail checks across the proxy.

The PR appears safe to merge; no new actionable finding or outstanding previous finding remains.

Summary

The PR applies configured wall-clock timeouts to non-streaming guardrail checks, including policy-pipeline steps, while preserving custom-code guardrails’ own execution timeout.

• Timeout handling supports fail-open or 408 responses and records timeout metrics.
• Unit and integration tests cover stalled, failing, and healthy guardrails.

_{Reviews (7) · Last reviewed commit: "fix(guardrails): keep custom_code on its..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_guardrail_hook_timeout (e9dbe27) with main (f5a1c9f)}

[codecov]

Codecov Report

❌ Patch coverage is 98.57143% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/custom_guardrail.py	97.61%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e9dbe27. Configure here.}

[devin-ai-integration]

Admin UI Playground on gpt-5.4-mini with a moderation sink stalled for 30s and timeout: 2

Base db9c307 fail-closed answers after 31.01s. Head e9dbe27 fail-open answers in 2.80s, and fail-closed shows the 408, delayed to 11.55s by the SDK's default retries

Base fail-closed	Head fail-closed

Base mid-wait	Head fail-open