feat(guardrails): add Compresr guardrail for query-aware context compression

[yucheng-berri]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live-proxy QA against a real proxy, real Compresr API, and real OpenAI/Anthropic traffic (captured at 43a640d, identical code to this branch's HEAD after rebase):

• /v1/chat/completions, non-stream and stream: true: works end to end. Compression applied on the wire (compressed 1 message, 277 -> 169 tokens), recovery markers and the compresr_retrieve tool injected, and the retrieve -> recover agentic loop returns the original content
• /v1/messages (Anthropic): works end to end, including the injected tool surviving format translation and the recovery round-trip
• fail_closed policy: invalid Compresr API key returns HTTP 502 with a generic redacted error, no upstream body leaked
• /v1/responses: works for plain text content targets. For tool outputs (the compress_tool_outputs: true default) compression is computed and logged but the original function_call_output text still reaches the provider; see the known limitation below

Live before/after proof of the _merge_tools_after_guardrail fix (captured at 9662a95): a local proxy with the Compresr API stubbed and the provider replaced by a request-capturing stub was sent the same /v1/responses request (one existing web_search function tool plus a bulky function_call_output) twice. With the fix in place the outgoing provider body contains tools = [web_search, compresr_retrieve]; with only the two extend lines removed the identical request produces tools = [web_search], the injected tool silently dropped. Stubs record the wire body only; no mocks inside litellm

Demo video from the original author showing the guardrail applied on a live LiteLLM gateway: https://www.youtube.com/watch?v=4Ktiwv3ka40

Known limitation: /v1/responses tool outputs

The Responses guardrail translation handler (_extract_input_text_and_images) only reads message.get("content") and never extracts the text a function_call_output item carries under output, so tool-output text never enters the guardrail texts channel and value-matched write-back cannot replace it. This is a pre-existing gap (last touched in #17338), is not introduced by this PR, and affects every apply_guardrail guardrail on /v1/responses tool-output text. The net effect for Compresr on that one surface is a no-op on the wire while tokens_saved is still reported. The handler fix (extract and write back function_call_output text) belongs in a separate PR against the shared handler; this PR documents the limitation rather than expanding scope

Type

🆕 New Feature

Changes

A new Compresr guardrail that shrinks bulky context (tool outputs, RAG chunks, search results) before the request hits the model, so callers spend fewer tokens with zero application changes. It runs on apply_guardrail and covers /chat/completions and /v1/messages fully; on /v1/responses it covers plain text content, with tool-output compression blocked by a pre-existing handler gap described in the known limitation above.

Two properties distinguish it from whole-conversation compressors. Compression is query-aware; each chunk is compressed against what produced it (a tool result against its tool call, otherwise the last user message), so the parts that matter for the turn survive. Compression is also recoverable; every compressed chunk leaves a marker and the request gets a compresr_retrieve tool, and if the model needs the original the agentic loop feeds it back. Originals live briefly in a bounded in-process store scoped per caller (no cross-tenant reads) with a TTL and byte caps, and markers are only issued for entries that survive the byte cap.

The guardrail fails closed by default (configurable), validates api_base against non-HTTP schemes and cloud-metadata hosts including encoded IP forms, redacts upstream error bodies, and sanitizes model-supplied input before logging. When nothing is compressed the request passes through untouched. It is auto-discovered via the guardrail_hooks registry and follows the same shape as the existing guardrail integrations.

The PR also carries a small fix in the shared Responses API guardrail translation handler: _merge_tools_after_guardrail previously dropped guardrail-appended tools whenever the original request already had tools. The fix preserves appended tools and is covered by a dedicated regression test; it also benefits other tool-injecting guardrails such as headroom.

Files changed: litellm/proxy/guardrails/guardrail_hooks/compresr/compresr.py (the guardrail), compresr/__init__.py (registry wiring), litellm/types/proxy/guardrails/guardrail_hooks/compresr.py (config model), litellm/types/guardrails.py (registration), litellm/llms/openai/responses/guardrail_translation/handler.py (tool merge fix), plus 78 unit tests in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_compresr.py and 2 in tests/test_litellm/llms/openai/responses/test_openai_responses_guardrail_handler.py.

Quickstart:

guardrails:
  - guardrail_name: compresr
    litellm_params:
      guardrail: compresr
      mode: pre_call
      default_on: true
      api_key: os.environ/COMPRESR_API_KEY

Behavior changes

• New opt-in guardrail integration; no behavior change unless guardrail: compresr is configured
• _merge_tools_after_guardrail in the Responses API guardrail translation handler now preserves tools a guardrail appends beyond the original tool list; previously such tools were silently dropped. This changes behavior for any guardrail that injects tools on /v1/responses requests that already carry tools (headroom recovery starts working there)
• The Compresr retrieval follow-up strips other guardrails' pre-call-executed markers from its rerun request, so input guardrails (PII redaction, content filters) re-inspect the restored uncompressed content; only Compresr's own marker is preserved to avoid recompressing what it just retrieved. The headroom guardrail's recovery loop shares the original marker-replay pattern and needs the same fix in a follow-up

Credit

Adopted from #33019 by @charafkamel. Mirrored onto a litellm_ branch so CircleCI and the internal lint workflow run

[yucheng-berri]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 92.19969% with 50 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...xy/guardrails/guardrail_hooks/compresr/compresr.py	94.04%	35 Missing ⚠️
...xy/guardrails/guardrail_hooks/compresr/__init__.py	37.50%	15 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR introduces a new opt-in Compresr guardrail that applies query-aware, recoverable context compression to bulky messages (tool outputs, RAG chunks, etc.) before they reach the LLM. It also fixes a pre-existing bug in the Responses API guardrail translation handler where tools injected by a guardrail beyond the original tool list were silently dropped.

• New CompresrGuardrail: Compresses eligible messages via the Compresr API, embeds hash markers, injects a compresr_retrieve tool, and stores originals in a bounded in-process OrderedDict (per-tenant scoping, TTL + byte caps) so the model can recover the original through the agentic loop. Fail-open/fail-closed policies, SSRF-resistant api_base validation, and multimodal content handling are all implemented defensively.
• _merge_tools_after_guardrail fix: Adds result.extend(remapped[j:]) to keep guardrail-appended tools that fall beyond the original tool count on the /v1/responses path; covered by a dedicated regression test.
• Tests: 83 new mock-only unit tests for the guardrail plus 4 new tests for the handler fix and Anthropic tool-injection path; no real network calls.

Confidence Score: 5/5

Safe to merge: new opt-in guardrail with no behavior change unless explicitly configured, plus a narrowly scoped bug fix in the Responses handler.

The new guardrail is purely additive and gated behind explicit YAML configuration. The handler fix corrects a silent tool-dropping bug and is covered by regression tests. The implementation is defensive throughout: bounded memory, TTL eviction, SSRF-resistant URL validation, provider error bodies never leaked to clients, and all 83 unit tests use mocks with no real network calls.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/compresr/compresr.py	New 1214-line Compresr guardrail: query-aware compression, bounded in-memory recovery store (OrderedDict + byte caps + TTL), SSRF-resistant api_base validation, fail_open/fail_closed policies, Anthropic/Responses/chat paths all handled.
litellm/llms/openai/responses/guardrail_translation/handler.py	Bug fix: adds result.extend(remapped[j:]) to preserve guardrail-appended tools that extend beyond the original tool count. Correct fix; covered by a dedicated regression test.
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_compresr.py	2091 lines of new unit tests (83 test functions). All tests use AsyncMock/MagicMock, no real network calls. Covers compression logic, recovery store, agentic loop, fail policies, and security edge cases.
litellm/types/proxy/guardrails/guardrail_hooks/compresr.py	Pydantic config models (CompresrGuardrailOptionalParams, CompresrGuardrailConfigModel) with full field docs.
litellm/types/guardrails.py	Registers COMPRESR in SupportedGuardrailIntegrations, adds CompresrGuardrailConfigModel to LitellmParams, and extends the unreachable_fallback docstring.

_{Reviews (4): Last reviewed commit: "chore: remove unrelated deepkeep files c..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR introduces the Compresr guardrail — a query-aware, recoverable context compressor that shrinks tool outputs and RAG chunks before the request reaches the LLM. It also fixes a pre-existing bug in _merge_tools_after_guardrail where guardrail-injected tools were silently dropped when the original request already carried tools.

• New guardrail (compresr/compresr.py, __init__.py, types): compression via the Compresr API with per-tenant in-process recovery store (TTL + byte caps), agentic loop retrieval hook, api_base SSRF hardening, fail-open/fail-closed policy, and registry wiring following the existing guardrail pattern.
• Bug fix (handler.py): _merge_tools_after_guardrail now appends remapped[j:] after the loop, so tools a guardrail appends beyond the original list are preserved for Responses API requests; covered by a dedicated regression test.
• Tests (78 compresr unit tests + 2 regression tests for the handler fix): all mocked, no real network calls.

Confidence Score: 3/5

The handler bug fix and registry wiring are clean; the main risk is that the compresr_retrieve tool is always built in OpenAI CC format and may not translate correctly for the Anthropic /v1/messages path, and Azure IMDS hostnames are absent from the SSRF blocklist.

Two open questions need resolution: whether the CC-format compresr_retrieve tool survives provider format translation for Anthropic (if not, markers are embedded but the model can never call the recovery tool on that path), and whether the api_base SSRF blocklist should include Azure IMDS hostnames. Neither is tested or documented as a deliberate omission.

litellm/proxy/guardrails/guardrail_hooks/compresr/compresr.py — specifically _build_compresr_retrieve_tool (CC format vs. Anthropic format) and _BLOCKED_METADATA_HOSTS (missing Azure IMDS hostnames).

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/compresr/compresr.py	Core guardrail implementation (~1 180 lines): compression dispatch, in-process recovery store with TTL+byte caps, agentic-loop retrieval, SSRF defence on api_base. Azure IMDS hostnames missing from the blocked list; the injected compresr_retrieve tool is in OpenAI CC format and may not translate correctly for the Anthropic /v1/messages path.
litellm/llms/openai/responses/guardrail_translation/handler.py	Bug fix: _merge_tools_after_guardrail now appends remapped[j:] after the original-tools loop, preventing silent loss of guardrail-injected tools when the request already has tools. Change is correct and covered by a dedicated regression test.
litellm/proxy/guardrails/guardrail_hooks/compresr/init.py	Registry wiring and initializer factory; maps COMPRESR enum to initialize_guardrail and CompresrGuardrail. No issues found.
litellm/types/proxy/guardrails/guardrail_hooks/compresr.py	Pydantic config model for all tunable parameters; well-documented. No issues found.
litellm/types/guardrails.py	Adds COMPRESR to SupportedGuardrailIntegrations enum and inherits CompresrGuardrailConfigModel into LitellmParams. Follows existing pattern exactly; no issues found.
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_compresr.py	78 unit tests; all network calls are mocked via MagicMock/AsyncMock/patch — no real HTTP. Covers compression, recovery, agentic loop, fail-open/closed, bypass header, and multimodal content. Solid coverage.
tests/test_litellm/llms/openai/responses/test_openai_responses_guardrail_handler.py	Regression tests for the tool-merge fix and streaming handler edge cases; properly mocked, no live network calls.

_{Reviews (1): Last reviewed commit: "refactor(guardrails): extract _existing_..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_compresr_guardrail (9662a95) with litellm_internal_staging (b2202cb)}

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

Chat recovery hooks never run

High Severity

The CompresrGuardrail's agentic loop for content recovery is not handled for /v1/chat/completions. The guardrail only overrides agentic loop methods for /v1/messages and /v1/responses, leaving the chat completion-specific hooks unhandled. This prevents the compresr_retrieve tool from fulfilling requests for chat completions.

 

^{Reviewed by Cursor Bugbot for commit 8ddf97f. Configure here.}

[yucheng-berri]

This is not correct; the hooks this guardrail overrides are the chat completions hooks. litellm/litellm_core_utils/chat_completion_agentic_loop.py invokes async_should_run_agentic_loop (line 219) and async_build_agentic_loop_plan (line 260), exactly the two methods CompresrGuardrail implements, and lines 32-39 of that file check for these overrides to decide whether a callback participates in the chat completions loop. There is no separate chat-specific hook left unimplemented. async_build_agentic_loop_plan's else branch builds chat-format follow-ups (assistant message plus role tool results), the same contract the existing headroom guardrail ships on. Live QA also verified the full retrieve -> recover round-trip on /v1/chat/completions (stream and non-stream) with real traffic; see the proof section in the PR body.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9662a95. Configure here.}

[yuneng-berri]

Can I get more context on this? Why do we need to modify the base guardrail translation handler for this integration?

[yucheng-berri]

I think this change the base handler because it is the code that rebuilds the tool list after a guardrail runs, and it currently drops the compresr_retrieve tool that Compresr adds.

[yucheng-berri]

do you have any concerns

[yuneng-berri]

the potential blast radius for this PR now much higher.

rebuilds the tool list after a guardrail runs

Have you check if this would affect any other guardrail integration as well as our custom one?

[yucheng-berri]

This is an existing pattern with /chat/completions and /messages, where the full tool list returned by the guardrail is preserved.

1. /v1/chat/completions — litellm/llms/openai/chat/guardrail_translation/handler.py:133-135
2. /v1/messages (Anthropic) — litellm/llms/anthropic/chat/guardrail_translation/handler.py:332-342