feat(guardrails): add TypeSafe Jev relevance-based compaction guardrail

[devin-ai-integration]

TLDR

Problem this solves:

• Compaction today means a summarization prompt: slow, lossy, extra LLM spend
• Stale tool results keep eating prompt tokens across long agent loops

How it solves it:

• New typesafe guardrail scores each completed tool exchange with TypeSafe's Jev model
• One POST /v1/systemone call, one yes/no question per exchange, no summarization
• Tool results Jev judges irrelevant are blanked; system, last user, and last exchange are never touched
• Opt in via guardrail: typesafe, mode: pre_call, or add it from the Admin UI as "TypeSafe (Jev) Compaction"
• Fails open by default

User Flow

Before: an agent developer with a long tool-call history sends the whole thing to the model every turn and pays for all of it

1. The proxy admin opens http://localhost:3000/guardrails/, clicks Add New Guardrail, and searches the provider list for "TypeSafe": it shows "No matching providers"
2. A guardrail saved under that name in the DB is skipped at proxy boot with Unsupported guardrail: typesafe
3. The developer sends POST http://localhost:4000/v1/chat/completions with "guardrails": ["typesafe-compaction"] and three finished tool exchanges (weather, stock quote, deployment runbook) plus a final question about the runbook
4. The response is 200 with prompt_tokens: 604; every stale tool result was billed
5. http://localhost:3000/logs/ shows the request with all three tool results intact and no guardrail evaluation

After: the same request drops the two off-topic tool results before it reaches the model

1. The proxy admin opens http://localhost:3000/guardrails/, clicks Add New Guardrail, picks "TypeSafe (Jev) Compaction", enters os.environ/TYPESAFE_API_KEY as the api_key, keeps pre_call, and clicks Create Guardrail
2. The guardrail list shows typesafe-compaction with provider TypeSafe (Jev) Compaction, mode pre_call, Default Off
3. The developer sends the same POST http://localhost:4000/v1/chat/completions with "guardrails": ["typesafe-compaction"]
4. The response is 200 with the same correct answer and prompt_tokens: 406; the weather and stock results were replaced by a short removal marker, the runbook result was kept
5. http://localhost:3000/logs/ shows Pre-call guardrail: typesafe-compaction PASSED, the raw guardrail response exchanges_evaluated: 2, exchanges_dropped: 2, chars_removed: 744, and the two replaced tool messages in the request history
6. In http://localhost:3000/playground/ the developer can pick typesafe-compaction in the Guardrails selector and every chat sent from there runs through it

Relevant issues

Affected release

Linear ticket

Resolves LIT-8088

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both runs: Postgres-backed proxy on port 4000 started with python litellm/proxy/proxy_cli.py --config ~/typesafe_qa_config.yaml --detailed_debug --reload --use_v2_migration_resolver, Admin UI dev server (npm run dev in ui/litellm-dashboard) on port 3000 at the PR tip, TYPESAFE_API_KEY and OPENAI_API_KEY exported in the proxy shell. The guardrail is created once through the UI (After, case 1) and stored in the DB; the Before run boots the merge-base proxy against the same DB and config. Real OpenAI (gpt-5.6) and TypeSafe (jev-latest) calls, no mocks

# ~/typesafe_qa_config.yaml
model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: sk-1234
  database_url: os.environ/DATABASE_URL
  store_model_in_db: true
litellm_settings:
  store_prompts_in_spend_logs: true

The request payload (~/typesafe_payload.json) carries a system prompt, a user task, three completed tool exchanges (a Lisbon weather report, an ACME stock quote, and a deployment runbook, each 400 to 600 characters), and a final user question that only the runbook answers. It ends with "tool_choice": "none" so the model answers instead of calling another tool

typesafe_payload.json

{
  "model": "gpt-5.6",
  "guardrails": ["typesafe-compaction"],
  "max_completion_tokens": 500,
  "tool_choice": "none",
  "messages": [
    {"role": "system", "content": "You are a helpful operations assistant. Answer the latest user question using the relevant tool result. Be concise and do not invent values."},
    {"role": "user", "content": "First check the weather in Lisbon and a stock quote, then read the deployment runbook. I will ask a follow-up about the runbook."},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_weather", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\":\"Lisbon\"}"}}]},
    {"role": "tool", "tool_call_id": "call_weather", "content": "Lisbon weather station report: temperature 23 C, feels like 24 C, humidity 61 percent, wind from the northwest at 14 km/h. Skies are mostly clear with scattered coastal clouds. The afternoon forecast expects a high of 26 C and less than 5 percent chance of rain. Sunset is at 19:42 local time. Visibility is 12 km and barometric pressure is steady at 1017 hPa. Outdoor walking conditions are comfortable; bring a light jacket for the evening waterfront breeze."},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_stock", "type": "function", "function": {"name": "get_stock_quote", "arguments": "{\"symbol\":\"ACME\"}"}}]},
    {"role": "tool", "tool_call_id": "call_stock", "content": "Market quote for synthetic ticker ACME: last trade 142.35 USD, change +1.12 USD, daily change +0.79 percent. The session opened at 141.20, reached a high of 143.10 and a low of 140.85. Trading volume is 2,430,000 shares, with a bid of 142.34 and ask of 142.36. The prior close was 141.23. The 52-week trading range is 112.40 to 156.80. This delayed quote is intended for a market overview only and does not contain infrastructure settings or deployment instructions."},
    {"role": "assistant", "content": null, "tool_calls": [{"id": "call_runbook", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"/runbooks/deployment.md\"}"}}]},
    {"role": "tool", "tool_call_id": "call_runbook", "content": "Deployment runbook, revision 12. The production application is deployed in AWS region eu-west-1. Operational request logs must be retained for 30 days and then deleted by the lifecycle policy. Deployments use a blue-green rollout with readiness checks before traffic switches. On-call engineers verify health metrics and error rates during the first fifteen minutes. Roll back to the previous image if the error budget alert fires. Database backups run daily and are verified separately."},
    {"role": "user", "content": "Ignore the earlier weather and stock quote. According only to the deployment runbook, which AWS region is production in and how many days are operational request logs retained? Answer in one sentence."}
  ],
  "tools": [
    {"type": "function", "function": {"name": "get_weather", "description": "Get a city weather report", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
    {"type": "function", "function": {"name": "get_stock_quote", "description": "Get a market quote", "parameters": {"type": "object", "properties": {"symbol": {"type": "string"}}, "required": ["symbol"]}}},
    {"type": "function", "function": {"name": "read_file", "description": "Read an operations runbook", "parameters": {"type": "object", "properties": {"path": {"type": "string"}}, "required": ["path"]}}}
  ]
}

Before (db37977)

Admin UI: register the guardrail

1. Booted the merge-base proxy against the DB that already holds the typesafe-compaction guardrail created in the After run. Startup skipped it:

LiteLLM Proxy:ERROR: proxy_server.py:8070 - ProxyConfig:_init_guardrails_in_db - skipping guardrail 'typesafe-compaction' (ID: 17dcc0d0-...): ValueError: Guardrail initialization failed: Unsupported guardrail: typesafe

2. Opened http://localhost:3000/guardrails/, clicked Add New Guardrail, then Add Provider Guardrail, and typed TypeSafe into the Guardrail Provider dropdown. Observed: "No matching providers"

Request logs

1. Sent the payload:

curl -sS -w 'HTTP %{http_code}\n' http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  --data-binary @~/typesafe_payload.json

Observed: HTTP 200, prompt_tokens: 604, completion_tokens: 25, answer "Production is in AWS region eu-west-1, and operational request logs are retained for 30 days." The unknown guardrail name is silently ignored

2. Opened http://localhost:3000/logs/ and clicked the request. Observed: Input Tokens 604, all three tool results intact in History, and no guardrail evaluation in the Request Lifecycle

Playground

1. Opened http://localhost:3000/playground/, endpoint /v1/chat/completions, model gpt-5.6, and opened the Guardrails selector. Observed: the DB-stored typesafe-compaction name is listed because the dashboard reads the guardrails table, even though the proxy could not load it

2. Sent "Reply with exactly: TypeSafe playground ready" with it selected. Observed: the reply arrived, and its log entry shows no guardrail ran

After (0765f6d)

Admin UI: register the guardrail

1. Opened http://localhost:3000/guardrails/, clicked Add New Guardrail, then Add Provider Guardrail, named it typesafe-compaction, and picked "TypeSafe (Jev) Compaction" from the provider dropdown

2. Kept mode pre_call and Always On No, entered os.environ/TYPESAFE_API_KEY as api_key, https://api.typesafe.ai as api_base, jev-latest as model, fail_open as unreachable_fallback, and clicked Next

3. Filled the optional parameters relevance_threshold 0.2, min_chars_to_evaluate 200, max_result_chars_in_state 4000 and clicked Create Guardrail. Observed: "Guardrail created successfully"

4. Observed the list row: typesafe-compaction, provider TypeSafe (Jev) Compaction, mode pre_call, Default Off

5. Clicked the row. Observed: Overview and Settings tabs show the saved provider settings with the API key masked and the optional values persisted

Request logs

1. Sent the same payload with the same curl command. Observed: HTTP 200, prompt_tokens: 406, completion_tokens: 25, the same answer "Production is in AWS region eu-west-1, and operational request logs are retained for 30 days." Prompt tokens fell 604 -> 406 (about 33%). The proxy logged:

LiteLLM Proxy:INFO: typesafe.py:388 - TypeSafe: evaluated 2 tool exchange(s), dropped 2, ~744 chars removed

2. Sent the same payload without the guardrails field for comparison. Observed: HTTP 200, prompt_tokens: 604. Both rows in http://localhost:3000/logs/: 629 total tokens (604 + 25) unguarded, 431 (406 + 25) guarded

3. Clicked the guarded request. Observed: Request Lifecycle shows Pre-call guardrail: typesafe-compaction PASSED and the Raw Guardrail Response is {"model": "jev-latest", "chars_removed": 744, "exchanges_dropped": 2, "exchanges_evaluated": 2}

4. Expanded Request & Response, History (7 messages). Observed: the weather and stock tool results read [Tool result removed by TypeSafe compaction: judged no longer relevant to the current task], the runbook result, system prompt, and final user question are unchanged

Playground

1. Opened http://localhost:3000/playground/, endpoint /v1/chat/completions, model gpt-5.6, and picked typesafe-compaction in the Guardrails selector

2. Sent "Reply with exactly: TypeSafe playground ready". Observed: the reply "TypeSafe playground ready"

3. Opened the request in http://localhost:3000/logs/. Observed: Pre-call guardrail: typesafe-compaction PASSED in the lifecycle. The Playground chat has no completed tool exchanges (no MCP server was configured in this environment), so the guardrail had nothing to evaluate and forwarded the request unchanged

The run before this one (tip 22e6947) was uncompacted: the registries had been wrapped in MappingProxyType for the LIT002 gate and guardrail_registry.py only discovers plain dicts, so the proxy skipped the guardrail while CI stayed green. 0765f6d reverts that and adds test_typesafe_initializer_discoverable_via_hook_registries, which fails on 22e6947 and passes now

Taxonomy audit

O3: evaluator outage cannot widen anything, it only forwards the request uncompacted and logs guardrail_failed_to_respond. O4: pre_call only by design, response and streaming paths never compacted. W1: input structured_messages and rows are never mutated, compaction returns new objects (tests assert identity on the no-op paths). X1: is None checks throughout, the only or fallback is api_key/api_base where empty means unset. V1: api_base is operator config, never request controlled. H7: buildAutoRouterCompression.ts changed but the prebuilt _experimental/out bundle is refreshed by the scheduled build-artifacts automation, so the served UI lists typesafe after the next artifact run. Q3: the Jev call is an unattributed outbound call, same as compresr and headroom. H5: DROPPED_RESULT_TEXT and defaults live in the guardrail module, matching compresr

Type

🆕 New Feature

Caveats (if any)

Medium

• Only OpenAI-shaped tool rows are blanked; Anthropic tool_result blocks pass through untouched
• Adds one Jev round trip (30s timeout) to every request that has a droppable exchange

Low

• unreachable_fallback defaults to fail_open here, unlike other guardrails, since compaction is an optimization
• Default threshold 0.2 and question wording were tuned on one live sample only
• A request with no eligible exchange shows the guardrail as PASSED in Logs but records no counters (no exchanges_evaluated: 0 entry)
• Playground compaction was not exercised with real tool exchanges: the Playground only produces them through MCP servers and none was configured
• Docs page lives in a separate litellm-docs PR

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/6e41d4677f5a497289d8e0189b261a63
Open in Devin Desktop: https://app.devin.ai/desktop/session/6e41d4677f5a497289d8e0189b261a63?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding previous findings or accepted new issues

Summary

This PR adds an opt-in TypeSafe Jev pre-call guardrail that removes irrelevant completed tool results while preserving protected conversation history

• Registers TypeSafe in guardrail configuration, discovery, auto-router compression, OpenAPI, and dashboard provider lists
• Uses LiteLLM’s shared asynchronous HTTP infrastructure and supports fail-open or fail-closed evaluator handling
• Adds configuration validation and focused tests for compaction, protection, failures, initialization, and registry discovery
• The changes since the previous review restore registry discoverability and add a regression test for proxy startup registration

_{Reviews (4) · Last reviewed commit: "fix(guardrails): keep typesafe registrie..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_typesafe_compaction_guardrail (0765f6d) with main (fc49ad7)}

[codecov]

Codecov Report

❌ Patch coverage is 98.62385% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...xy/guardrails/guardrail_hooks/typesafe/__init__.py	92.59%	2 Missing ⚠️
...xy/guardrails/guardrail_hooks/typesafe/typesafe.py	99.42%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai