Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support#20619
Conversation
…rails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily
allows the same guardrails for llm's to be applied to agents as well
…examples of pre/post calls for custom code guardrails
allows custom code guardrails to work on mcp input
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile OverviewGreptile Summary
Confidence Score: 2/5
|
| Filename | Overview |
|---|---|
| docs/my-website/docs/proxy/guardrails/custom_code_guardrail.md | Documented async apply_guardrail support and new async HTTP primitives for custom code guardrails. |
| litellm/llms/a2a/chat/guardrail_translation/README.md | Added documentation for A2A guardrail translation handler and how to pass guardrails via params.metadata. |
| litellm/llms/a2a/chat/guardrail_translation/init.py | Registered A2A guardrail translation mappings for CallTypes.send_message and CallTypes.asend_message. |
| litellm/llms/a2a/chat/guardrail_translation/handler.py | Implemented A2A input/output translation for unified guardrails by extracting text parts and mapping updates back into JSON-RPC structures. |
| litellm/proxy/_experimental/mcp_server/guardrail_translation/handler.py | Refactored MCP guardrail translation to use tool definitions; currently drops actual tool call arguments from guardrail inputs (see review comment). |
| litellm/proxy/_experimental/mcp_server/mcp_server_manager.py | Minor formatting + SEP-986 tool name validation import adjustments; no behavioral changes spotted in diff. |
| litellm/proxy/_experimental/mcp_server/rest_endpoints.py | Updated MCP REST tool call endpoint to run common pre-call processing and enforce server access via shared helper. |
| litellm/proxy/agent_endpoints/a2a_endpoints.py | Added support for passing LiteLLM params (e.g., guardrails) through A2A params and moved to ProxyBaseLLMRequestProcessing for pre-call processing. |
| litellm/proxy/common_request_processing.py | Extended ProxyBaseLLMRequestProcessing route types to include A2A send_message and MCP tool calls; no issues found in diff. |
| litellm/proxy/guardrails/guardrail_hooks/custom_code/custom_code_guardrail.py | Added async apply_guardrail support (await coroutine results) and enabled MCP-related event hooks for custom code guardrails. |
| litellm/proxy/guardrails/guardrail_hooks/custom_code/primitives.py | Added async HTTP request primitives (http_request/get/post) using cached async httpx client; list-body handling currently routes to data= instead of json= (see review comment). |
| litellm/proxy/guardrails/guardrail_hooks/unified_guardrail/unified_guardrail.py | Adjusted unified guardrail hook to use pre_mcp_call/during_mcp_call for CallTypes.call_mcp_tool. |
| ui/litellm-dashboard/src/components/guardrails/custom_code/CustomCodeModal.tsx | Updated custom code modal templates and primitive list to include async HTTP primitives and async apply_guardrail example. |
| ui/litellm-dashboard/src/components/guardrails/guardrail_info.tsx | UI updates for guardrail configuration screen to integrate custom code modal changes. |
| ui/litellm-dashboard/src/components/mcp_tools/MCPToolArgumentsForm.tsx | Added structured MCP tool arguments form generation from JSON schema, with ref-based submit value extraction. |
| ui/litellm-dashboard/src/components/networking.tsx | Added callMCPTool options to pass guardrails via litellm_metadata and improved error parsing for MCP tool calls. |
| ui/litellm-dashboard/src/components/playground/chat_ui/ChatUI.tsx | Added MCP direct tool-call mode and passing guardrails to MCP/A2A; introduces a syntax-breaking indentation error in handleSendMessage (see review comment). |
| ui/litellm-dashboard/src/components/playground/chat_ui/chatConstants.ts | Added MCP endpoint option label to playground endpoint dropdown. |
| ui/litellm-dashboard/src/components/playground/chat_ui/mode_endpoint_mapping.tsx | Added EndpointType.MCP to endpoint mapping enum for playground mode selection. |
Sequence Diagram
sequenceDiagram
autonumber
participant UI as Playground UI (ChatUI)
participant Net as UI networking.callMCPTool
participant API as Proxy /mcp-rest/tools/call
participant Proc as ProxyBaseLLMRequestProcessing
participant UG as UnifiedLLMGuardrails
participant MTr as MCPGuardrailTranslationHandler
participant GR as CustomGuardrail.apply_guardrail
participant MCP as MCPServerManager.execute_mcp_tool
UI->>Net: callMCPTool(server_id,name,arguments, litellm_metadata.guardrails)
Net->>API: POST /mcp-rest/tools/call
API->>Proc: common_processing_pre_call_logic(route_type=call_mcp_tool)
Proc-->>API: enriched data + logging_obj
API->>UG: pre-call hook (CallTypes.call_mcp_tool)
UG->>MTr: process_input_messages(data)
MTr->>GR: apply_guardrail(inputs, request_data)
UG-->>API: allow/block
API->>MCP: execute_mcp_tool(name, arguments, allowed_servers)
MCP-->>API: CallToolResult
API-->>UI: tool result
| const handleSendMessage = async () => { | ||
| if ( | ||
| inputMessage.trim() === "" && |
There was a problem hiding this comment.
Broken function indentation
handleSendMessage is now declared with extra leading spaces (" const handleSendMessage = async () => {"), which breaks the component’s block structure and will fail to compile (the subsequent if (...) is also mis-indented). This needs to be aligned with other top-level handlers in the component.
| const handleSendMessage = async () => { | |
| if ( | |
| inputMessage.trim() === "" && | |
| const handleSendMessage = async () => { |
| if isinstance(body, list): | ||
| return None, json.dumps(body) | ||
| if isinstance(body, str): | ||
| return None, body | ||
| return None, str(body) |
There was a problem hiding this comment.
Incorrect JSON body handling
_prepare_http_body() returns (None, json.dumps(body)) for list inputs, which routes list bodies to the data= parameter instead of json=. That sends a string payload without setting JSON encoding semantics and differs from the docstring (“JSON-encoded if dict/list”), causing APIs expecting JSON arrays to misinterpret the request.
| if isinstance(body, list): | |
| return None, json.dumps(body) | |
| if isinstance(body, str): | |
| return None, body | |
| return None, str(body) | |
| if isinstance(body, list): | |
| return body, None |
| "type": "function", | ||
| "function": ChatCompletionToolParamFunctionChunk( | ||
| name=fn["name"], | ||
| description=fn.get("description") or "", | ||
| parameters=fn.get("parameters") | ||
| or { | ||
| "type": "object", | ||
| "properties": {}, | ||
| "additionalProperties": False, | ||
| }, | ||
| strict=fn.get("strict", False) or False, # Default to False if None | ||
| ), | ||
| } |
There was a problem hiding this comment.
Tool call missing args
process_input_messages() now passes only tools=[...] into GenericGuardrailAPIInputs and does not include the actual tool call + arguments anywhere in inputs. On the custom-code path, inputs["tool_calls"] will be empty, so guardrails that validate/modify tool arguments won’t see the call payload.
Concrete failure: a custom guardrail expecting inputs["tool_calls"][0]["function"]["arguments"] will never trigger for MCP tool calls.
Consider adding a tool_calls entry (name + JSON arguments) alongside tools.
Update tests to match new implementation:
1. test_invoke_agent_a2a_adds_litellm_data:
- Mock ProxyBaseLLMRequestProcessing.common_processing_pre_call_logic
instead of add_litellm_data_to_request (which is no longer used)
2. test_process_input_messages_* (MCP guardrail handler):
- Handler now processes mcp_tool_name/mcp_arguments instead of messages
- Passes GenericGuardrailAPIInputs(tools=[...]) to guardrails
- Updated tests to verify new tool-based guardrail behavior
- Renamed test_process_input_messages_skips_when_no_messages to
test_process_input_messages_skips_when_no_tool_name
…ng format for Claude Code (#20631) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(websearch_interception): convert agentic loop response to streaming format when original request was streaming Fixes #20187 - When using websearch_interception in Bedrock with Claude Code: 1. Output tokens were showing as 0 because the agentic loop response wasn't being converted back to streaming format 2. The response from the agentic loop (follow-up request) was returned as a non-streaming dict, but Claude Code expects a streaming response This fix adds streaming format conversion for the agentic loop response when the original request was streaming (detected via the websearch_interception_converted_stream flag in logging_obj). The fix ensures: - Output tokens are correctly included in the message_delta event - stop_reason is properly preserved - The response format matches what Claude Code expects --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com>
…iohttp tracing (#20630) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing When users pass a shared_session with trace_configs to acompletion(), the get_async_httpx_client() function was ignoring it and returning a cached client without the user's tracing configuration. This fix bypasses the cache when shared_session is provided, ensuring the user's ClientSession (with its trace_configs, connector settings, etc.) is actually used for the request. Fixes #20174 --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
* Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(ui/): add mcp input as an example for custom code guardrails * feat(a2a/): ensure a2a guardrails works on response output * feat(a2a/): support streaming guardrails * test: address greptile comments * test: address greptile comments
…iohttp tracing (#20630) * Add http support to custom code guardrails + Unified guardrails for MCP + Agent guardrail support (#20619) * fix: fix styling * fix(custom_code_guardrail.py): add http support for custom code guardrails allows users to call external guardrails on litellm with minimal code changes (no custom handlers) Test guardrail integrations more easily * feat(a2a/): add guardrails for agent interactions allows the same guardrails for llm's to be applied to agents as well * fix(a2a/): support passing guardrails to a2a from the UI * style(code-editor): allow editing custom code guardrails on ui + add examples of pre/post calls for custom code guardrails * feat(mcp/): support custom code guardrails for mcp calls allows custom code guardrails to work on mcp input * feat(chatui.tsx): support guardrails on mcp tool calls on playground * fix(mypy): resolve missing return statements and type casting issues (#20618) * fix(mypy): resolve missing return statements and type casting issues * fix(pangea): use elif to prevent UnboundLocalError and handle None messages Address Greptile review feedback: - Make branches mutually exclusive using elif to prevent input_messages from being overwritten - Handle case where data.get('messages') returns None to avoid passing invalid payload to Pangea API --------- Co-authored-by: Shin <shin@openclaw.ai> * [Feat] MCP Gateway - Allow setting MCP Servers as Private/Public available on Internet (#20607) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * fixing user_id * [Feat] IP-Based Access Control for MCP Servers (#20620) * update MCPAuthenticatedUser * add available_on_public_internet for MCPs * update claude.md * init IPAddressUtils * init available_on_public_internet * add on REST endpoints * filter with IP * TestIsInternalIp * _extract_mcp_headers_from_request * init get_mcp_client_ip * _get_general_settings * allowed_server_ids * address PR comments * get_mcp_server_by_name fix * fix server * fix review comments * get_public_mcp_servers * address _get_allowed_mcp_servers * test fix * fix linting * inint ui types * add ui for managing MCP private/public * add ui * fixes * add to schema * add types * fix endpoint * add endpoint * update manager * test mcp * dont use external party for ip address * Add OpenAI/Azure release test suite with HTTP client lifecycle regression detection (#20622) * docs (#20626) * docs * fix(mypy): resolve type checking errors in 5 files (#20627) - a2a_protocol/exception_mapping_utils.py: Fix type ignore comment for None assignment - caching/redis_cache.py: Add type ignore for async ping return type - caching/redis_cluster_cache.py: Add type ignore for async ping return type - llms/deprecated_providers/palm.py: Add type ignore for palm.generate_text - proxy/auth/handle_jwt.py: Add type ignore for jwt.decode options argument All changes add appropriate type: ignore comments to handle library typing inconsistencies. * fix(test): update deprecated gemini embedding model (#20621) Replace text-embedding-004 with gemini-embedding-001. The old model was deprecated and returns 404: 'models/text-embedding-004 is not found for API version v1beta' Co-authored-by: Shin <shin@openclaw.ai> * ui new buil * fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing When users pass a shared_session with trace_configs to acompletion(), the get_async_httpx_client() function was ignoring it and returning a cached client without the user's tracing configuration. This fix bypasses the cache when shared_session is provided, ensuring the user's ClientSession (with its trace_configs, connector settings, etc.) is actually used for the request. Fixes #20174 --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shin <shin@openclaw.ai> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com>
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes