Hiddenlayer Integration: Add V2 Integration - #22708
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds a V2 HiddenLayer guardrail implementation (
Confidence Score: 4/5Functionally correct for the happy path; a few edge cases flagged in prior rounds remain open but were acknowledged by the maintainer, and the new V2 class is well-tested overall. Prior review rounds surfaced several robustness issues (missing isinstance guard on last_msg, potential IndexError on empty choices list, null fields in V2 payload, empty-payload API call on response path) — some were replied to as intentional, others were not addressed. New findings in this round are limited to a misleading test and outdated docs, both P2. The core V2 logic and routing are sound, but the open prior-round items prevent a confident 5/5. litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py (V2 edge cases), tests/test_litellm/proxy/guardrails/guardrail_hooks/test_hiddenlayer.py (no-op error test), docs/my-website/docs/proxy/guardrails/hiddenlayer.md (stale V1 reference)
|
| Filename | Overview |
|---|---|
| litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py | Adds V2 guardrail class with new API endpoints; several edge cases remain from prior review rounds (empty choices IndexError, null fields in payload, empty-payload API call on response path, missing isinstance guard on last_msg). |
| litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/init.py | Routes to V1 or V2 based on version field; guardrail_class_registry still maps to V1 class only while V2 is the new default. |
| litellm/types/proxy/guardrails/guardrail_hooks/hiddenlayer.py | Adds version field defaulting to 2; silently migrates existing users to V2 API on upgrade (accepted by maintainer as intentional). |
| tests/test_litellm/proxy/guardrails/guardrail_hooks/test_hiddenlayer.py | Good coverage for V1 and V2 happy paths and block/redact cases; test_apply_guardrail_api_error_handling is a no-op (empty inputs → API never called, mock exception never fires). |
| litellm/types/guardrails.py | Adds HiddenlayerGuardrailConfigModel to LitellmParams inheritance chain; straightforward import addition. |
| docs/my-website/docs/proxy/guardrails/hiddenlayer.md | Documentation still references the V1 /detection/v1/interactions endpoint as the integration point, but V2 is now the default — misleading for new users. |
Sequence Diagram
sequenceDiagram
participant Client
participant LiteLLM
participant HL as HiddenLayer API
participant LLM
Note over LiteLLM: apply_guardrail (pre_call)
Client->>LiteLLM: POST /chat/completions
alt V1 (version < 2)
LiteLLM->>HL: POST /detection/v1/interactions
HL-->>LiteLLM: evaluation + analysis + modified_data
alt action == Block
LiteLLM-->>Client: 400 HTTPException (block_reasons, threat_level)
else action == Redact
LiteLLM->>LLM: modified messages
else Allow
LiteLLM->>LLM: original messages
end
else V2 (version >= 2, default)
LiteLLM->>HL: POST /detection/v2/request-evaluations
HL-->>LiteLLM: response + hl-runtime-action header
alt header == block
LiteLLM-->>Client: 400 HTTPException
else pass
LiteLLM->>LLM: possibly modified messages
end
end
LLM-->>LiteLLM: response
Note over LiteLLM: apply_guardrail (post_call)
alt V1
LiteLLM->>HL: POST /detection/v1/interactions
HL-->>LiteLLM: evaluation result
else V2
LiteLLM->>HL: POST /detection/v2/response-evaluations
HL-->>LiteLLM: response + hl-runtime-action header
end
LiteLLM-->>Client: final response
Reviews (6): Last reviewed commit: "Add image support" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…er.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
| if scan_params := inputs.get("structured_messages"): | ||
| # Convert AllMessageValues to simple dict format for HiddenLayer API | ||
| messages = [ | ||
| {"role": msg.get("role", "user"), "content": msg.get("content", "")} | ||
| for msg in scan_params | ||
| if isinstance(msg, dict) | ||
| ] | ||
| last_msg = scan_params[-1] | ||
| result = await self._call_hiddenlayer( | ||
| project_id, hl_request_metadata, {"messages": messages}, input_type | ||
| project_id, | ||
| hl_request_metadata, | ||
| {"messages": [{"role": last_msg.get("role", "user"), "content": last_msg.get("content", "")}]}, | ||
| input_type, | ||
| ) |
There was a problem hiding this comment.
Missing test coverage for new behavior
The existing test test_apply_guardrail_request_with_violations mocks the HiddenLayer response as {"evaluation": {"action": "Block"}} without an analysis key, so it doesn't verify the new block_reasons or threat_level fields in the error detail. Similarly, no test covers the change from scanning all messages to scanning only the last message.
Consider adding a test that:
- Mocks a response with
analysisentries (e.g.,{"analysis": [{"name": "prompt_injection", "detected": True}], "evaluation": {"action": "Block", "threat_level": "high"}}) - Verifies that
exc_info.value.detail["block_reasons"]contains["prompt_injection"]andthreat_levelis"high" - Uses
structured_messageswith multiple messages and asserts only the last one is sent to the API
…thub.com/Ashton-Sidhu/litellm into hiddenlayer-guardrail-integration-update
| description="The Hiddenlayer Secret Key for the Hiddenlayer API.. If not provided, the `HIDDENLAYER_CLIENT_SECRET` environment variable is checked.", | ||
| ) | ||
|
|
||
| version: Optional[int] = Field(default=2, description="Hiddenlayer guardrail version to use.") |
There was a problem hiding this comment.
Setting default=2 means every existing HiddenLayer user who has not explicitly set a version field in their config will be silently upgraded from HiddenlayerGuardrail (V1) to HiddenlayerGuardrailV2 when they upgrade litellm. V1 calls /detection/v1/interactions while V2 calls /detection/v2/request-evaluations and /detection/v2/response-evaluations — these are completely different APIs with different request/response shapes. This will silently break any user on the V1 API.
Per the project's backwards-compatibility rule, the default should preserve existing behavior (1), and users should opt in to V2 explicitly:
| version: Optional[int] = Field(default=2, description="Hiddenlayer guardrail version to use.") | |
| version: Optional[int] = Field(default=1, description="Hiddenlayer guardrail version to use.") |
Rule Used: What: avoid backwards-incompatible changes without... (source)
There was a problem hiding this comment.
Not an issue in our use case!
Ashton-Sidhu
changed the title
…er.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
|
||
| new_texts = [] | ||
| if input_type == "request": | ||
| inputs["structured_messages"] = output |
There was a problem hiding this comment.
structured_messages set to full response dict instead of messages list
inputs["structured_messages"] = output assigns the entire JSON response object from HiddenLayer (e.g. {"messages": [...], "model": "...", "tools": [...]}) to structured_messages. But structured_messages is expected to be a list of message dicts. Any downstream litellm code that reads inputs["structured_messages"] to build the LLM request will receive a dict instead of a list, which will likely fail.
The intent appears to be to propagate the (potentially redacted/modified) messages from HiddenLayer's response. Only the messages array should be assigned:
inputs["structured_messages"] = output.get("messages", inputs.get("structured_messages", []))This also avoids silently dropping the original messages if HiddenLayer returns a response without a messages key.
There was a problem hiding this comment.
the hiddenlayer api returns the exact same api shape it receives, so if we send up messages, there will be a messages coming back
…thub.com/Ashton-Sidhu/litellm into hiddenlayer-guardrail-integration-update
| payload = { | ||
| "messages": inputs.get("structured_messages"), | ||
| "model": inputs.get("model"), | ||
| "tools": inputs.get("tools") | ||
| } |
There was a problem hiding this comment.
null fields for model and tools
When inputs.get("model") or inputs.get("tools") is None (e.g. when tools are not part of the request), the payload is serialized as {"messages": [...], "model": null, "tools": null}. Many REST APIs distinguish between an absent field and an explicit null — sending null for tools could cause a validation error on the HiddenLayer V2 side, whereas simply omitting the key would be safe.
Consider only including keys with non-None values:
payload: dict[str, Any] = {}
if messages := inputs.get("structured_messages"):
payload["messages"] = messages
if model := inputs.get("model"):
payload["model"] = model
if tools := inputs.get("tools"):
payload["tools"] = tools| else: | ||
| payload = {} | ||
|
|
||
| response = await self._call_hiddenlayer( | ||
| payload, # ty:ignore[invalid-argument-type] | ||
| input_type, | ||
| hl_headers | ||
| ) |
There was a problem hiding this comment.
When input_type == "response" and inputs has neither texts nor tool_calls, payload is set to {} and _call_hiddenlayer is still invoked, sending an empty body to detection/v2/response-evaluations. Compare this with V1's behaviour, which short-circuits with result = {} (no API call) in the same situation.
Sending an empty payload to the evaluation endpoint is wasteful and may trigger a validation error from HiddenLayer. The response path should mirror V1 and skip the API call when there is no content to scan:
else:
return inputs # nothing to scan, skip API call
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
59a9047
into
BerriAI:litellm_oss_staging_04_13_2026_p1
* Serialize error message to a string; only scan last message * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add v2 of hiddenlayer guardrail implementation * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix potential header issue * linting * Add image support --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Serialize error message to a string; only scan last message * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add v2 of hiddenlayer guardrail implementation * Update litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix potential header issue * linting * Add image support --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2 participants
Relevant issues
Fixes some issues with the hiddenlayer guardrails:
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🐛 Bug Fix