fix(llm_guard): apply sanitized prompt returned by moderation API to request

[devin-ai-integration]

Relevant issues

Fixes #33322

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This integration talks to a self-hosted LLM Guard API server (LLM_GUARD_API_BASE), so the moderation response is exercised through the same code path the proxy uses via mock_redacted_text, which stands in for the /analyze/prompt response body.

Before (at 65ca095d4d), moderation_check inspected only is_valid and returned None, so async_moderation_hook handed back None and the raw request was forwarded unchanged:

result: None
data: {'messages': [{'role': 'user', 'content': 'email: person@example.com'}]}   # NOT redacted

After this change:

result: {'messages': [{'role': 'user', 'content': 'email: [REDACTED]'}]}
data:   {'messages': [{'role': 'user', 'content': 'email: [REDACTED]'}]}

Repro used (captured at the head of this branch):

import asyncio, litellm
litellm.llm_guard_mode = "all"
from litellm_enterprise.enterprise_callbacks.llm_guard import _ENTERPRISE_LLMGuard
from litellm.proxy._types import UserAPIKeyAuth

guard = _ENTERPRISE_LLMGuard(mock_testing=True, mock_redacted_text={
    "sanitized_prompt": "email: [REDACTED]", "is_valid": True, "scanners": {"Regex": 0.0}})
data = {"messages": [{"role": "user", "content": "email: person@example.com"}]}
res = asyncio.run(guard.async_moderation_hook(
    data=data, user_api_key_dict=UserAPIKeyAuth(api_key="sk-1"), call_type="completion"))
print("result:", res)
print("data:", data)

I don't have an LLM Guard server to hit for a fully live proof; happy to run an end-to-end pass against a real deployment if you can point me at one

Type

🐛 Bug Fix

Changes

moderation_check now returns the sanitized_prompt from LLM Guard (falling back to the original text when the API omits it) instead of returning None on a valid response; it still raises on is_valid=False and on a missing response body.

async_moderation_hook applies that sanitized text back onto the outbound request rather than throwing away the result. Because LLM Guard's /analyze/prompt sanitizes one prompt string at a time, the previous approach of concatenating every message into a single formatted prompt could never be mapped back onto individual messages, so sanitization now runs per segment (mirroring the presidio guardrail): each string message content, each text part of multimodal content, and each input entry for embeddings/moderation is sanitized independently and non-text parts are left untouched. The hook returns the mutated data for every accepted call type.

QA runbook

The changed tests are unit tests under tests/local_testing/test_llm_guard.py, not e2e harness tests, so no e2e runbook applies

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/826a2e8517264430a65d89a6b4c037be

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR fixes a bug where LLM Guard's /analyze/prompt response was silently discarded on valid requests, meaning PII sanitization was never applied to outbound messages. The fix makes moderation_check return the sanitized_prompt string and rewires async_moderation_hook to apply it back onto each segment of the request.

• moderation_check now returns the sanitized text (falling back to the original when the API omits sanitized_prompt) and raises HTTPException(400) on policy violations.
• _moderate_request dispatches sanitization per-segment using asyncio.gather for concurrency in most branches.
• Tests are updated to assert mutated content on the returned data dict, and a new test covers multimodal messages and embedding input.

Confidence Score: 4/5

Safe to merge; the core sanitization fix is correct and well-tested, with two minor quality issues that do not affect correctness.

The core bug fix is clearly correct and covered by new assertions. The two remaining issues are: the _moderate_input list branch awaits items sequentially rather than concurrently (a performance gap for large embedding batches), and HTTPException raised for policy violations is caught by the broad except handler and logged at error level before re-raising. Neither affects correctness.

The _moderate_input method in llm_guard.py deserves a second look for the sequential-await pattern on list inputs.

Important Files Changed

Filename	Overview
enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py	Core bug fix: moderation_check now returns the sanitized prompt and async_moderation_hook applies it back to request data; new per-segment sanitization methods introduced, but _moderate_input list branch uses sequential awaits instead of asyncio.gather like the rest of the code.
tests/local_testing/test_llm_guard.py	Tests updated and extended to verify the sanitized prompt is applied to request data; test_llm_guard_valid_response now asserts mutated content, and test_llm_guard_sanitizes_multimodal_and_input covers multimodal and embedding inputs.

Comments Outside Diff (1)

1. enterprise/litellm_enterprise/enterprise_callbacks/llm_guard.py, line 89-95 (link)

   HTTPException logged as error on every policy rejection

   The broad except Exception block catches the HTTPException raised for is_valid=False and calls verbose_proxy_logger.exception(...) (which records a full traceback) before re-raising. Every content-safety rejection will produce a spurious exception-level log entry, making log noise hard to suppress and potentially obscuring real errors. Consider catching HTTPException separately and re-raising it without logging, then logging only truly unexpected exceptions.

_{Reviews (1): Last reviewed commit: "fix(llm_guard): apply sanitized prompt r..." | Re-trigger Greptile}

[greptile-apps]

Sequential awaits in _moderate_input list branch serialize LLM Guard API calls. For a batch embedding request with N string inputs, this method will issue N back-to-back HTTP calls to the LLM Guard server, while _moderate_message and _moderate_request both use asyncio.gather for concurrency. In production with large batches (e.g. 50 embeddings), this could add 50× the round-trip latency of a single guard call.

Suggested change

	if isinstance(input_, list):
	return [
	await self.moderation_check(text=item)
	if isinstance(item, str)
	else item
	for item in input_
	]
	if isinstance(input_, list):
	async def _check_or_pass(item: object) -> object:
	return await self.moderation_check(text=item) if isinstance(item, str) else item
	return list(await asyncio.gather(*(_check_or_pass(item) for item in input_)))

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_llm_guard_sanitized_prompt (e0b09f5) with litellm_internal_staging (f1f33f5)}

[yassin-berriai]

not a fan of this pattern

[devin-ai-integration]

Good call, inlined the fetch back into moderation_check so the mock/real branch stays in one place; dropped the separate _get_moderation_response helper in a6171cd