feat: guardrail tracing UI - policy, detection method, match details#21349
feat: guardrail tracing UI - policy, detection method, match details#21349ishaan-jaff merged 14 commits intomainfrom
Conversation
…dardLoggingGuardrailInformation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds guardrail tracing metadata to the logging pipeline and renders it in the dashboard UI. A new
Confidence Score: 4/5
|
| Filename | Overview |
|---|---|
| litellm/types/utils.py | Adds new optional tracing fields to StandardLoggingGuardrailInformation (guardrail_id, policy_template, detection_method, etc.) and a new GuardrailTracingDetail TypedDict. All fields are optional with total=False, so no impact on existing consumers. |
| litellm/types/guardrails.py | Adds optional policy_template field to the Guardrail TypedDict. Minimal change, no impact on existing code. |
| litellm/integrations/custom_guardrail.py | Adds tracing_detail parameter to add_standard_logging_guardrail_information_to_request_data() and unpacks it via **(tracing_detail or {}) into the SLG dict. Clean, backward-compatible change. |
| litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/content_filter.py | Adds tracing detail population: _build_match_details, _get_detection_methods, _get_patterns_checked_count, _get_policy_templates helper methods. Passes GuardrailTracingDetail to the logging method. Minor issue: two datetime.now() calls in _log_guardrail_information (pre-existing). |
| ui/litellm-dashboard/src/components/view_logs/GuardrailViewer/GuardrailViewer.tsx | Adds five new UI components: PolicyDetectionRow, MatchDetailsTable, ClassificationDetails, ExecutionTimeline, and policy template badges in the header. Well-structured with proper null guards. Minor concern: hasData shows row even when only patterns_checked is 0. |
| tests/test_litellm/integrations/test_custom_guardrail.py | Adds 4 unit tests for tracing fields: set, default absent, multiple policies, classification. All tests use mocks, no network calls. |
| tests/test_litellm/proxy/guardrails/guardrail_hooks/content_filter/test_content_filter.py | Adds 5 E2E tests for content filter tracing: pattern detection, fallback ID, category keywords, blocked request, no detections. All tests use local guardrail instances, no network calls. |
Sequence Diagram
sequenceDiagram
participant CF as ContentFilterGuardrail
participant CG as CustomGuardrail (base)
participant SLG as StandardLoggingGuardrailInformation
participant UI as GuardrailViewer (UI)
CF->>CF: apply_guardrail()
CF->>CF: _filter_single_text() → detections
CF->>CF: _build_match_details(detections)
CF->>CF: _get_detection_methods(detections)
CF->>CF: _get_patterns_checked_count()
CF->>CF: _get_policy_templates()
CF->>CF: Build GuardrailTracingDetail
CF->>CG: add_standard_logging_guardrail_information_to_request_data(tracing_detail)
CG->>SLG: Create SLG with **(tracing_detail or {})
CG->>CG: Append SLG to request_data["metadata"]
Note over SLG: Fields: guardrail_id, policy_template,<br/>detection_method, match_details,<br/>patterns_checked, confidence_score, etc.
SLG-->>UI: Logged data rendered in dashboard
UI->>UI: PolicyDetectionRow (badges, chips)
UI->>UI: MatchDetailsTable (per-detection table)
UI->>UI: ClassificationDetails (LLM-judge panel)
UI->>UI: ExecutionTimeline (multi-guardrail view)
Last reviewed commit: d648e0f
| class GuardrailTracingDetail(TypedDict, total=False): | ||
| """ | ||
| Typed fields for guardrail tracing metadata. | ||
|
|
||
| Passed to add_standard_logging_guardrail_information_to_request_data() | ||
| to enrich the StandardLoggingGuardrailInformation with provider-specific details. | ||
| """ | ||
|
|
||
| guardrail_id: Optional[str] | ||
| policy_template: Optional[str] | ||
| detection_method: Optional[str] | ||
| confidence_score: Optional[float] | ||
| classification: Optional[dict] | ||
| match_details: Optional[List[dict]] | ||
| patterns_checked: Optional[int] | ||
| alert_recipients: Optional[List[str]] |
There was a problem hiding this comment.
Duplicated field definitions across TypedDicts
GuardrailTracingDetail duplicates all 8 field definitions from StandardLoggingGuardrailInformation (lines 2623-2645). If a field is added, renamed, or has its type changed on one but not the other, the **(tracing_detail or {}) unpacking in custom_guardrail.py will silently drop the mismatched field.
Consider deriving one from the other or extracting a shared base, e.g.:
class _GuardrailTracingFields(TypedDict, total=False):
guardrail_id: Optional[str]
policy_template: Optional[str]
detection_method: Optional[str]
confidence_score: Optional[float]
classification: Optional[dict]
match_details: Optional[List[dict]]
patterns_checked: Optional[int]
alert_recipients: Optional[List[str]]
# Then use _GuardrailTracingFields as a base or as GuardrailTracingDetail directly
GuardrailTracingDetail = _GuardrailTracingFieldsTwo-column layout with request lifecycle timeline on the left and compact evaluation detail cards on the right. Header shows guardrail count, pass/fail status, total overhead, policy info, and an export button.
…l-tracing-ui # Conflicts: # litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/content_filter.py # litellm/types/utils.py # ui/litellm-dashboard/src/components/view_logs/GuardrailViewer/GuardrailViewer.tsx
…ION_PARAMS (#21360) * allow filtering by user in global usage * add server root path test to github actions * Update .github/workflows/test_server_root_path.yml Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * address greptile review feedback (greploop iteration 1) - Fix HTTPException swallowed by broad except block in get_user_daily_activity and get_user_daily_activity_aggregated: re-raise HTTPException before the generic handler so 403 status codes propagate correctly - Add status_code assertions in non-admin access tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * address greptile review feedback (greploop iteration 2) - Default user_id to caller's own ID for non-admins instead of 403 when omitted, preserving backward compatibility for API consumers - Apply same fix to aggregated endpoint - Update test to verify defaulting behavior instead of expecting 403 - Add useEffect to sync selectedUserId when auth state settles in UsagePageView to handle async auth initialization Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fixing syntax * remove artifacts * feat: guardrail tracing UI - policy, detection method, match details (#21349) * feat: add GuardrailTracingDetail TypedDict and tracing fields to StandardLoggingGuardrailInformation * feat: add policy_template field to Guardrail config TypedDict * feat: accept GuardrailTracingDetail in base guardrail logging method * feat: populate tracing fields in content filter guardrail * test: add tracing fields tests for custom guardrail base class * test: add tracing fields e2e tests for content filter guardrail * feat: add guardrail tracing UI - policy badges, match details, timeline * feat: redesign GuardrailViewer to Guardrails & Policy Compliance layout Two-column layout with request lifecycle timeline on the left and compact evaluation detail cards on the right. Header shows guardrail count, pass/fail status, total overhead, policy info, and an export button. * feat: add clickable guardrail link in metrics + show policy names * feat: add risk_score field to StandardLoggingGuardrailInformation * feat: compute risk_score in content filter guardrail * feat: display backend risk_score badge on evaluation cards * fix: fallback to frontend risk score when backend doesn't provide one * passing in masster key for api calls * Fix: Add blog as incident report * Fix: Add blog as incident report * remove timeline * feat(models): add github_copilot/gpt-5.3-codex and github_copilot/claude-opus-4.6-fast (#21316) Add missing GitHub Copilot model entries for gpt-5.3-codex (GA) and claude-opus-4.6-fast (Public Preview) to both the root and backup model pricing JSON files. * only tests for /ui * bump: version 1.81.12 → 1.81.13 * Fixing mapped tests * fixing no_config test * fixing container tests * fixing test_basic_openai_responses_api * Adding bedrock thinking budget tokens to docs * fixing regen key tests * fix: add missing OpenAI chat completion params to OPENAI_CHAT_COMPLETION_PARAMS Add store, prompt_cache_key, prompt_cache_retention, safety_identifier, and verbosity to OPENAI_CHAT_COMPLETION_PARAMS list. These params were already in DEFAULT_CHAT_COMPLETION_PARAM_VALUES but missing from the OPENAI_CHAT_COMPLETION_PARAMS list, causing them to be dropped when passed to OpenAI-compatible providers. --------- Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Summary
GuardrailTracingDetailTypedDict with fields:guardrail_id,policy_template,detection_method,confidence_score,classification,match_details,patterns_checked,alert_recipientsTest plan