fix(guardrails): show YAML-defined guardrails in Guardrail Monitor - #26830
fix(guardrails): show YAML-defined guardrails in Guardrail Monitor#26830yuneng-berri wants to merge 3 commits into
Conversation
The /guardrails/usage/{overview,detail,logs} endpoints only consulted the
litellm_guardrailstable Prisma table, so guardrails defined in config.yaml
(stored only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible: detail 404'd,
overview rendered them as "Custom"/"Guardrail" via the orphan-metric path or
not at all, and logs missed the logical-name alias.
Mirror the existing list_guardrails_v2 / get_guardrail_info pattern: union DB
rows with IN_MEMORY_GUARDRAIL_HANDLER.list_in_memory_guardrails() (deduped by
guardrail_id) for overview, fall back to get_guardrail_by_id() for detail and
logs. Adds tests covering DB-only, YAML-only, both, and DB-takes-precedence.
Fixes LIT-2529.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address PR review feedback. InMemoryGuardrailHandler.initialize_guardrail was
constructing the parsed Guardrail without guardrail_info, so even after the
endpoint fallback fix, /guardrails/usage/{overview,detail} couldn't render
type or description for YAML-defined guardrails — both fields would silently
default to "Guardrail" / None at runtime.
Pass guardrail_info through into IN_MEMORY_GUARDRAILS, and add a regression
test that exercises the real handler (not a mock) end-to-end.
Also refactors _get_guardrail_attrs to use the new _get_guardrail_field helper
for consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…itellm_guardrailMonitorYamlFix
Greptile SummaryFixes the Guardrail Monitor never showing YAML-defined guardrails by unioning Confidence Score: 4/5Safe to merge; logic is correct and well-tested with only a minor style suggestion. No P0 or P1 issues found. The union/dedup/fallback logic is correct — the registry mutates the input dict before constructing the stored TypedDict, so No files require special attention;
|
| Filename | Overview |
|---|---|
| litellm/proxy/guardrails/guardrail_registry.py | One-line addition: guardrail_info is now passed when constructing the in-memory Guardrail TypedDict, so detail/overview endpoints can read type and description for YAML guardrails. Change is correct — guardrail["guardrail_id"] is mutated before this line, so the stored TypedDict always has a non-None ID. |
| litellm/proxy/guardrails/usage_endpoints.py | Overview, detail, and logs endpoints now union DB guardrails with IN_MEMORY_GUARDRAIL_HANDLER entries, with dedup and correct fallback ordering. Helper functions _get_guardrail_field / _to_dict cleanly unify access across Prisma rows and TypedDicts/Pydantic models. Minor: _to_dict is narrowly typed to LitellmParams specifically. |
| tests/test_litellm/proxy/guardrails/test_usage_endpoints.py | 7 new unit tests covering all three fixed endpoints, including DB-precedence dedup, orphan-metric suppression, logical-name alias in logs query, and a real-handler integration test. All tests use mocked Prisma/in-memory handlers with no real network calls — consistent with the tests/test_litellm/ no-network rule. |
Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
| def _to_dict(value: Any) -> Dict[str, Any]: | ||
| """Coerce a LitellmParams / dict / None into a plain dict.""" | ||
| if isinstance(value, LitellmParams): | ||
| return value.model_dump(exclude_none=True) | ||
| if isinstance(value, dict): | ||
| return value | ||
| return {} |
There was a problem hiding this comment.
_to_dict hardcodes LitellmParams type
_to_dict special-cases LitellmParams specifically, which means any other Pydantic model stored in these fields (e.g. a custom subclass or a future guardrail_info model) silently returns {} rather than its actual data. Using BaseModel from Pydantic as the isinstance check would be more resilient.
| def _to_dict(value: Any) -> Dict[str, Any]: | |
| """Coerce a LitellmParams / dict / None into a plain dict.""" | |
| if isinstance(value, LitellmParams): | |
| return value.model_dump(exclude_none=True) | |
| if isinstance(value, dict): | |
| return value | |
| return {} | |
| def _to_dict(value: Any) -> Dict[str, Any]: | |
| """Coerce a Pydantic BaseModel / dict / None into a plain dict.""" | |
| from pydantic import BaseModel as _BaseModel | |
| if isinstance(value, _BaseModel): | |
| return value.model_dump(exclude_none=True) | |
| if isinstance(value, dict): | |
| return value | |
| return {} |
|
Superseded by #32853, which reimplements this fix from scratch against current staging (the code here had drifted to the GuardrailsRepository accessor since this branch was cut). Closing in favor of that PR |
Summary
Fixes LIT-2529 — the dashboard's Guardrail Monitor never showed details for guardrails defined in
config.yaml.The three endpoints serving the monitor (
/guardrails/usage/overview,/guardrails/usage/detail/{id},/guardrails/usage/logs) only read from thelitellm_guardrailstablePrisma table. YAML-defined guardrails live inIN_MEMORY_GUARDRAIL_HANDLERand never get persisted, so:provider="Custom"/type="Guardrail"via the orphan-metric fallback (and not at all when they had no metrics)This PR mirrors the union/fallback pattern already used by
list_guardrails_v2andget_guardrail_info:usage/overview— union DB rows withIN_MEMORY_GUARDRAIL_HANDLER.list_in_memory_guardrails(), deduped byguardrail_id, before building rows.usage/detail/{id}— fall back toIN_MEMORY_GUARDRAIL_HANDLER.get_guardrail_by_id()when the DB lookup misses; only 404 if both miss.usage/logs— same fallback so the YAML guardrail's logical name gets added toeffective_guardrail_ids._get_guardrail_field/_to_dicthelpers that uniformly handle Prisma rows, plain dicts, TypedDicts, andLitellmParamsPydantic instances.Test plan
tests/test_litellm/proxy/guardrails/test_usage_endpoints.py(7 tests) — written red first, all pass after the fixpytest tests/test_litellm/proxy/guardrails/→ 1358 passed, no regressions intest_guardrail_endpoints.py/test_guardrail_registry.py/ hooksIN_MEMORY_GUARDRAIL_HANDLERsingleton seeded as if from a YAML load (mockedprisma_client) — overview row, detail response, and logs query all return the YAML guardrail's real metadatablack+ruff checkcleanReopened from #26648 — the previous branch was missing the
litellm_prefix so CircleCI did not run.