cost for prompt guardrails - #4931
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (28)
🚧 Files skipped from review as they are similar to previous changes (26)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR adds request-scoped guardrail and semantic-cache debug data. It propagates guardrail data through responses and streams, calculates related costs, persists log metadata, records embedding metadata, and displays guardrail judge-call details. ChangesGuardrail and Cache Debug Billing
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant GuardrailPlugin
participant SemanticCache
participant BifrostContext
participant ResponsePipeline
participant LoggingPlugin
participant LogStore
participant LogDetailView
GuardrailPlugin->>BifrostContext: append judge-call debug
SemanticCache->>BifrostContext: store embedding debug
ResponsePipeline->>BifrostContext: read debug snapshots
ResponsePipeline->>LoggingPlugin: propagate response and stream metadata
LoggingPlugin->>LogStore: persist debug data and sidecar costs
LogStore-->>LogDetailView: provide guardrail_debug
LogDetailView-->>LogDetailView: render judge-call details
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Confidence Score: 4/5Safe to merge with awareness of the streaming error path gap noted in the previous review cycle. The guardrail debug and cost-attribution logic is correct for input-blocked requests (Path A), non-streaming requests (Path C), and streaming success (Path B final chunk). The one unresolved gap — streaming Path B when bifrostErr is non-nil and result is also non-nil — was identified in the previous review: in that branch, applyGuardrailCost is never called and the accumulated cost in the stream response is also not applied to the log entry. All other paths are properly handled, tests are thorough, and the migration is safe. plugins/logging/main.go — the streaming Path B error branch (bifrostErr != nil, result != nil) does not apply guardrail cost to the log entry. Important Files Changed
Reviews (8): Last reviewed commit: "cost for prompt guardrails" | Re-trigger Greptile |
43cd183 to
8c9a3dd
Compare
8c9a3dd to
2e58579
Compare
2e58579 to
c6669ce
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
framework/tracing/tracer.go (1)
113-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding test coverage for
SetTraceRedactionReplacements.The guard clauses look correct and the underlying
Trace.SetRedactionReplacementsis mutex-protected, but this new exported method (used for redaction, a security-sensitive path) doesn't appear to have accompanying unit tests in the provided context.As per path instructions,
framework/**: "Apply standard Go review practices... and tests that cover edge cases and failure paths."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@framework/tracing/tracer.go` around lines 113 - 123, Add unit tests for the exported Tracer.SetTraceRedactionReplacements method to cover the security-sensitive redaction path and its guard clauses. Verify it returns early when Tracer is nil, store is nil, traceID is blank, replacements is empty, or GetTrace returns nil, and confirm it calls Trace.SetRedactionReplacements for a valid trace. Use the Tracer and SetTraceRedactionReplacements symbols, and include a test case that exercises the traceID trimming behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@framework/tracing/tracer.go`:
- Around line 113-123: Add unit tests for the exported
Tracer.SetTraceRedactionReplacements method to cover the security-sensitive
redaction path and its guard clauses. Verify it returns early when Tracer is
nil, store is nil, traceID is blank, replacements is empty, or GetTrace returns
nil, and confirm it calls Trace.SetRedactionReplacements for a valid trace. Use
the Tracer and SetTraceRedactionReplacements symbols, and include a test case
that exercises the traceID trimming behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: cacdfacd-f37b-4a42-a39e-020b12433272
📒 Files selected for processing (24)
core/schemas/bifrost.gocore/schemas/chatcompletions.gocore/schemas/guardraildebug.gocore/schemas/guardraildebug_test.gocore/schemas/mux.gocore/schemas/tracer.goframework/logstore/migrations.goframework/logstore/payload.goframework/logstore/payload_test.goframework/logstore/tables.goframework/modelcatalog/datasheet/cost.goframework/modelcatalog/datasheet/cost_test.goframework/modelcatalog/pricing.goframework/streaming/chat.goframework/streaming/responses.goframework/streaming/types.goframework/tracing/tracer.goplugins/logging/main.goplugins/logging/operations.goplugins/logging/operations_test.goplugins/logging/utils.goplugins/logging/writer.goui/app/workspace/logs/sheets/logDetailView.tsxui/lib/types/logs.ts
✅ Files skipped from review due to trivial changes (1)
- core/schemas/tracer.go
🚧 Files skipped from review as they are similar to previous changes (22)
- core/schemas/mux.go
- core/schemas/bifrost.go
- plugins/logging/writer.go
- plugins/logging/utils.go
- framework/modelcatalog/pricing.go
- ui/lib/types/logs.ts
- core/schemas/guardraildebug_test.go
- framework/streaming/responses.go
- ui/app/workspace/logs/sheets/logDetailView.tsx
- framework/logstore/payload_test.go
- framework/logstore/tables.go
- plugins/logging/operations_test.go
- framework/modelcatalog/datasheet/cost_test.go
- core/schemas/chatcompletions.go
- core/schemas/guardraildebug.go
- plugins/logging/operations.go
- framework/streaming/chat.go
- framework/modelcatalog/datasheet/cost.go
- framework/logstore/migrations.go
- framework/streaming/types.go
- framework/logstore/payload.go
- plugins/logging/main.go
c6669ce to
f497d95
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
ui/app/workspace/logs/sheets/logDetailView.tsx (1)
1575-1577: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider a typed constant for the
GUARDRAIL_INTERVENEDaction string.The comparison hardcodes the raw literal
"GUARDRAIL_INTERVENED"against a loosely-typedaction?: stringfield. A shared enum/constant (co-located with theGuardrailJudgeCalltype) would prevent drift if the backend action values change or add new states (e.g., a future "warn" action would silently fall into the "Allowed" bucket here).♻️ Suggested constant extraction
+// in ui/lib/types/logs.ts or a shared constants file +export const GUARDRAIL_ACTION_INTERVENED = "GUARDRAIL_INTERVENED";-<Badge variant={call.action === "GUARDRAIL_INTERVENED" ? "destructive" : "success"}> - {call.action === "GUARDRAIL_INTERVENED" ? "Blocked" : "Allowed"} +<Badge variant={call.action === GUARDRAIL_ACTION_INTERVENED ? "destructive" : "success"}> + {call.action === GUARDRAIL_ACTION_INTERVENED ? "Blocked" : "Allowed"} </Badge>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ui/app/workspace/logs/sheets/logDetailView.tsx` around lines 1575 - 1577, The `GUARDRAIL_INTERVENED` check in `logDetailView.tsx` is hardcoded against a loosely typed `call.action` string, so extract a shared typed constant or enum near `GuardrailJudgeCall` and use it here instead of the raw literal. Update the badge logic to compare against that shared symbol so future backend action changes stay in sync and new states don’t silently map to the wrong label.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@ui/app/workspace/logs/sheets/logDetailView.tsx`:
- Around line 1575-1577: The `GUARDRAIL_INTERVENED` check in `logDetailView.tsx`
is hardcoded against a loosely typed `call.action` string, so extract a shared
typed constant or enum near `GuardrailJudgeCall` and use it here instead of the
raw literal. Update the badge logic to compare against that shared symbol so
future backend action changes stay in sync and new states don’t silently map to
the wrong label.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 8fd76a77-37cc-4bd7-94bd-14618303c3df
📒 Files selected for processing (24)
core/schemas/bifrost.gocore/schemas/chatcompletions.gocore/schemas/guardraildebug.gocore/schemas/guardraildebug_test.gocore/schemas/mux.gocore/schemas/tracer.goframework/logstore/migrations.goframework/logstore/payload.goframework/logstore/payload_test.goframework/logstore/tables.goframework/modelcatalog/datasheet/cost.goframework/modelcatalog/datasheet/cost_test.goframework/modelcatalog/pricing.goframework/streaming/chat.goframework/streaming/responses.goframework/streaming/types.goframework/tracing/tracer.goplugins/logging/main.goplugins/logging/operations.goplugins/logging/operations_test.goplugins/logging/utils.goplugins/logging/writer.goui/app/workspace/logs/sheets/logDetailView.tsxui/lib/types/logs.ts
✅ Files skipped from review due to trivial changes (2)
- framework/tracing/tracer.go
- plugins/logging/writer.go
🚧 Files skipped from review as they are similar to previous changes (21)
- plugins/logging/utils.go
- framework/modelcatalog/pricing.go
- core/schemas/tracer.go
- ui/lib/types/logs.ts
- framework/streaming/responses.go
- core/schemas/guardraildebug_test.go
- framework/logstore/payload_test.go
- core/schemas/mux.go
- core/schemas/bifrost.go
- framework/logstore/tables.go
- plugins/logging/main.go
- core/schemas/chatcompletions.go
- core/schemas/guardraildebug.go
- framework/logstore/payload.go
- framework/logstore/migrations.go
- plugins/logging/operations.go
- plugins/logging/operations_test.go
- framework/streaming/types.go
- framework/modelcatalog/datasheet/cost_test.go
- framework/streaming/chat.go
- framework/modelcatalog/datasheet/cost.go
f497d95 to
4791e32
Compare
The merge-base changed after approval.
78d7e32 to
8467be6
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/schemas/guardraildebug.go`:
- Around line 100-109: The read-modify-write sequence in
AppendGuardrailJudgeCallOnContext is not atomic, so concurrent appends can lose
judge calls. Add synchronization covering the GuardrailDebugFromContext,
JudgeCalls append, and SetGuardrailDebugOnContext operations, and add a
concurrent test that invokes multiple appends and verifies every call is
retained.
In `@plugins/logging/main.go`:
- Around line 267-276: Update the module/workspace configuration for
plugins/logging so GOWORK=off resolves a core version containing
schemas.BifrostGuardrailDebug, using a workspace with the local core, framework,
and plugins/logging modules or publishing and pinning compatible versions. Do
not add a duplicate schema type, and ensure dependency resolution succeeds
independently of go mod tidy.
- Around line 278-299: Update the streaming fallback/error handling around the
code that creates log entries when streamResponse is nil or both result and
bifrostErr are non-nil, ensuring these Path B entries still calculate and apply
provider, cache-embedding, and guardrail-judge costs. Reuse the existing
CalculateCost flow and applyInternalCallCosts symbols without double-counting
costs already calculated; add tests covering both fallback paths and verify all
applicable cost components are preserved.
In `@plugins/semanticcache/search.go`:
- Around line 69-75: The semantic-cache module must resolve a core revision
containing schemas.SetCacheDebugOnContext and schemas.CacheDebugFromContext.
Update plugins/semanticcache/go.mod and go.sum, then run go mod tidy in
plugins/semanticcache; apply this dependency-only fix for
plugins/semanticcache/search.go:69-75 and
plugins/semanticcache/plugin_paths_test.go:473-475.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d809cbce-3389-429e-95ca-f63dc8426bef
📒 Files selected for processing (28)
core/schemas/bifrost.gocore/schemas/cachedebug.gocore/schemas/cachedebug_test.gocore/schemas/chatcompletions.gocore/schemas/guardraildebug.gocore/schemas/guardraildebug_test.gocore/schemas/mux.gocore/schemas/tracer.goframework/logstore/migrations.goframework/logstore/payload.goframework/logstore/payload_test.goframework/logstore/tables.goframework/modelcatalog/datasheet/cost.goframework/modelcatalog/datasheet/cost_test.goframework/modelcatalog/pricing.goframework/streaming/chat.goframework/streaming/responses.goframework/streaming/types.goframework/tracing/tracer.goplugins/logging/main.goplugins/logging/operations.goplugins/logging/operations_test.goplugins/logging/utils.goplugins/logging/writer.goplugins/semanticcache/plugin_paths_test.goplugins/semanticcache/search.goui/app/workspace/logs/sheets/logDetailView.tsxui/lib/types/logs.ts
🚧 Files skipped from review as they are similar to previous changes (19)
- plugins/logging/writer.go
- plugins/logging/utils.go
- framework/streaming/responses.go
- ui/lib/types/logs.ts
- framework/logstore/payload.go
- core/schemas/mux.go
- framework/modelcatalog/pricing.go
- framework/logstore/payload_test.go
- framework/streaming/chat.go
- framework/tracing/tracer.go
- ui/app/workspace/logs/sheets/logDetailView.tsx
- framework/streaming/types.go
- core/schemas/chatcompletions.go
- core/schemas/guardraildebug_test.go
- plugins/logging/operations.go
- framework/logstore/migrations.go
- framework/logstore/tables.go
- core/schemas/tracer.go
- framework/modelcatalog/datasheet/cost.go
8467be6 to
2544761
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Merge activity
|
## Summary Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI. ## Changes - Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state. - Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types. - Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response. - Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution. - Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support. - Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming. - Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly. - Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: 1. Send a request through a guardrail rule that triggers a judge call. 2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts. 3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts. 4. Confirm `cost` on the log entry reflects both the main request and the judge call spend. 5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations `guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI. ## Changes - Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state. - Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types. - Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response. - Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution. - Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support. - Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming. - Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly. - Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: 1. Send a request through a guardrail rule that triggers a judge call. 2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts. 3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts. 4. Confirm `cost` on the log entry reflects both the main request and the judge call spend. 5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations `guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI. ## Changes - Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state. - Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types. - Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response. - Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution. - Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support. - Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming. - Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly. - Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: 1. Send a request through a guardrail rule that triggers a judge call. 2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts. 3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts. 4. Confirm `cost` on the log entry reflects both the main request and the judge call spend. 5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations `guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI. ## Changes - Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state. - Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types. - Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response. - Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution. - Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support. - Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming. - Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly. - Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: 1. Send a request through a guardrail rule that triggers a judge call. 2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts. 3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts. 4. Confirm `cost` on the log entry reflects both the main request and the judge call spend. 5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations `guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable
## Summary Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI. ## Changes - Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state. - Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types. - Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response. - Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution. - Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support. - Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming. - Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly. - Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [ ] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Core/Transports go version go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/... # UI cd ui pnpm i || npm i pnpm build || npm run build ``` To validate end-to-end: 1. Send a request through a guardrail rule that triggers a judge call. 2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts. 3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts. 4. Confirm `cost` on the log entry reflects both the main request and the judge call spend. 5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log. ## Breaking changes - [ ] Yes - [x] No ## Related issues ## Security considerations `guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive. ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable

Summary
Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new
guardrail_debugfield on responses, log entries, and the UI.Changes
BifrostGuardrailDebugandBifrostGuardrailJudgeCallschema types in a newguardraildebug.gofile, with typed context helpers (GuardrailDebugFromContext,SetGuardrailDebugOnContext,AppendGuardrailJudgeCallOnContext) that enforce copy-on-read isolation so callers cannot mutate context state.BifrostContextKeyGuardrailDebugcontext key andGuardrailDebug *BifrostGuardrailDebugtoBifrostResponseExtraFields, propagated through all response conversion paths (ToTextCompletionResponse,ToBifrostTextCompletionResponse) and all streaming accumulators and chunk types.StreamAccumulatorResultandAccumulatedDatawithGuardrailDebugso streaming pipelines carry the field through to the final assembled response.CalculateGuardrailCostto the model catalog datasheet and exposed it viaModelCatalog.CalculateCostnow adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.guardrail_debugcolumn to the logstoreLogtable via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.PostLLMHookto read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost toentry.Cost— including for error paths and streaming.calculateCostForLogto treat a non-nilguardrailDebugas sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.GuardrailDebugandGuardrailJudgeCallTypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.Type of change
Affected areas
How to test
To validate end-to-end:
extra_fields.guardrail_debug.judge_callsis populated with provider, model, and token counts.coston the log entry reflects both the main request and the judge call spend.guardrail_debugand cost are still written to the log.Breaking changes
Related issues
Security considerations
guardrail_debugis written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). Thereasonfield may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.Checklist
docs/contributing/README.mdand followed the guidelines