Skip to content

cost for prompt guardrails - #4931

Merged
akshaydeo merged 1 commit into
devfrom
07-05-cost_for_prompt_guardrails
Aug 10, 2026
Merged

cost for prompt guardrails#4931
akshaydeo merged 1 commit into
devfrom
07-05-cost_for_prompt_guardrails

Conversation

@Madhuvod

@Madhuvod Madhuvod commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new guardrail_debug field on responses, log entries, and the UI.

Changes

  • Introduced BifrostGuardrailDebug and BifrostGuardrailJudgeCall schema types in a new guardraildebug.go file, with typed context helpers (GuardrailDebugFromContext, SetGuardrailDebugOnContext, AppendGuardrailJudgeCallOnContext) that enforce copy-on-read isolation so callers cannot mutate context state.
  • Added BifrostContextKeyGuardrailDebug context key and GuardrailDebug *BifrostGuardrailDebug to BifrostResponseExtraFields, propagated through all response conversion paths (ToTextCompletionResponse, ToBifrostTextCompletionResponse) and all streaming accumulators and chunk types.
  • Extended StreamAccumulatorResult and AccumulatedData with GuardrailDebug so streaming pipelines carry the field through to the final assembled response.
  • Added CalculateGuardrailCost to the model catalog datasheet and exposed it via ModelCatalog. CalculateCost now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
  • Added a guardrail_debug column to the logstore Log table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
  • Updated the logging plugin's PostLLMHook to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to entry.Cost — including for error paths and streaming.
  • Updated calculateCostForLog to treat a non-nil guardrailDebug as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
  • Added GuardrailDebug and GuardrailJudgeCall TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

Type of change

  • Bug fix
  • Feature
  • Refactor
  • Documentation
  • Chore/CI

Affected areas

  • Core (Go)
  • Transports (HTTP)
  • Providers/Integrations
  • Plugins
  • UI (React)
  • Docs

How to test

# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build

To validate end-to-end:

  1. Send a request through a guardrail rule that triggers a judge call.
  2. Confirm the response extra_fields.guardrail_debug.judge_calls is populated with provider, model, and token counts.
  3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
  4. Confirm cost on the log entry reflects both the main request and the judge call spend.
  5. For an input-blocked request (no provider response), confirm guardrail_debug and cost are still written to the log.

Breaking changes

  • Yes
  • No

Related issues

Security considerations

guardrail_debug is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The reason field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

Checklist

  • I read docs/contributing/README.md and followed the guidelines
  • I added/updated tests where appropriate
  • I updated documentation where needed
  • I verified builds succeed (Go and UI)
  • I verified the CI pipeline passes locally if applicable

@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 17ef8a60-125b-4f31-982a-aa3168243760

📥 Commits

Reviewing files that changed from the base of the PR and between 4742c6c and 2544761.

📒 Files selected for processing (28)
  • core/schemas/bifrost.go
  • core/schemas/cachedebug.go
  • core/schemas/cachedebug_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • core/schemas/guardraildebug_test.go
  • core/schemas/mux.go
  • core/schemas/tracer.go
  • framework/logstore/migrations.go
  • framework/logstore/payload.go
  • framework/logstore/payload_test.go
  • framework/logstore/tables.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/pricing.go
  • framework/streaming/chat.go
  • framework/streaming/responses.go
  • framework/streaming/types.go
  • framework/tracing/tracer.go
  • plugins/logging/main.go
  • plugins/logging/operations.go
  • plugins/logging/operations_test.go
  • plugins/logging/utils.go
  • plugins/logging/writer.go
  • plugins/semanticcache/plugin_paths_test.go
  • plugins/semanticcache/search.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • ui/lib/types/logs.ts
🚧 Files skipped from review as they are similar to previous changes (26)
  • plugins/logging/utils.go
  • ui/lib/types/logs.ts
  • framework/tracing/tracer.go
  • core/schemas/tracer.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • framework/modelcatalog/pricing.go
  • framework/logstore/tables.go
  • core/schemas/chatcompletions.go
  • plugins/logging/operations.go
  • core/schemas/cachedebug_test.go
  • plugins/semanticcache/plugin_paths_test.go
  • framework/logstore/migrations.go
  • framework/streaming/responses.go
  • core/schemas/guardraildebug_test.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/logstore/payload_test.go
  • framework/streaming/types.go
  • core/schemas/cachedebug.go
  • framework/streaming/chat.go
  • core/schemas/guardraildebug.go
  • core/schemas/mux.go
  • core/schemas/bifrost.go
  • framework/logstore/payload.go
  • framework/modelcatalog/datasheet/cost_test.go
  • plugins/logging/writer.go
  • plugins/logging/operations_test.go

📝 Walkthrough

Summary by CodeRabbit

  • New Features
    • Added guardrail debug details to responses, streaming results, and stored logs.
    • Added semantic-cache embedding metadata and related cost calculation.
    • Added per-judge guardrail cost calculation to reported request costs.
    • Added a “Guardrail Details” section showing rules, actions, models, reasons, and token usage.
  • Bug Fixes
    • Preserved guardrail details across response and logging paths, including token-less scenarios.
  • Tests
    • Added coverage for guardrail context handling, cache metadata, and billing calculations.

Walkthrough

This PR adds request-scoped guardrail and semantic-cache debug data. It propagates guardrail data through responses and streams, calculates related costs, persists log metadata, records embedding metadata, and displays guardrail judge-call details.

Changes

Guardrail and Cache Debug Billing

Layer / File(s) Summary
Debug schemas and context snapshots
core/schemas/bifrost.go, core/schemas/cachedebug.go, core/schemas/guardraildebug.go, core/schemas/*_test.go
Defines debug records, context helpers, response metadata, cloning, validation, and snapshot tests.
Response and streaming propagation
core/schemas/chatcompletions.go, core/schemas/mux.go, core/schemas/tracer.go, framework/streaming/*, framework/tracing/tracer.go, plugins/logging/utils.go
Carries guardrail debug data through response conversions, stream chunks, accumulation, tracing, and processed responses.
Guardrail and cache cost calculation
framework/modelcatalog/datasheet/cost.go, framework/modelcatalog/datasheet/cost_test.go, framework/modelcatalog/pricing.go
Prices guardrail judge calls and semantic-cache embeddings. Adds these costs to request and cache-related pricing paths.
Log persistence and payload handling
framework/logstore/migrations.go, framework/logstore/tables.go, framework/logstore/payload.go, framework/logstore/payload_test.go
Adds the persisted guardrail debug column, parsed log data, migration handling, payload operations, serialization, deserialization, and round-trip tests.
Logging capture and sidecar costs
plugins/logging/main.go, plugins/logging/operations.go, plugins/logging/operations_test.go, plugins/logging/utils.go, plugins/logging/writer.go
Captures debug data, applies guardrail and cache sidecar costs, supports streaming and error entries, and updates log size estimates.
Semantic-cache metadata and guardrail details UI
plugins/semanticcache/search.go, plugins/semanticcache/plugin_paths_test.go, ui/lib/types/logs.ts, ui/app/workspace/logs/sheets/logDetailView.tsx
Records embedding metadata in context and adds typed guardrail data and judge-call details to the log UI.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested reviewers: akshaydeo, danpiths, pratham-mishra04

Sequence Diagram(s)

sequenceDiagram
  participant GuardrailPlugin
  participant SemanticCache
  participant BifrostContext
  participant ResponsePipeline
  participant LoggingPlugin
  participant LogStore
  participant LogDetailView
  GuardrailPlugin->>BifrostContext: append judge-call debug
  SemanticCache->>BifrostContext: store embedding debug
  ResponsePipeline->>BifrostContext: read debug snapshots
  ResponsePipeline->>LoggingPlugin: propagate response and stream metadata
  LoggingPlugin->>LogStore: persist debug data and sidecar costs
  LogStore-->>LogDetailView: provide guardrail_debug
  LogDetailView-->>LogDetailView: render judge-call details
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title identifies the main change: adding cost calculation for prompt guardrail judge calls.
Description check ✅ Passed The description covers the purpose, implementation, affected areas, testing steps, security considerations, and checklist for this feature.
Docstring Coverage ✅ Passed Docstring coverage is 81.40% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 07-05-cost_for_prompt_guardrails

Comment @coderabbitai help to get the list of available commands.

Madhuvod commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Madhuvod
Madhuvod marked this pull request as ready for review July 5, 2026 17:10
@coderabbitai
coderabbitai Bot requested review from akshaydeo and danpiths July 5, 2026 17:11
@greptile-apps

greptile-apps Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Confidence Score: 4/5

Safe to merge with awareness of the streaming error path gap noted in the previous review cycle.

The guardrail debug and cost-attribution logic is correct for input-blocked requests (Path A), non-streaming requests (Path C), and streaming success (Path B final chunk). The one unresolved gap — streaming Path B when bifrostErr is non-nil and result is also non-nil — was identified in the previous review: in that branch, applyGuardrailCost is never called and the accumulated cost in the stream response is also not applied to the log entry. All other paths are properly handled, tests are thorough, and the migration is safe.

plugins/logging/main.go — the streaming Path B error branch (bifrostErr != nil, result != nil) does not apply guardrail cost to the log entry.

Important Files Changed

Filename Overview
core/schemas/guardraildebug.go New schema types and context helpers with correct nil-safe Clone(), copy-on-read semantics verified by tests.
plugins/logging/main.go Guardrail debug read/propagation and cost application added to all major paths; streaming Path B error-with-non-nil-result branch (flagged in a previous review) still missing applyGuardrailCost.
framework/modelcatalog/datasheet/cost.go CalculateCost now adds guardrail judge-call cost; calculateBaseCost called on the non-cache branch with correct structure.
framework/logstore/migrations.go Adds guardrail_debug TEXT column via addColumnIfNotExists; nullable with no default, uses same transactional pattern as all prior column migrations, rollback provided.
plugins/logging/operations.go applyStreamingOutputToEntry and calculateCostForLog updated correctly; guardrail-only input-blocked cost path works via buildResponseForRequestType default ChatResponse branch.

Reviews (8): Last reviewed commit: "cost for prompt guardrails" | Re-trigger Greptile

Comment thread framework/modelcatalog/datasheet/cost.go
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 5, 2026
@Madhuvod
Madhuvod force-pushed the 07-05-cost_for_prompt_guardrails branch from 43cd183 to 8c9a3dd Compare July 5, 2026 17:43
@coderabbitai
coderabbitai Bot requested a review from roroghost17 July 5, 2026 17:44
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 5, 2026
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 5, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
framework/tracing/tracer.go (1)

113-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding test coverage for SetTraceRedactionReplacements.

The guard clauses look correct and the underlying Trace.SetRedactionReplacements is mutex-protected, but this new exported method (used for redaction, a security-sensitive path) doesn't appear to have accompanying unit tests in the provided context.

As per path instructions, framework/**: "Apply standard Go review practices... and tests that cover edge cases and failure paths."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@framework/tracing/tracer.go` around lines 113 - 123, Add unit tests for the
exported Tracer.SetTraceRedactionReplacements method to cover the
security-sensitive redaction path and its guard clauses. Verify it returns early
when Tracer is nil, store is nil, traceID is blank, replacements is empty, or
GetTrace returns nil, and confirm it calls Trace.SetRedactionReplacements for a
valid trace. Use the Tracer and SetTraceRedactionReplacements symbols, and
include a test case that exercises the traceID trimming behavior.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@framework/tracing/tracer.go`:
- Around line 113-123: Add unit tests for the exported
Tracer.SetTraceRedactionReplacements method to cover the security-sensitive
redaction path and its guard clauses. Verify it returns early when Tracer is
nil, store is nil, traceID is blank, replacements is empty, or GetTrace returns
nil, and confirm it calls Trace.SetRedactionReplacements for a valid trace. Use
the Tracer and SetTraceRedactionReplacements symbols, and include a test case
that exercises the traceID trimming behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cacdfacd-f37b-4a42-a39e-020b12433272

📥 Commits

Reviewing files that changed from the base of the PR and between 2e58579 and c6669ce.

📒 Files selected for processing (24)
  • core/schemas/bifrost.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • core/schemas/guardraildebug_test.go
  • core/schemas/mux.go
  • core/schemas/tracer.go
  • framework/logstore/migrations.go
  • framework/logstore/payload.go
  • framework/logstore/payload_test.go
  • framework/logstore/tables.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/pricing.go
  • framework/streaming/chat.go
  • framework/streaming/responses.go
  • framework/streaming/types.go
  • framework/tracing/tracer.go
  • plugins/logging/main.go
  • plugins/logging/operations.go
  • plugins/logging/operations_test.go
  • plugins/logging/utils.go
  • plugins/logging/writer.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • ui/lib/types/logs.ts
✅ Files skipped from review due to trivial changes (1)
  • core/schemas/tracer.go
🚧 Files skipped from review as they are similar to previous changes (22)
  • core/schemas/mux.go
  • core/schemas/bifrost.go
  • plugins/logging/writer.go
  • plugins/logging/utils.go
  • framework/modelcatalog/pricing.go
  • ui/lib/types/logs.ts
  • core/schemas/guardraildebug_test.go
  • framework/streaming/responses.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • framework/logstore/payload_test.go
  • framework/logstore/tables.go
  • plugins/logging/operations_test.go
  • framework/modelcatalog/datasheet/cost_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • plugins/logging/operations.go
  • framework/streaming/chat.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/logstore/migrations.go
  • framework/streaming/types.go
  • framework/logstore/payload.go
  • plugins/logging/main.go

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 7, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
ui/app/workspace/logs/sheets/logDetailView.tsx (1)

1575-1577: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider a typed constant for the GUARDRAIL_INTERVENED action string.

The comparison hardcodes the raw literal "GUARDRAIL_INTERVENED" against a loosely-typed action?: string field. A shared enum/constant (co-located with the GuardrailJudgeCall type) would prevent drift if the backend action values change or add new states (e.g., a future "warn" action would silently fall into the "Allowed" bucket here).

♻️ Suggested constant extraction
+// in ui/lib/types/logs.ts or a shared constants file
+export const GUARDRAIL_ACTION_INTERVENED = "GUARDRAIL_INTERVENED";
-<Badge variant={call.action === "GUARDRAIL_INTERVENED" ? "destructive" : "success"}>
-  {call.action === "GUARDRAIL_INTERVENED" ? "Blocked" : "Allowed"}
+<Badge variant={call.action === GUARDRAIL_ACTION_INTERVENED ? "destructive" : "success"}>
+  {call.action === GUARDRAIL_ACTION_INTERVENED ? "Blocked" : "Allowed"}
 </Badge>
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ui/app/workspace/logs/sheets/logDetailView.tsx` around lines 1575 - 1577, The
`GUARDRAIL_INTERVENED` check in `logDetailView.tsx` is hardcoded against a
loosely typed `call.action` string, so extract a shared typed constant or enum
near `GuardrailJudgeCall` and use it here instead of the raw literal. Update the
badge logic to compare against that shared symbol so future backend action
changes stay in sync and new states don’t silently map to the wrong label.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@ui/app/workspace/logs/sheets/logDetailView.tsx`:
- Around line 1575-1577: The `GUARDRAIL_INTERVENED` check in `logDetailView.tsx`
is hardcoded against a loosely typed `call.action` string, so extract a shared
typed constant or enum near `GuardrailJudgeCall` and use it here instead of the
raw literal. Update the badge logic to compare against that shared symbol so
future backend action changes stay in sync and new states don’t silently map to
the wrong label.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8fd76a77-37cc-4bd7-94bd-14618303c3df

📥 Commits

Reviewing files that changed from the base of the PR and between c6669ce and f497d95.

📒 Files selected for processing (24)
  • core/schemas/bifrost.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • core/schemas/guardraildebug_test.go
  • core/schemas/mux.go
  • core/schemas/tracer.go
  • framework/logstore/migrations.go
  • framework/logstore/payload.go
  • framework/logstore/payload_test.go
  • framework/logstore/tables.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/pricing.go
  • framework/streaming/chat.go
  • framework/streaming/responses.go
  • framework/streaming/types.go
  • framework/tracing/tracer.go
  • plugins/logging/main.go
  • plugins/logging/operations.go
  • plugins/logging/operations_test.go
  • plugins/logging/utils.go
  • plugins/logging/writer.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • ui/lib/types/logs.ts
✅ Files skipped from review due to trivial changes (2)
  • framework/tracing/tracer.go
  • plugins/logging/writer.go
🚧 Files skipped from review as they are similar to previous changes (21)
  • plugins/logging/utils.go
  • framework/modelcatalog/pricing.go
  • core/schemas/tracer.go
  • ui/lib/types/logs.ts
  • framework/streaming/responses.go
  • core/schemas/guardraildebug_test.go
  • framework/logstore/payload_test.go
  • core/schemas/mux.go
  • core/schemas/bifrost.go
  • framework/logstore/tables.go
  • plugins/logging/main.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • framework/logstore/payload.go
  • framework/logstore/migrations.go
  • plugins/logging/operations.go
  • plugins/logging/operations_test.go
  • framework/streaming/types.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/streaming/chat.go
  • framework/modelcatalog/datasheet/cost.go

coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 7, 2026
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jul 9, 2026
@akshaydeo
akshaydeo dismissed coderabbitai[bot]’s stale review July 14, 2026 08:11

The merge-base changed after approval.

@akshaydeo
akshaydeo requested a review from a team as a code owner July 14, 2026 08:11
coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 6, 2026
@Madhuvod Madhuvod mentioned this pull request Aug 6, 2026
18 tasks
Comment thread framework/modelcatalog/datasheet/cost.go Outdated
Comment thread plugins/logging/main.go Outdated
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@coderabbitai
coderabbitai Bot requested a review from Pratham-Mishra04 August 7, 2026 08:00

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/schemas/guardraildebug.go`:
- Around line 100-109: The read-modify-write sequence in
AppendGuardrailJudgeCallOnContext is not atomic, so concurrent appends can lose
judge calls. Add synchronization covering the GuardrailDebugFromContext,
JudgeCalls append, and SetGuardrailDebugOnContext operations, and add a
concurrent test that invokes multiple appends and verifies every call is
retained.

In `@plugins/logging/main.go`:
- Around line 267-276: Update the module/workspace configuration for
plugins/logging so GOWORK=off resolves a core version containing
schemas.BifrostGuardrailDebug, using a workspace with the local core, framework,
and plugins/logging modules or publishing and pinning compatible versions. Do
not add a duplicate schema type, and ensure dependency resolution succeeds
independently of go mod tidy.
- Around line 278-299: Update the streaming fallback/error handling around the
code that creates log entries when streamResponse is nil or both result and
bifrostErr are non-nil, ensuring these Path B entries still calculate and apply
provider, cache-embedding, and guardrail-judge costs. Reuse the existing
CalculateCost flow and applyInternalCallCosts symbols without double-counting
costs already calculated; add tests covering both fallback paths and verify all
applicable cost components are preserved.

In `@plugins/semanticcache/search.go`:
- Around line 69-75: The semantic-cache module must resolve a core revision
containing schemas.SetCacheDebugOnContext and schemas.CacheDebugFromContext.
Update plugins/semanticcache/go.mod and go.sum, then run go mod tidy in
plugins/semanticcache; apply this dependency-only fix for
plugins/semanticcache/search.go:69-75 and
plugins/semanticcache/plugin_paths_test.go:473-475.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d809cbce-3389-429e-95ca-f63dc8426bef

📥 Commits

Reviewing files that changed from the base of the PR and between 4742c6c and 8467be6.

📒 Files selected for processing (28)
  • core/schemas/bifrost.go
  • core/schemas/cachedebug.go
  • core/schemas/cachedebug_test.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug.go
  • core/schemas/guardraildebug_test.go
  • core/schemas/mux.go
  • core/schemas/tracer.go
  • framework/logstore/migrations.go
  • framework/logstore/payload.go
  • framework/logstore/payload_test.go
  • framework/logstore/tables.go
  • framework/modelcatalog/datasheet/cost.go
  • framework/modelcatalog/datasheet/cost_test.go
  • framework/modelcatalog/pricing.go
  • framework/streaming/chat.go
  • framework/streaming/responses.go
  • framework/streaming/types.go
  • framework/tracing/tracer.go
  • plugins/logging/main.go
  • plugins/logging/operations.go
  • plugins/logging/operations_test.go
  • plugins/logging/utils.go
  • plugins/logging/writer.go
  • plugins/semanticcache/plugin_paths_test.go
  • plugins/semanticcache/search.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • ui/lib/types/logs.ts
🚧 Files skipped from review as they are similar to previous changes (19)
  • plugins/logging/writer.go
  • plugins/logging/utils.go
  • framework/streaming/responses.go
  • ui/lib/types/logs.ts
  • framework/logstore/payload.go
  • core/schemas/mux.go
  • framework/modelcatalog/pricing.go
  • framework/logstore/payload_test.go
  • framework/streaming/chat.go
  • framework/tracing/tracer.go
  • ui/app/workspace/logs/sheets/logDetailView.tsx
  • framework/streaming/types.go
  • core/schemas/chatcompletions.go
  • core/schemas/guardraildebug_test.go
  • plugins/logging/operations.go
  • framework/logstore/migrations.go
  • framework/logstore/tables.go
  • core/schemas/tracer.go
  • framework/modelcatalog/datasheet/cost.go

Comment thread core/schemas/guardraildebug.go
Comment thread plugins/logging/main.go
Comment thread plugins/logging/main.go
Comment thread plugins/semanticcache/search.go
@Madhuvod
Madhuvod force-pushed the 07-05-cost_for_prompt_guardrails branch from 8467be6 to 2544761 Compare August 7, 2026 08:27
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

akshaydeo commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Merge activity

  • Aug 10, 3:12 PM UTC: A user started a stack merge that includes this pull request via Graphite.
  • Aug 10, 3:14 PM UTC: @akshaydeo merged this pull request with Graphite.

@akshaydeo
akshaydeo merged commit dfffd11 into dev Aug 10, 2026
14 checks passed
@akshaydeo
akshaydeo deleted the 07-05-cost_for_prompt_guardrails branch August 10, 2026 15:14
akshaydeo pushed a commit that referenced this pull request Aug 10, 2026
## Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI.

## Changes

- Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state.
- Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types.
- Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response.
- Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
- Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
- Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming.
- Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
- Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end:
1. Send a request through a guardrail rule that triggers a judge call.
2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts.
3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
4. Confirm `cost` on the log entry reflects both the main request and the judge call spend.
5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log.

## Breaking changes

- [ ] Yes
- [x] No

## Related issues

## Security considerations

`guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

## Checklist

- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Aug 13, 2026
## Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI.

## Changes

- Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state.
- Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types.
- Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response.
- Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
- Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
- Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming.
- Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
- Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end:
1. Send a request through a guardrail rule that triggers a judge call.
2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts.
3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
4. Confirm `cost` on the log entry reflects both the main request and the judge call spend.
5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log.

## Breaking changes

- [ ] Yes
- [x] No

## Related issues

## Security considerations

`guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

## Checklist

- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Aug 13, 2026
## Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI.

## Changes

- Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state.
- Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types.
- Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response.
- Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
- Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
- Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming.
- Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
- Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end:
1. Send a request through a guardrail rule that triggers a judge call.
2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts.
3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
4. Confirm `cost` on the log entry reflects both the main request and the judge call spend.
5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log.

## Breaking changes

- [ ] Yes
- [x] No

## Related issues

## Security considerations

`guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

## Checklist

- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Aug 14, 2026
## Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI.

## Changes

- Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state.
- Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types.
- Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response.
- Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
- Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
- Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming.
- Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
- Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end:
1. Send a request through a guardrail rule that triggers a judge call.
2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts.
3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
4. Confirm `cost` on the log entry reflects both the main request and the judge call spend.
5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log.

## Breaking changes

- [ ] Yes
- [x] No

## Related issues

## Security considerations

`guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

## Checklist

- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
akshaydeo pushed a commit that referenced this pull request Aug 19, 2026
## Summary

Adds end-to-end observability for guardrail judge calls — the internal LLM invocations made by the enterprise guardrails plugin to evaluate rules. Previously, these calls were invisible: their token spend was untracked, their outcomes were not logged, and their cost was not reflected in billing. This PR surfaces that data through a new `guardrail_debug` field on responses, log entries, and the UI.

## Changes

- Introduced `BifrostGuardrailDebug` and `BifrostGuardrailJudgeCall` schema types in a new `guardraildebug.go` file, with typed context helpers (`GuardrailDebugFromContext`, `SetGuardrailDebugOnContext`, `AppendGuardrailJudgeCallOnContext`) that enforce copy-on-read isolation so callers cannot mutate context state.
- Added `BifrostContextKeyGuardrailDebug` context key and `GuardrailDebug *BifrostGuardrailDebug` to `BifrostResponseExtraFields`, propagated through all response conversion paths (`ToTextCompletionResponse`, `ToBifrostTextCompletionResponse`) and all streaming accumulators and chunk types.
- Extended `StreamAccumulatorResult` and `AccumulatedData` with `GuardrailDebug` so streaming pipelines carry the field through to the final assembled response.
- Added `CalculateGuardrailCost` to the model catalog datasheet and exposed it via `ModelCatalog`. `CalculateCost` now adds judge-call cost on top of the main request cost (including cache-hit paths). Judge cost is attributed to the judge's own provider/model, preserving virtual-key attribution.
- Added a `guardrail_debug` column to the logstore `Log` table via a new migration, with full serialize/deserialize, payload extraction, merge, and clear support.
- Updated the logging plugin's `PostLLMHook` to read guardrail debug from context (covering input-block cases where no provider response exists) and from the response, write it to the log entry, and apply guardrail cost to `entry.Cost` — including for error paths and streaming.
- Updated `calculateCostForLog` to treat a non-nil `guardrailDebug` as sufficient to proceed with cost calculation, so input-blocked requests are billed correctly.
- Added `GuardrailDebug` and `GuardrailJudgeCall` TypeScript types and rendered a "Guardrail Details" section in the log detail view showing rule, phase, action (Blocked/Allowed badge), guardrail name and provider, judge provider and model, token counts, and reason.

## Type of change

- [ ] Bug fix
- [x] Feature
- [ ] Refactor
- [ ] Documentation
- [ ] Chore/CI

## Affected areas

- [x] Core (Go)
- [ ] Transports (HTTP)
- [ ] Providers/Integrations
- [x] Plugins
- [x] UI (React)
- [ ] Docs

## How to test

```sh
# Core/Transports
go version
go test ./core/schemas/... ./framework/logstore/... ./framework/modelcatalog/... ./framework/streaming/... ./plugins/logging/...

# UI
cd ui
pnpm i || npm i
pnpm build || npm run build
```

To validate end-to-end:
1. Send a request through a guardrail rule that triggers a judge call.
2. Confirm the response `extra_fields.guardrail_debug.judge_calls` is populated with provider, model, and token counts.
3. Open the log detail view and verify the "Guardrail Details" section appears with correct phase, action badge, and token counts.
4. Confirm `cost` on the log entry reflects both the main request and the judge call spend.
5. For an input-blocked request (no provider response), confirm `guardrail_debug` and cost are still written to the log.

## Breaking changes

- [ ] Yes
- [x] No

## Related issues

## Security considerations

`guardrail_debug` is written to the log store and returned in API responses. It does not contain prompt content — only metadata (rule name, provider, model, token counts, action, reason). The `reason` field may contain guardrail-generated explanations; ensure content logging policies are applied consistently if reason strings are considered sensitive.

## Checklist

- [x] I read `docs/contributing/README.md` and followed the guidelines
- [x] I added/updated tests where appropriate
- [ ] I updated documentation where needed
- [x] I verified builds succeed (Go and UI)
- [ ] I verified the CI pipeline passes locally if applicable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants