feat(guardrails): roll up Bedrock guardrail cost per usage counter - #39196
Conversation
The daily guardrail usage rollup stored billable units per counter but no
cost, so the usage endpoints could only report units. The Bedrock hook now
stamps guardrail_cost_by_unit next to guardrail_usage, the spend-log
aggregator sums it into a new nullable cost column on
LiteLLM_DailyGuardrailUsageUnits, and /guardrails/usage/overview and
/guardrails/usage/detail/{id} return cost, totalCost and cost_by_unit /
cost_by_team / cost_by_key alongside the existing unit breakdowns.
Cost is nullable on purpose. Rows written before this migration, and rows
whose hook had no pricing entry, read as null rather than $0, and a single
unpriced increment keeps that row's cost unknown instead of partial.
guardrail_cost and the spend/budget path are untouched.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Greptile SummaryThe PR adds per-counter Bedrock guardrail cost tracking and exposes priced subtotals alongside explicitly untracked usage.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/llm_cost_calc/guardrail_cost.py | Produces nullable per-counter costs so missing prices remain distinguishable from explicitly free counters. |
| litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py | Stamps Bedrock usage with per-counter pricing while preserving the existing scalar spend calculation. |
| litellm/proxy/guardrails/usage_tracking.py | Accumulates priced cost and untracked units independently, resolving the previously reported mixed-increment loss. |
| litellm/proxy/guardrails/usage_endpoints.py | Exposes tracked cost and untracked-unit totals across overview, daily, unit, team, and key breakdowns. |
| litellm-proxy-extras/litellm_proxy_extras/migrations/20260901000001_add_guardrail_usage_units_cost/migration.sql | Adds the nullable cost and additive untracked_units columns required by the new rollup representation. |
Reviews (4): Last reviewed commit: "feat(guardrails): store untracked units ..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 4914914. Configure here.
A counter missing from the cost map entry was priced at 0.0 per unit, so the rollup recorded it as known-free usage. It now stamps None for that counter and the rollup writes NULL, while the per-request guardrail_cost that feeds spend and budgets still sums only the known prices. Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
|
@greptileai pushed the unknown-counter fix: an unpriced counter now stamps null and the rollup stores NULL, not $0 |
A row's cost sums only the daily rows that carry a tracked cost, so it silently under-reports whenever some rows are NULL (pre-migration days, old pods mid-rollout, an unpriced counter). Both usage endpoints now return the per-counter units behind those NULL rows next to the cost (untrackedUsageUnits / totalUntrackedUsageUnits on the overview, untracked_usage_units on the detail), so a partial cost is never mistaken for a complete one and the reader can see exactly what it excludes Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
|
@greptileai please re-review: each cost now ships with the per-counter units it leaves out (untrackedUsageUnits), plus tests and a live rolling-deploy run |
…nulling cost A row that received both priced and unpriced increments used to collapse to cost NULL, throwing away the priced subtotal and making every unit on it read as untracked. The rollup now carries a second column, untracked_units, that the aggregator increments for units with no known price while cost keeps accruing for the rest, so cost covers exactly units - untracked_units. Rows written before the migration keep cost NULL and still read as untracked in full The endpoints read untracked units off the column (or the whole row for a legacy NULL) rather than from a NULL filter, and the policies overview now fills totalUntrackedUsageUnits, which the previous commit missed Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
|
@greptileai please re-review: the rollup row now stores untracked_units, so mixed priced and unpriced increments keep the priced subtotal |
…itellm_guardrail_usage_cost_rollup # Conflicts: # type-discipline-budget.json
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 9bd34ad. Configure here.
d23bec8
into
litellm_internal_staging
TLDR
Problem this solves:
How it solves it:
guardrail_cost_by_unitnext toguardrail_usagecost(USD for the priced units) anduntracked_units(units that had no price), so a row that mixes priced and unpriced increments keeps its priced subtotal and says exactly how many units it leaves out/guardrails/usage/overviewgainscostanduntrackedUsageUnitsper row plustotalCostandtotalUntrackedUsageUnits/guardrails/usage/detail/{id}gainscost, daily cost,cost_by_unit/team/keyanduntracked_usage_unitscostNULL and read as untracked in full, never as $0User Flow
Before: a proxy admin wants to know what their Bedrock guardrail cost them this month and can only get unit counts
guardrail_costof0.0001"usageUnits": {"sensitiveInformationPolicyUnits": 3}but no cost field at allguardrail_costacross every spend log row by handAfter: the same admin reads cost straight off the guardrail usage endpoints
"guardrail_cost_by_unit": {"sensitiveInformationPolicyUnits": 0.0001}"usageUnits": {"sensitiveInformationPolicyUnits": 3}, "cost": 0.0003, "untrackedUsageUnits": {}with"totalCost": 0.0003at the top level"cost": 0.0003, acoston each daily point,cost_by_unit,cost_by_team,cost_by_keymaps next to the existing unit maps, and"untracked_usage_units": {}"cost": nullrather than pretending it was free, and itsuntrackedUsageUnitslists every unit the null coverscoststill covers the priced units anduntrackedUsageUnitsreads{"sensitiveInformationPolicyUnits": 2}, so they know the $0.0003 covers 3 of the 5 unitsRelevant issues
Linear ticket
Refs LIT-5652
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Shared setup. A Bedrock guardrail in
us-east-1with a sensitive information policy (SSN, anonymize) runsdefault_oninpre_callmode. Each request bills onesensitiveInformationPolicyUnits, priced at $0.0001 in the cost map. Both runs hit the real Bedrock ApplyGuardrail and Claude Haiku 4.5 APIs and use theproxy_batch_write_at: 5flush so the rollup lands within secondsThe request sent in both runs:
Before (8bc862f)
Send the request above, then wait for the spend flush
curl -s "http://localhost:4000/guardrails/usage/overview?start_date=2026-09-01&end_date=2026-09-01" -H 'Authorization: Bearer sk-1234', guardrail row and totals only (no cost fields exist){ "row": { "id": "eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c", "name": "bedrock-pii-mask", "type": "Guardrail", "provider": "bedrock", "requestsEvaluated": 4, "failRate": 0.0, "avgScore": null, "avgLatency": null, "status": "healthy", "trend": "stable", "usageUnits": { "sensitiveInformationPolicyUnits": 4 } }, "totalUsageUnits": { "sensitiveInformationPolicyUnits": 4 }, "totalCost": "<absent>" }curl -s "http://localhost:4000/guardrails/usage/detail/eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c?start_date=2026-09-01&end_date=2026-09-01" -H 'Authorization: Bearer sk-1234', unit fields only{ "usage_units": { "sensitiveInformationPolicyUnits": 4 }, "usage_units_daily": [ { "date": "2026-09-01", "units": { "sensitiveInformationPolicyUnits": 4 } } ], "cost": "<absent>", "cost_by_unit": "<absent>", "cost_by_team": "<absent>", "cost_by_key": "<absent>" }After (6c81a5c)
Three ways units end up unpriced, all on the same guardrail and day: a row written by the old build (no cost column at all), a priced row on this build, then an unpriced increment landing on that same priced row because the cost map had no price for the counter at the time
Boot log from this branch's first boot on the DB shows the schema-only migration applying before traffic is served (this run booted with both columns already present)
The old build (8bc862f) sends the request above once with the master key, then waits for the spend flush. Its aggregator knows nothing about cost, so the row lands with
costNULLSwap to this build,
curl -s http://localhost:4000/key/generate -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"key_alias":"guardrail-cost-qa","models":["bedrock-invoke-haiku-4-5"]}', and send the request above three times with the returned key (one of the three got a Bedrock 503 on the model call after the guardrail had already billed its unit, which is why it still counts)Restart this build with
LITELLM_MODEL_COST_MAP_URLpointing at a copy of the cost map whosebedrock/guardrailsentry has nosensitiveInformationPolicyUnitsprice (the state a deployment is in whenever Bedrock bills a counter before the map catches up), then send the request once more with the same keyThe rollup rows for the day, read straight from
LiteLLM_DailyGuardrailUsageUnits(date, guardrail_id, api_key, usage_unit, units, cost, untracked_units). The same row holds the priced subtotal and the unpriced unit instead of collapsing to NULLcurl -s "http://localhost:4000/guardrails/usage/overview?start_date=2026-09-04&end_date=2026-09-04" -H 'Authorization: Bearer sk-1234', the row prices the 3 units it could and names the 2 it could not (1 from the old build's row, 1 from the unpriced increment), and the totals do the same{ "row": { "id": "eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c", "name": "bedrock-pii-mask", "type": "Guardrail", "provider": "bedrock", "requestsEvaluated": 5, "failRate": 0.0, "avgScore": null, "avgLatency": null, "status": "healthy", "trend": "stable", "usageUnits": { "sensitiveInformationPolicyUnits": 5 }, "cost": 0.0003, "untrackedUsageUnits": { "sensitiveInformationPolicyUnits": 2 } }, "totalUsageUnits": { "sensitiveInformationPolicyUnits": 5 }, "totalCost": 0.0003, "totalUntrackedUsageUnits": { "sensitiveInformationPolicyUnits": 2 } }curl -s "http://localhost:4000/guardrails/usage/detail/eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c?start_date=2026-09-04&end_date=2026-09-04" -H 'Authorization: Bearer sk-1234', cost rides every unit breakdown, the master key's row (the old build's) readsnullincost_by_key, anduntracked_usage_unitscounts both unpriced units{ "usage_units": { "sensitiveInformationPolicyUnits": 5 }, "usage_units_daily": [ { "date": "2026-09-04", "units": { "sensitiveInformationPolicyUnits": 5 }, "cost": 0.0003 } ], "usage_units_by_key": { "a7eebae8ede9f9a6c815e56366ae1906d067f801fd8e3c3b1080046a3628a0e9": { "sensitiveInformationPolicyUnits": 4 }, "litellm_proxy_master_key": { "sensitiveInformationPolicyUnits": 1 } }, "cost": 0.0003, "cost_by_unit": { "sensitiveInformationPolicyUnits": 0.0003 }, "cost_by_team": { "": 0.0003 }, "cost_by_key": { "a7eebae8ede9f9a6c815e56366ae1906d067f801fd8e3c3b1080046a3628a0e9": 0.0003, "litellm_proxy_master_key": null }, "untracked_usage_units": { "sensitiveInformationPolicyUnits": 2 } }curl -s -G "http://localhost:4000/spend/logs/ui" --data-urlencode "start_date=2026-09-04 00:00:00" --data-urlencode "end_date=2026-09-05 00:00:00" --data-urlencode "page_size=1" -H 'Authorization: Bearer sk-1234', the unpriced request from step 4. The per-counter split stampssensitiveInformationPolicyUnitsasnull(unknown, which is what became the untracked unit above) while counters the map prices at 0.0 stay0.0(free); the per-request scalar that feeds spend counts the unknown as 0, unchanged from before{ "request_id": "chatcmpl-5be8e174-4e2c-4069-a04b-1bc20ef57719", "spend": 8.58e-05, "guardrail_cost": 0.0, "guardrail_cost_by_unit": { "wordPolicyUnits": 0.0, "topicPolicyUnits": 0.0, "contentPolicyUnits": 0.0, "contentPolicyImageUnits": 0.0, "automatedReasoningPolicies": null, "automatedReasoningPolicyUnits": 0.0, "contextualGroundingPolicyUnits": 0.0, "sensitiveInformationPolicyUnits": null, "sensitiveInformationPolicyFreeUnits": 0.0 } }Type
🆕 New Feature
Caveats (if any)
Low
unitswithout touchingcostoruntracked_units. A row an old pod creates lands withcostNULL and reads as untracked in full (step 2 of the After run), but an old pod's increment to a row this build already priced is indistinguishable from a priced unit until the fleet is on this buildguardrail_cost_by_unit; other providers showcost: nullguardrail_cost_by_unitechoes every counter Bedrock returns, zeros and nulls included, mirroringguardrail_usageguardrail_costthat feeds spend and budgets still counts an unpriced counter as $0, unchanged from beforeFinal Attestation
https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
Note
Medium Risk
Touches spend rollup, schema migration, and billing semantics (null vs $0 for unpriced units); API adds required fields but behavior is backward-compatible for spend totals.
Overview
Adds per-counter guardrail USD to the daily usage rollup and guardrails usage APIs, so admins see priced spend alongside unit counts without hand-summing spend logs.
The Bedrock hook now stamps
guardrail_cost_by_unitnext to usage; pricing treats missing map entries as unknown (null), not $0, while explicit 0.0 still means free. Spend billing keeps summing only known prices viaguardrail_cost_total, unchanged in behavior for unpriced counters.LiteLLM_DailyGuardrailUsageUnitsgains nullablecostanduntracked_units(with migration). The spend-log aggregator increments both alongside units, including on DB retry queues. Legacy rows withcostNULL read as fully untracked./guardrails/usage/overviewand detail exposecost,untrackedUsageUnits, totals, and detail breakdowns by unit/day/team/key; OpenAPI and dashboardschema.d.tsare updated. Policies overview fills the same response shape with null/empty cost fields.Reviewed by Cursor Bugbot for commit 9bd34ad. Bugbot is set up for automated code reviews on this repo. Configure here.