Skip to content

feat(guardrails): roll up Bedrock guardrail cost per usage counter - #39196

Merged
ryan-crabbe-berri merged 5 commits into
litellm_internal_stagingfrom
litellm_guardrail_usage_cost_rollup
Sep 4, 2026
Merged

feat(guardrails): roll up Bedrock guardrail cost per usage counter#39196
ryan-crabbe-berri merged 5 commits into
litellm_internal_stagingfrom
litellm_guardrail_usage_cost_rollup

Conversation

@ryan-crabbe-berri

@ryan-crabbe-berri ryan-crabbe-berri commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Guardrail usage endpoints report Bedrock billable units but never cost
  • Cost per counter was never stored, only a per-request total
  • Old rollup rows would read as $0 if cost defaulted to zero

How it solves it:

  • Bedrock hook stamps guardrail_cost_by_unit next to guardrail_usage
  • Aggregator sums it into two new columns on the daily units rollup: cost (USD for the priced units) and untracked_units (units that had no price), so a row that mixes priced and unpriced increments keeps its priced subtotal and says exactly how many units it leaves out
  • /guardrails/usage/overview gains cost and untrackedUsageUnits per row plus totalCost and totalUntrackedUsageUnits
  • /guardrails/usage/detail/{id} gains cost, daily cost, cost_by_unit/team/key and untracked_usage_units
  • Rows written before the migration keep cost NULL and read as untracked in full, never as $0
  • A counter missing from the cost map is untracked too; only an explicit 0.0 is free

User Flow

Before: a proxy admin wants to know what their Bedrock guardrail cost them this month and can only get unit counts

  1. They configure a Bedrock guardrail with a sensitive information policy and send POST http://localhost:4000/v1/chat/completions with a message containing an SSN
  2. The response comes back with the SSN handled by the guardrail and the request shows up in the spend log with a guardrail_cost of 0.0001
  3. They call GET http://localhost:4000/guardrails/usage/overview?start_date=2026-09-04&end_date=2026-09-04 and the guardrail row has "usageUnits": {"sensitiveInformationPolicyUnits": 3} but no cost field at all
  4. They call GET http://localhost:4000/guardrails/usage/detail/{guardrail_id}?start_date=2026-09-04&end_date=2026-09-04 and get units per day, team and key, again with no cost anywhere
  5. They fall back to summing guardrail_cost across every spend log row by hand

After: the same admin reads cost straight off the guardrail usage endpoints

  1. They configure the same Bedrock guardrail and send the same POST http://localhost:4000/v1/chat/completions with an SSN in the message
  2. The response and spend log look the same, and the spend log entry now also carries "guardrail_cost_by_unit": {"sensitiveInformationPolicyUnits": 0.0001}
  3. They call GET http://localhost:4000/guardrails/usage/overview?start_date=2026-09-04&end_date=2026-09-04 and the guardrail row reads "usageUnits": {"sensitiveInformationPolicyUnits": 3}, "cost": 0.0003, "untrackedUsageUnits": {} with "totalCost": 0.0003 at the top level
  4. They call GET http://localhost:4000/guardrails/usage/detail/{guardrail_id}?start_date=2026-09-04&end_date=2026-09-04 and get "cost": 0.0003, a cost on each daily point, cost_by_unit, cost_by_team, cost_by_key maps next to the existing unit maps, and "untracked_usage_units": {}
  5. A guardrail whose rows predate this change, or whose provider has no pricing entry, shows "cost": null rather than pretending it was free, and its untrackedUsageUnits lists every unit the null covers
  6. When some units had no price at the time (an old pod during a rolling deploy, a counter the cost map does not list yet), cost still covers the priced units and untrackedUsageUnits reads {"sensitiveInformationPolicyUnits": 2}, so they know the $0.0003 covers 3 of the 5 units

Relevant issues

Linear ticket

Refs LIT-5652

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup. A Bedrock guardrail in us-east-1 with a sensitive information policy (SSN, anonymize) runs default_on in pre_call mode. Each request bills one sensitiveInformationPolicyUnits, priced at $0.0001 in the cost map. Both runs hit the real Bedrock ApplyGuardrail and Claude Haiku 4.5 APIs and use the proxy_batch_write_at: 5 flush so the rollup lands within seconds

model_list:
  - model_name: bedrock-invoke-haiku-4-5
    litellm_params:
      model: bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
      aws_region_name: us-east-1

guardrails:
  - guardrail_name: bedrock-pii-mask
    litellm_params:
      guardrail: bedrock
      mode: pre_call
      guardrailIdentifier: 164nbov72q77
      guardrailVersion: DRAFT
      aws_region_name: us-east-1
      default_on: true

general_settings:
  master_key: sk-1234
  proxy_batch_write_at: 5

The request sent in both runs:

curl -s http://localhost:4000/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"bedrock-invoke-haiku-4-5","messages":[{"role":"user","content":"My SSN is 123-45-6789, reply with just the word ok"}],"max_tokens":10}'

Before (8bc862f)

  1. Send the request above, then wait for the spend flush

    chatcmpl-bacfc97b-91fd-43de-a596-f5483e2d6eed "I can't process or acknowledge Social Security Numbers or"
    
  2. curl -s "http://localhost:4000/guardrails/usage/overview?start_date=2026-09-01&end_date=2026-09-01" -H 'Authorization: Bearer sk-1234', guardrail row and totals only (no cost fields exist)

    {
      "row": {
        "id": "eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c",
        "name": "bedrock-pii-mask",
        "type": "Guardrail",
        "provider": "bedrock",
        "requestsEvaluated": 4,
        "failRate": 0.0,
        "avgScore": null,
        "avgLatency": null,
        "status": "healthy",
        "trend": "stable",
        "usageUnits": {
          "sensitiveInformationPolicyUnits": 4
        }
      },
      "totalUsageUnits": {
        "sensitiveInformationPolicyUnits": 4
      },
      "totalCost": "<absent>"
    }
  3. curl -s "http://localhost:4000/guardrails/usage/detail/eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c?start_date=2026-09-01&end_date=2026-09-01" -H 'Authorization: Bearer sk-1234', unit fields only

    {
      "usage_units": {
        "sensitiveInformationPolicyUnits": 4
      },
      "usage_units_daily": [
        {
          "date": "2026-09-01",
          "units": {
            "sensitiveInformationPolicyUnits": 4
          }
        }
      ],
      "cost": "<absent>",
      "cost_by_unit": "<absent>",
      "cost_by_team": "<absent>",
      "cost_by_key": "<absent>"
    }

After (6c81a5c)

Three ways units end up unpriced, all on the same guardrail and day: a row written by the old build (no cost column at all), a priced row on this build, then an unpriced increment landing on that same priced row because the cost map had no price for the counter at the time

  1. Boot log from this branch's first boot on the DB shows the schema-only migration applying before traffic is served (this run booted with both columns already present)

    Applying migration `20260901000001_add_guardrail_usage_units_cost`
    All migrations have been successfully applied.
    
  2. The old build (8bc862f) sends the request above once with the master key, then waits for the spend flush. Its aggregator knows nothing about cost, so the row lands with cost NULL

    chatcmpl-46a6b105-d9d1-41f4-90cb-528b30af4e69 "I can't process or acknowledge Social Security Numbers or"
    
  3. Swap to this build, curl -s http://localhost:4000/key/generate -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"key_alias":"guardrail-cost-qa","models":["bedrock-invoke-haiku-4-5"]}', and send the request above three times with the returned key (one of the three got a Bedrock 503 on the model call after the guardrail had already billed its unit, which is why it still counts)

    chatcmpl-534ce0b3-0d38-4071-909f-76c9acee1d20 "I can't process or acknowledge Social Security Numbers or"
    chatcmpl-5cc6186b-f251-4502-b051-18bcb8f07e36 "I can't process or acknowledge social security numbers or"
    
  4. Restart this build with LITELLM_MODEL_COST_MAP_URL pointing at a copy of the cost map whose bedrock/guardrails entry has no sensitiveInformationPolicyUnits price (the state a deployment is in whenever Bedrock bills a counter before the map catches up), then send the request once more with the same key

    chatcmpl-5be8e174-4e2c-4069-a04b-1bc20ef57719 "I can't process that. Please don't share"
    
  5. The rollup rows for the day, read straight from LiteLLM_DailyGuardrailUsageUnits (date, guardrail_id, api_key, usage_unit, units, cost, untracked_units). The same row holds the priced subtotal and the unpriced unit instead of collapsing to NULL

    2026-09-04 bedrock-pii-mask litellm_proxy_master_key sensitiveInformationPolicyUnits 1 None   0
    2026-09-04 bedrock-pii-mask a7eebae8ede9...          sensitiveInformationPolicyUnits 4 0.0003 1
    
  6. curl -s "http://localhost:4000/guardrails/usage/overview?start_date=2026-09-04&end_date=2026-09-04" -H 'Authorization: Bearer sk-1234', the row prices the 3 units it could and names the 2 it could not (1 from the old build's row, 1 from the unpriced increment), and the totals do the same

    {
      "row": {
        "id": "eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c",
        "name": "bedrock-pii-mask",
        "type": "Guardrail",
        "provider": "bedrock",
        "requestsEvaluated": 5,
        "failRate": 0.0,
        "avgScore": null,
        "avgLatency": null,
        "status": "healthy",
        "trend": "stable",
        "usageUnits": {
          "sensitiveInformationPolicyUnits": 5
        },
        "cost": 0.0003,
        "untrackedUsageUnits": {
          "sensitiveInformationPolicyUnits": 2
        }
      },
      "totalUsageUnits": {
        "sensitiveInformationPolicyUnits": 5
      },
      "totalCost": 0.0003,
      "totalUntrackedUsageUnits": {
        "sensitiveInformationPolicyUnits": 2
      }
    }
  7. curl -s "http://localhost:4000/guardrails/usage/detail/eb6c258a-dd2e-5ebc-b8f3-e4a10d5adb5c?start_date=2026-09-04&end_date=2026-09-04" -H 'Authorization: Bearer sk-1234', cost rides every unit breakdown, the master key's row (the old build's) reads null in cost_by_key, and untracked_usage_units counts both unpriced units

    {
      "usage_units": {
        "sensitiveInformationPolicyUnits": 5
      },
      "usage_units_daily": [
        {
          "date": "2026-09-04",
          "units": {
            "sensitiveInformationPolicyUnits": 5
          },
          "cost": 0.0003
        }
      ],
      "usage_units_by_key": {
        "a7eebae8ede9f9a6c815e56366ae1906d067f801fd8e3c3b1080046a3628a0e9": {
          "sensitiveInformationPolicyUnits": 4
        },
        "litellm_proxy_master_key": {
          "sensitiveInformationPolicyUnits": 1
        }
      },
      "cost": 0.0003,
      "cost_by_unit": {
        "sensitiveInformationPolicyUnits": 0.0003
      },
      "cost_by_team": {
        "": 0.0003
      },
      "cost_by_key": {
        "a7eebae8ede9f9a6c815e56366ae1906d067f801fd8e3c3b1080046a3628a0e9": 0.0003,
        "litellm_proxy_master_key": null
      },
      "untracked_usage_units": {
        "sensitiveInformationPolicyUnits": 2
      }
    }
  8. curl -s -G "http://localhost:4000/spend/logs/ui" --data-urlencode "start_date=2026-09-04 00:00:00" --data-urlencode "end_date=2026-09-05 00:00:00" --data-urlencode "page_size=1" -H 'Authorization: Bearer sk-1234', the unpriced request from step 4. The per-counter split stamps sensitiveInformationPolicyUnits as null (unknown, which is what became the untracked unit above) while counters the map prices at 0.0 stay 0.0 (free); the per-request scalar that feeds spend counts the unknown as 0, unchanged from before

    {
      "request_id": "chatcmpl-5be8e174-4e2c-4069-a04b-1bc20ef57719",
      "spend": 8.58e-05,
      "guardrail_cost": 0.0,
      "guardrail_cost_by_unit": {
        "wordPolicyUnits": 0.0,
        "topicPolicyUnits": 0.0,
        "contentPolicyUnits": 0.0,
        "contentPolicyImageUnits": 0.0,
        "automatedReasoningPolicies": null,
        "automatedReasoningPolicyUnits": 0.0,
        "contextualGroundingPolicyUnits": 0.0,
        "sensitiveInformationPolicyUnits": null,
        "sensitiveInformationPolicyFreeUnits": 0.0
      }
    }

Type

🆕 New Feature

Caveats (if any)

Low

  • During a rolling deploy, old pods increment units without touching cost or untracked_units. A row an old pod creates lands with cost NULL and reads as untracked in full (step 2 of the After run), but an old pod's increment to a row this build already priced is indistinguishable from a priced unit until the fleet is on this build
  • Only the Bedrock hook stamps guardrail_cost_by_unit; other providers show cost: null
  • guardrail_cost_by_unit echoes every counter Bedrock returns, zeros and nulls included, mirroring guardrail_usage
  • The per-request guardrail_cost that feeds spend and budgets still counts an unpriced counter as $0, unchanged from before
  • No UI change yet; the Guardrails Monitor still discards these fields (follow-up PR)

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW


Note

Medium Risk
Touches spend rollup, schema migration, and billing semantics (null vs $0 for unpriced units); API adds required fields but behavior is backward-compatible for spend totals.

Overview
Adds per-counter guardrail USD to the daily usage rollup and guardrails usage APIs, so admins see priced spend alongside unit counts without hand-summing spend logs.

The Bedrock hook now stamps guardrail_cost_by_unit next to usage; pricing treats missing map entries as unknown (null), not $0, while explicit 0.0 still means free. Spend billing keeps summing only known prices via guardrail_cost_total, unchanged in behavior for unpriced counters.

LiteLLM_DailyGuardrailUsageUnits gains nullable cost and untracked_units (with migration). The spend-log aggregator increments both alongside units, including on DB retry queues. Legacy rows with cost NULL read as fully untracked.

/guardrails/usage/overview and detail expose cost, untrackedUsageUnits, totals, and detail breakdowns by unit/day/team/key; OpenAPI and dashboard schema.d.ts are updated. Policies overview fills the same response shape with null/empty cost fields.

Reviewed by Cursor Bugbot for commit 9bd34ad. Bugbot is set up for automated code reviews on this repo. Configure here.

The daily guardrail usage rollup stored billable units per counter but no
cost, so the usage endpoints could only report units. The Bedrock hook now
stamps guardrail_cost_by_unit next to guardrail_usage, the spend-log
aggregator sums it into a new nullable cost column on
LiteLLM_DailyGuardrailUsageUnits, and /guardrails/usage/overview and
/guardrails/usage/detail/{id} return cost, totalCost and cost_by_unit /
cost_by_team / cost_by_key alongside the existing unit breakdowns.

Cost is nullable on purpose. Rows written before this migration, and rows
whose hook had no pricing entry, read as null rather than $0, and a single
unpriced increment keeps that row's cost unknown instead of partial.
guardrail_cost and the spend/budget path are untouched.

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
@codspeed-hq

codspeed-hq Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_guardrail_usage_cost_rollup (9bd34ad) with litellm_internal_staging (336891b)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (f59021f) during the generation of this report, so 336891b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds per-counter Bedrock guardrail cost tracking and exposes priced subtotals alongside explicitly untracked usage.

  • Adds nullable cost and untracked-unit columns to the daily guardrail usage rollup.
  • Preserves unknown counter prices as untracked units while retaining explicit zero-cost counters as free.
  • Extends guardrail usage overview and detail responses with cost totals and breakdowns.
  • Adds focused coverage for priced, free, unknown, legacy, and mixed rollup states.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/litellm_core_utils/llm_cost_calc/guardrail_cost.py Produces nullable per-counter costs so missing prices remain distinguishable from explicitly free counters.
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py Stamps Bedrock usage with per-counter pricing while preserving the existing scalar spend calculation.
litellm/proxy/guardrails/usage_tracking.py Accumulates priced cost and untracked units independently, resolving the previously reported mixed-increment loss.
litellm/proxy/guardrails/usage_endpoints.py Exposes tracked cost and untracked-unit totals across overview, daily, unit, team, and key breakdowns.
litellm-proxy-extras/litellm_proxy_extras/migrations/20260901000001_add_guardrail_usage_units_cost/migration.sql Adds the nullable cost and additive untracked_units columns required by the new rollup representation.

Reviews (4): Last reviewed commit: "feat(guardrails): store untracked units ..." | Re-trigger Greptile

Comment thread litellm/litellm_core_utils/llm_cost_calc/guardrail_cost.py Outdated
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.09524% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/proxy/guardrails/usage_endpoints.py 94.73% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4914914. Configure here.

A counter missing from the cost map entry was priced at 0.0 per unit, so
the rollup recorded it as known-free usage. It now stamps None for that
counter and the rollup writes NULL, while the per-request guardrail_cost
that feeds spend and budgets still sums only the known prices.

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai pushed the unknown-counter fix: an unpriced counter now stamps null and the rollup stores NULL, not $0

A row's cost sums only the daily rows that carry a tracked cost, so it
silently under-reports whenever some rows are NULL (pre-migration days,
old pods mid-rollout, an unpriced counter). Both usage endpoints now
return the per-counter units behind those NULL rows next to the cost
(untrackedUsageUnits / totalUntrackedUsageUnits on the overview,
untracked_usage_units on the detail), so a partial cost is never mistaken
for a complete one and the reader can see exactly what it excludes

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review: each cost now ships with the per-counter units it leaves out (untrackedUsageUnits), plus tests and a live rolling-deploy run

Comment thread litellm/proxy/guardrails/usage_tracking.py
…nulling cost

A row that received both priced and unpriced increments used to collapse
to cost NULL, throwing away the priced subtotal and making every unit on
it read as untracked. The rollup now carries a second column,
untracked_units, that the aggregator increments for units with no known
price while cost keeps accruing for the rest, so cost covers exactly
units - untracked_units. Rows written before the migration keep cost
NULL and still read as untracked in full

The endpoints read untracked units off the column (or the whole row for
a legacy NULL) rather than from a NULL filter, and the policies overview
now fills totalUntrackedUsageUnits, which the previous commit missed

Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai please re-review: the rollup row now stores untracked_units, so mixed priced and unpriced increments keep the priced subtotal

…itellm_guardrail_usage_cost_rollup

# Conflicts:
#	type-discipline-budget.json
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 9bd34ad. Configure here.

@ryan-crabbe-berri
ryan-crabbe-berri merged commit d23bec8 into litellm_internal_staging Sep 4, 2026
200 of 201 checks passed
@ryan-crabbe-berri
ryan-crabbe-berri deleted the litellm_guardrail_usage_cost_rollup branch September 4, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants