fix(datadog_llm_obs): keep the guardrail audit record under message redaction by yucheng-berri · Pull Request #39702 · BerriAI/litellm

yucheng-berri · 2026-09-04T03:34:04Z

TLDR

Problem this solves:

Message redaction blanked the guardrail record on Datadog spans
Operators could no longer tell what a guardrail caught
Any API key could erase its own guardrail record with one header

How it solves it:

Replace only the four guardrail fields that can quote the prompt
Keep name, mode, status, timings and masked-entity counts
Reuse the field list the spend-log redaction already applies

User Flow

Before: a compliance engineer who turned message logging off cannot answer "did a guardrail catch anything on that request"

The proxy admin sets turn_off_message_logging: true and turns on a PII guardrail
A developer sends POST https://litellm-domain/v1/chat/completions with mail alice@acme.com and bob@acme.com
The guardrail masks both addresses and the request succeeds
The engineer opens the request in Datadog LLM Observability and sees guardrail_information: null
Nothing on the span says which guardrail ran, whether it intervened, or how many entities it masked
Any developer holding an ordinary key can produce the same blank record on demand, by adding x-litellm-enable-message-redaction: true to their own request

After: the same span still hides the prompt and now carries the guardrail record

The proxy admin sets turn_off_message_logging: true and turns on a PII guardrail
A developer sends the same POST https://litellm-domain/v1/chat/completions
The guardrail masks both addresses and the request succeeds
The engineer opens the request in Datadog LLM Observability and sees the guardrail entry with guardrail_name: presidio-pii, guardrail_status, its timings, and masked_entity_count: {"EMAIL_ADDRESS": 2}
The prompt is still redacted-by-litellm, and the guardrail's own copy of it reads REDACTED_BY_LITELM
A developer sending x-litellm-enable-message-redaction: true still hides their own prompt and no longer removes the guardrail record

Relevant issues

Linear ticket

Refs LIT-6728

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Both cases are end to end against real Datadog LLM Observability (us5) with a real OpenAI call and a real Presidio guardrail; the spans are read back through Datadog's own LLM Obs search API. Nothing is mocked.

Shared setup:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY
guardrails:
  - guardrail_name: "presidio-pii"
    litellm_params:
      guardrail: presidio
      mode: "pre_call"
      default_on: true
litellm_settings:
  callbacks: ["datadog_llm_observability"]
  turn_off_message_logging: true    # case 1 only; case 2 runs with this false
general_settings:
  master_key: sk-1234

Read-back command used in every step below:

curl -sS -X POST "https://api.us5.datadoghq.com/api/unstable/llm-obs/v1/spans/events/search" \
  -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"data":{"type":"spans","attributes":{"filter":{"from":"now-15m","to":"now","query":"env:<env>"},"page":{"limit":2},"sort":"-timestamp"}}}'

Before (`aec083c`, `litellm_internal_staging` with #39402 as merged)

Case 1: operator sets turn_off_message_logging

Send the request

curl -sS http://127.0.0.1:20403/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"mail alice@acme.com and bob@acme.com"}],"max_tokens":12}'
# http 200

Read the span back from Datadog

spans: 1
 input:                     {"value": "redacted-by-litellm", "messages": [{"content": "redacted-by-litellm", "role": "user"}]}
 guardrail_information:     null
 applied_guardrails:        ["presidio-pii"]
 guardrail_overhead_time_ms: 88.551

The span records that 88 ms went to guardrails and gives no way to find out which one, whether it intervened, or what it masked.

Case 2: ordinary key sends the redaction header

Mint an ordinary virtual key and confirm it is not an admin

curl -sS http://127.0.0.1:20403/key/generate -H "Authorization: Bearer sk-1234" \
  -H 'Content-Type: application/json' -d '{"models":["gpt-4o"],"key_alias":"ordinary-caller-39402"}'
curl -sS http://127.0.0.1:20403/key/list    -H "Authorization: Bearer $KEY" -o /dev/null -w "%{http_code}\n"   # 403
curl -sS http://127.0.0.1:20403/config/update -H "Authorization: Bearer $KEY" -d '{}' -o /dev/null -w "%{http_code}\n"  # 401

Send a request with the redaction header, with turn_off_message_logging: false in the config

curl -sS http://127.0.0.1:20403/v1/responses \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -H 'x-litellm-enable-message-redaction: true' \
  -d '{"model":"gpt-4o","input":"mail alice@acme.com and bob@acme.com","max_output_tokens":16}'
# http 200

Read the span back from Datadog

spans: 1
 input:                 {"value": "redacted-by-litellm", "messages": [{"content": "redacted-by-litellm", "role": "user"}]}
 key_alias:             ordinary-caller-39402
 guardrail_information: null

After (`64fc74b`, this PR)

Case 1: operator sets turn_off_message_logging

Send the same request

curl -sS http://127.0.0.1:20402/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H 'Content-Type: application/json' \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"mail alice@acme.com and bob@acme.com"}],"max_tokens":12}'
# http 200

Read the span back from Datadog

{
  "input": {"value": "redacted-by-litellm", "messages": [{"content": "redacted-by-litellm", "role": "user"}]},
  "metadata": {
    "applied_guardrails": ["presidio-pii"],
    "guardrail_information": [
      {"guardrail_name": "presidio-pii", "guardrail_provider": "presidio", "guardrail_mode": "pre_call",
       "guardrail_status": "success", "guardrail_response": "REDACTED_BY_LITELM",
       "masked_entity_count": {"EMAIL_ADDRESS": 2},
       "start_time": 1788555468.743473, "end_time": 1788555468.80831, "duration": 0.064839},
      {"guardrail_name": "presidio-pii", "guardrail_provider": "presidio", "guardrail_mode": "post_call",
       "guardrail_status": "success", "guardrail_response": "REDACTED_BY_LITELM",
       "start_time": 1788555470.004475, "end_time": 1788555470.031875, "duration": 0.027402}
    ],
    "latency_metrics": {"guardrail_overhead_time_ms": 92.241}
  }
}

The prompt is still gone. The record of what the guardrail did is back.

Case 2: ordinary key sends the redaction header

Send the same request with the same ordinary key and the same header

curl -sS http://127.0.0.1:20402/v1/responses \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -H 'x-litellm-enable-message-redaction: true' \
  -d '{"model":"gpt-4o","input":"mail alice@acme.com and bob@acme.com","max_output_tokens":16}'
# http 200

Read the span back from Datadog

spans: 1
 input:                 {"value": "redacted-by-litellm", "messages": [{"content": "redacted-by-litellm", "role": "user"}]}
 key_alias:             ordinary-caller-39402
 guardrail_information: [{"guardrail_name": "presidio-pii", "guardrail_provider": "presidio",
                          "guardrail_mode": "pre_call", "guardrail_status": "success",
                          "guardrail_response": "REDACTED_BY_LITELM",
                          "masked_entity_count": {"EMAIL_ADDRESS": 2},
                          "start_time": 1788555528.537273, "end_time": 1788555528.58951, "duration": 0.052238},
                         {"guardrail_name": "presidio-pii", "guardrail_mode": "post_call", ...}]

The key still redacts its own prompt and can no longer erase what the guardrail caught.

Type

🐛 Bug Fix

Caveats (if any)

Medium

Span field changes from null to an array under redaction
- Only for the redacted case; the unredacted span is unchanged
- This restores the shape that shipped before feat(datadog_llm_obs): cost tag dimensions, router decision fields, reasoning token metric, redaction gating #39402
- A Datadog monitor written against null in the last day would see the type change
- No in-repo consumer reads this span field

Low

Four other metadata records stay dropped under redaction
- requester_metadata, prompt_management_metadata, mcp_tool_call_metadata, vector_store_request_metadata
- Those carry caller text, tool arguments and retrieved passages, so dropping them is correct
routing_decision is still re-emitted flattened, not restored nested
- Deliberately left alone; the router already strips its prompt-quoting fields upstream
- It is a routing telemetry record, not an enforcement audit record

Reviewer notes

A redacted span can carry two different redaction markers at once
- The message body reads redacted-by-litellm and the guardrail's own copy of the prompt reads REDACTED_BY_LITELM
- Two different constants that already existed; this PR reuses the one the spend-log path uses for the same field, rather than introducing a third
Datadog spans still do not carry the numeric guardrail compression stats that spend logs keep
- Pre-existing on both sides of this diff and out of scope here
A guardrail record left as a single mapping rather than a list is normalized before redaction and before latency extraction
- guardrail_information is typed as a list, and Prometheus already normalizes the same shape at _guardrail_overhead_seconds
- Before this PR that shape raised inside _get_latency_metrics and the span was dropped, on both sides of the diff
- No first-party guardrail produces it today, so it is covered by unit tests rather than a live leg

Testing

tests/test_litellm/integrations/datadog/test_datadog_llm_obs.py — 68 passed
Mutation check: reverting datadog_llm_obs.py to the merge-base version fails 5 of the new tests, and dropping the single-record normalization alone fails one more
Live: real Datadog LLM Observability (us5), real OpenAI call, real Presidio guardrail, both triggers, before and after
Live regression legs re-run on this head: unredacted-span A/B base vs head identical, guardrail_overhead_time_ms still computed on both sides; spend-log sanitizer output byte-identical base vs head; 4 workers with 8-way concurrency (16/16 200s, one span shape, no prompt text); paused-Postgres fault leg (3/3 200s, no malformed records)

ran /live-pr-risk and found no regressions/backward incompatible risks

Note

Low Risk
Observability-only redaction behavior change: redacted spans expose guardrail audit metadata instead of null, with stricter field filtering; spend-log sanitizer uses the same constant with no logic change beyond import.

Overview
When message redaction is on (turn_off_message_logging, per-request header, etc.), Datadog LLM Observability spans no longer drop guardrail_information entirely. They now redact only the four prompt-carrying fields (guardrail_request, guardrail_response, match_details, classification) to REDACTED_BY_LITELM, while keeping audit fields such as name, provider, mode, status, timings, and masked-entity counts.

PROMPT_CARRYING_GUARDRAIL_FIELDS and AUDIT_GUARDRAIL_FIELDS are defined in litellm/types/utils.py and reused by the Datadog logger and spend-log sanitizer (replacing a local duplicate). Redacted spans use an allow-list: unclassified guardrail keys are stripped so custom fields cannot leak prompts.

_guardrail_entries normalizes list vs single-dict guardrail_information for redaction and guardrail_overhead_time_ms computation. New unit tests cover audit survival, header redaction, odd shapes, and full field coverage.

^{Reviewed by Cursor Bugbot for commit 64fc74b. Bugbot is set up for automated code reviews on this repo. Configure here.}

…edaction Redaction nulled `guardrail_information` on the span whole, so an operator running `turn_off_message_logging` (or a caller sending `x-litellm-enable-message-redaction`) lost the record of which guardrails ran, what they returned, and what they masked. Four of the record's fields can quote the prompt; the rest report what the guardrail decided without reproducing it. Replace only those four, the way `_sanitize_guardrail_information_for_spend_logs` already does for spend logs, and declare the field list once in `litellm/types/utils.py` so both readers share it.

yucheng-berri · 2026-09-04T03:34:09Z

@greptileai please review the current head 844dcff

yucheng-berri · 2026-09-04T03:34:14Z

bugbot run

codspeed-hq · 2026-09-04T03:36:16Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_dd_llm_obs_guardrail_audit_redaction (64fc74b) with litellm_internal_staging (300d335)}

devin-ai-integration

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

devin-ai-integration · 2026-09-04T03:36:20Z

+        "detection_method",
+        "confidence_score",
+        "patterns_checked",
+        "alert_recipients",


🟥 Redacted spans expose recipient addresses

With message redaction enabled, alert_recipients remains unchanged. Datadog receives personal email addresses from otherwise redacted requests

Was this helpful? React with 👍 or 👎 to provide feedback.

alert_recipients has no writer anywhere in the repo, and it names operator alert destinations rather than caller prompt text. Keeping it

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 844dcff. Configure here.}

greptile-apps · 2026-09-04T03:38:25Z

Greptile Summary

The PR preserves guardrail audit metadata on redacted Datadog spans while replacing prompt-carrying guardrail fields and reusing the same field classification in spend-log sanitization.

Adds shared prompt-carrying and audit guardrail field sets.
Normalizes single and list-shaped guardrail records for redaction and latency extraction.
Adds behavioral coverage for redacted guardrail metadata and caller-requested message redaction.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/integrations/datadog/datadog_llm_obs.py	Preserves allow-listed guardrail audit fields during message redaction and normalizes guardrail records before latency calculation.
litellm/proxy/spend_tracking/spend_tracking_utils.py	Reuses the shared prompt-carrying guardrail field classification without changing spend-log sanitization behavior.
litellm/types/utils.py	Defines shared classifications for prompt-carrying and audit-safe guardrail fields.
tests/test_litellm/integrations/datadog/test_datadog_llm_obs.py	Exercises redacted Datadog span output across normal, caller-header, custom-field, and irregular guardrail-record shapes.

_{Reviews (2): Last reviewed commit: "fix(datadog_llm_obs): keep a lone guardr..." | Re-trigger Greptile}

codecov · 2026-09-04T03:47:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…the span Review round 1. A guardrail that writes the metadata key itself leaves a single record where the type says list, which Prometheus already normalizes at `_guardrail_overhead_seconds`. Redaction dropped that shape and the latency extraction raised on it, so the span was lost outright. Normalize once and use it in both places. The new tests now drive `create_llm_obs_payload` instead of reading the module's private helpers and the record's declared field names.

yucheng-berri · 2026-09-04T21:00:18Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 64fc74b. Configure here.}

yucheng-berri · 2026-09-04T21:25:33Z

@greptileai please review the current head 64fc74b

devin-ai-integration Bot reviewed Sep 4, 2026

View reviewed changes

cursor Bot reviewed Sep 4, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread tests/test_litellm/integrations/datadog/test_datadog_llm_obs.py Outdated

cursor Bot reviewed Sep 4, 2026

View reviewed changes

tin-berri approved these changes Sep 5, 2026

View reviewed changes

yucheng-berri merged commit e2741b5 into litellm_internal_staging Sep 5, 2026
187 of 190 checks passed

yucheng-berri deleted the litellm_dd_llm_obs_guardrail_audit_redaction branch September 5, 2026 01:24

This was referenced Sep 5, 2026

fix(datadog_llm_obs): keep guardrail_cost_by_unit on redacted spans #39848

Merged

feat(fireworks_ai): add native Responses API config #39826

Merged

fix(proxy): gate the OpenAI websocket passthrough behind an explicit opt-in #39841

Merged

Uh oh!

Uh oh!

Conversation

yucheng-berri commented Sep 4, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (aec083c, litellm_internal_staging with #39402 as merged)

Case 1: operator sets turn_off_message_logging

Case 2: ordinary key sends the redaction header

After (64fc74b, this PR)

Case 1: operator sets turn_off_message_logging

Case 2: ordinary key sends the redaction header

Type

Caveats (if any)

Medium

Low

Reviewer notes

Testing

Uh oh!

yucheng-berri commented Sep 4, 2026

Uh oh!

yucheng-berri commented Sep 4, 2026

Uh oh!

codspeed-hq Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devin-ai-integration Bot Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

yucheng-berri Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

codecov Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yucheng-berri commented Sep 4, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yucheng-berri commented Sep 4, 2026 •

edited by cursor Bot

Loading

Before (`aec083c`, `litellm_internal_staging` with #39402 as merged)

After (`64fc74b`, this PR)

codspeed-hq Bot commented Sep 4, 2026 •

edited

Loading

greptile-apps Bot commented Sep 4, 2026 •

edited

Loading

codecov Bot commented Sep 4, 2026 •

edited

Loading